Character encoding
"I am currently a Software Engineering student at ALX. I'm passionate about technology and enjoy conducting research to find answers on my own. I have a natural inclination to ask 'WHY' more often than 'HOW'.
"While working on projects at ALX, I have acquired a wealth of interesting and diverse knowledge about software engineering and computer science in general. Therefore, I needed a place to store and save all this information, allowing me to refer back to it whenever I forget."
Bytes are bytes; characters are an abstraction. A string is a sequence of Unicode characters. But a file on disk is not a sequence of Unicode characters; a file on disk is a sequence of bytes. So if you read a “text file” from disk, how does Python convert that sequence of bytes into a sequence of characters? It decodes the bytes according to a specific character encoding algorithm and returns a sequence of Unicode characters (otherwise known as a string).
# This example was created on Windows. Other platforms may
# behave differently, for reasons outlined below.
>>> file = open('examples/chinese.txt')
>>> a_string = file.read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python31\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 28: character maps to <undefine>>>
The default encoding is platform dependent.
What just happened?
You didn’t specify a character encoding, so Python is forced to use the default encoding.
What’s the default encoding?
If you look closely at the traceback, you can see that it’s dying in cp1252.py, meaning that Python is using CP-1252 as the default encoding here. (CP-1252 is a common encoding on computers running Microsoft Windows.) The CP-1252 character set doesn’t support the characters that are in this file, so the read fails with an ugly UnicodeDecodeError. But wait, it’s worse than that! The default encoding is platform-dependent, so this code might work on your computer (if your default encoding is UTF -8), but then it will fail when you distribute it to someone else (whose default encoding is different, like CP-1252).
If you need to get the default character encoding, import the locale module and call locale.getpreferredencoding().
On my Windows laptop, it returns 'cp1252', but on my Linux box, it returns 'UTF8'. Your results may be different (even on Windows) depending on which version of your operating system you have installed and how your regional/language settings are configured. This is why it’s so important to specify the encoding every time you open a file.
