Chapter 4

Text and Encodings

Every example so far has quietly assumed the text in these files is simple enough to read without telling Python anything about it. That assumption holds for plain English and breaks the moment a file has an accented name in it, a curly quote, or a word from another language. This chapter is what's actually going on underneath, and the one habit that makes the problem go away.

A file is bytes

Open a file in "rb" mode, "read binary", and Python hands you the raw bytes instead of decoding them into text:

with open("notes/one-hundred-years-of-solitude.txt", "rb") as f:
    f.readline()             # the title line
    author = f.readline()

print(author)

b'Gabriel Garc\xc3\xada M\xc3\xa1rquez\n'

That b prefix marks a bytes object. Most of it is recognisable: G, a, b, and so on are single bytes with their familiar values. But the í in García shows up as \xc3\xad, two bytes, and the á as \xc3\xa1. The accented letters take more than one byte each.

A byte is just a number from 0 to 255. On its own it doesn't mean "G" or "í". It only becomes a character when something decides how to read it, and that something is an encoding.

Encoding and decoding

An encoding is the rulebook that maps bytes to characters and back. Turning bytes into text is decoding. Turning text into bytes is encoding.

text = author.decode("utf-8").strip()
print(text)
print(text.encode("utf-8"))

Gabriel García Márquez
b'Gabriel Garc\xc3\xada M\xc3\xa1rquez'

decode("utf-8") read those bytes back into the right characters (.strip() just drops the trailing newline). encode("utf-8") turned them into the same bytes again. That round trip only works because both ends agreed on "utf-8".

UTF-8 is the encoding almost everything uses now: web pages, source files, most text files you'll be handed. It's variable width. The plain ASCII characters are one byte each, exactly as they've always been, and everything else is two, three, or four bytes. That's why García was longer in bytes than it is in characters.

Why you have to name the encoding

When you open a file in text mode without saying encoding=, Python picks a default for you. The trouble is the default is not the same on every computer. On Linux and current macOS it's UTF-8. On most Windows machines it's a Windows encoding called cp1252.

So this line:

with open("notes/one-hundred-years-of-solitude.txt") as f:
    text = f.read()

reads perfectly on your Linux server and comes back garbled on a colleague's Windows laptop, or the other way round. Same code, same file, different result, and nothing raised an error to tell you.

The fix is to say what you mean, every time:

with open("notes/one-hundred-years-of-solitude.txt", encoding="utf-8") as f:
    original_title = f.read().splitlines()[2]

print(original_title)

Original title: Cien años de soledad

Read the same file with the wrong encoding and you get mojibake: text decoded by the wrong rulebook. Latin-1 is a good stand-in for what a Windows default would do here:

with open("notes/one-hundred-years-of-solitude.txt", encoding="latin-1") as f:
    original_title = f.read().splitlines()[2]

print(original_title)

Original title: Cien años de soledad

años became años. The ñ was two UTF-8 bytes, and Latin-1 read each of them as its own character. No error, just wrong. If you have ever seen Ã where an accent should be, this is what happened.

When it does crash: UnicodeDecodeError

Some encodings refuse bytes they have no character for, and then you get a real error instead of silent mojibake.

ASCII is the strictest. It only defines the first 128 values, so any byte from a multi-byte UTF-8 character stops it:

"café".encode("utf-8").decode("ascii")

That's UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3.

cp1252 is looser but still leaves five byte values undefined. UTF-8 text hits one of them sooner or later:

b"caf\x9d".decode("cp1252")

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d. If you have met that exact charmap message before, it was almost always a UTF-8 file being read as cp1252 on Windows.

The message is doing you a favor. It names the byte and its position, which tells you the problem is the encoding and roughly where in the file to look.

errors=: what to do with a byte you can't read

decode and open take an errors= argument that says what to do when a byte doesn't fit:

raw = "café".encode("utf-8")

print(raw.decode("ascii", errors="replace"))
print(raw.decode("ascii", errors="ignore"))

caf��
caf

"strict" is the default and raises, which you saw above. "replace" swaps in the replacement character � and keeps going, so you can read the rest of the file and see where the damage is. "ignore" drops the bad bytes without a trace, which turns café into caf and gives you no hint that a character went missing.

Use "replace" when you're eyeballing a mangled log and want the readable parts. Reach for "ignore" almost never.

The byte order mark

Files that came out of Excel's "Save As CSV" often begin with three invisible bytes, EF BB BF, called a byte order mark, or BOM. Read such a file as plain UTF-8 and the mark rides along on the first value:

with open("from-excel.csv", "wb") as f:
    f.write(b"\xef\xbb\xbftitle,rating\n")
    f.write(b"Piranesi,5\n")

with open("from-excel.csv", encoding="utf-8") as f:
    with_bom = f.readline()

with open("from-excel.csv", encoding="utf-8-sig") as f:
    without_bom = f.readline()

print("plain utf-8, first character:", hex(ord(with_bom[0])))
print("utf-8-sig, first character: ", hex(ord(without_bom[0])))

plain utf-8, first character: 0xfeff
utf-8-sig, first character:  0x74

Read as plain "utf-8", the first character is 0xfeff, the byte order mark itself, glued to the front of title. That breaks a header check or a column lookup. Read as "utf-8-sig", the first character is 0x74, a plain t, and the mark is gone. Use "utf-8-sig" when you read files from Windows tools, and stick with plain "utf-8" when you write, so you don't add a BOM that the next program has to deal with.

Catching the mistake before it ships

Python can warn you every time you open a text file without naming the encoding. Run your script with a flag:

$ python -X warn_default_encoding check.py
check.py:2: EncodingWarning: 'encoding' argument not specified
  with open("reading-log.csv") as f:

Every bare open in text mode gets flagged with its line number. Run this once over a project, fix what it points at, and the class of bug in this chapter stops happening to you. (The flag is Python 3.10 and later. A future version will make UTF-8 the default everywhere and retire most of the problem, but you'll be reading code written before then for years.)

Practice

Try each of these before you read the solution under it.

1. Write read_text(path) that reads a file as UTF-8, and on UnicodeDecodeError retries with errors="replace" and prints a warning first. Return the text either way.
2. Write best_effort_decode(raw) that tries "utf-8", then "cp1252", then "latin-1", and returns the text from the first encoding that doesn't raise.
3. Take "Cien años de soledad", encode it to UTF-8 bytes, and show that string decoded as Latin-1 (the mojibake). Then turn the mojibake back into the original.

Solutions

1. Catch the specific error, fall back, and don't hide that you did.

def read_text(path):
    try:
        with open(path, encoding="utf-8") as f:
            return f.read()
    except UnicodeDecodeError:
        print(f"warning: {path} is not valid UTF-8, replacing bad bytes")
        with open(path, encoding="utf-8", errors="replace") as f:
            return f.read()

note = read_text("notes/piranesi.txt")
print(note.splitlines()[0])

Piranesi

2. Latin-1 maps all 256 byte values, so it never raises. That makes it the last resort: something always comes back, even if it's mojibake.

def best_effort_decode(raw):
    for encoding in ("utf-8", "cp1252", "latin-1"):
        try:
            return raw.decode(encoding)
        except UnicodeDecodeError:
            continue

print(best_effort_decode("Cien años de soledad".encode("utf-8")))
print(best_effort_decode(b"plain ascii"))

Cien años de soledad
plain ascii

3. Mojibake is reversible when you know both encodings: encode the wrong reading back to bytes with the wrong encoding, then decode those bytes correctly.

original = "Cien años de soledad"
utf8_bytes = original.encode("utf-8")

mojibake = utf8_bytes.decode("latin-1")
print(mojibake)

recovered = mojibake.encode("latin-1").decode("utf-8")
print(recovered)

Cien años de soledad
Cien años de soledad

Where this leaves you

Pass encoding="utf-8" every time you open a text file. You now know what that argument does, what breaks without it, how to read the crash when it comes, and how to salvage a file that was saved with the wrong encoding. From here on, every open in the book that touches text names its encoding, and so should yours. Chapter 5 moves from the contents of a file to the thing that points at it: the path.