Chapter 6

Finding Files

Chapter 5 gave you paths you already knew. This chapter is how you get paths you don't: everything in a folder, or every file whose name matches a pattern. "How do I list all the files in a directory" is the single most-asked file question in Python, so it's worth doing properly rather than copying the first answer you find.

What's in this folder: iterdir

from pathlib import Path

for entry in sorted(Path("notes").iterdir()):
    kind = "dir " if entry.is_dir() else "file"
    print(kind, entry.name)

Output

dir  archive
file klara-and-the-sun.txt
file one-hundred-years-of-solitude.txt
file piranesi.txt
file project-hail-mary.txt

.iterdir() yields a Path for every item directly inside the folder, files and subfolders alike, one level deep. It hands them back in whatever order the filesystem stored them, which is neither sorted nor stable between runs. Wrap it in sorted() whenever the order matters, which is most of the time.

Matching a pattern: glob

.glob(pattern) yields only the entries whose name matches a shell-style pattern:

from pathlib import Path

for p in sorted(Path("notes").glob("*.txt")):
    print(p.name)

Output

klara-and-the-sun.txt
one-hundred-years-of-solitude.txt
piranesi.txt
project-hail-mary.txt

The wildcards:

* matches any run of characters, including none
? matches exactly one character
[abc] matches one character from the set

So *.txt is "anything, then .txt", and p*.txt is "starts with p":

for p in sorted(Path("notes").glob("p*.txt")):
    print(p.name)

Output

piranesi.txt
project-hail-mary.txt

.glob only looks one level down. The note tucked away in notes/archive/ is not in either list above.

Going deeper: rglob

To search subfolders as well, use .rglob(pattern), "recursive glob":

from pathlib import Path

for p in sorted(Path("notes").rglob("*.txt")):
    print(p.as_posix())

Output

notes/archive/the-left-hand-of-darkness.txt
notes/klara-and-the-sun.txt
notes/one-hundred-years-of-solitude.txt
notes/piranesi.txt
notes/project-hail-mary.txt

.rglob("*.txt") is the same as .glob("**/*.txt"), where ** means "this folder and every folder beneath it." Use whichever one reads better to you.

Filtering and sorting

glob and rglob give you an iterator, not a list. That means you can only walk it once, and you get the normal tools for shaping it:

from pathlib import Path

notes = sorted(Path("notes").rglob("*.txt"))
print("total:", len(notes))

single_word = [p.stem for p in notes if "-" not in p.stem]
print("one-word titles:", single_word)

Output

total: 5
one-word titles: ['piranesi']

Which books have a note?

This is the payoff for the running example. Collect every note's stem, then walk the log and see which finished books are missing one:

from pathlib import Path

note_stems = {p.stem for p in Path("notes").rglob("*.txt")}

with open("reading-log.csv", encoding="utf-8") as f:
    next(f)
    for line in f:
        fields = line.rstrip("\n").split(",")
        title, finished = fields[0], fields[-1]
        if not finished:
            continue  # still reading
        stem = title.lower().replace(" ", "-")
        if stem not in note_stems:
            print("no note:", title)

Output

no note: "Rendezvous with Rama
no note: The Peripheral
no note: A Memory Called Empire

Three finished books with nothing written down. (The first title is mangled because split(",") still can't handle the comma inside it. Chapter 7 is the last you'll see of that.)

os.walk and Path.walk

Two other ways to cross a folder tree, worth knowing by name.

os.walk(top) is the old standby. It yields a (folder, subfolder_names, file_names) tuple for every folder in the tree, which lets you see the structure and skip whole branches, at the cost of more code. Path.walk() is the same idea in pathlib, added in Python 3.12.

For "find the files matching a pattern," rglob is shorter than both. Reach for walk when you care about the folder structure itself.

Common mistakes

Treating the result as a list. Path(".").glob("*.txt") is a generator. Loop it twice and the second loop is empty; print it and you get <generator object ...>. Call list() or sorted() on it first.

Expecting regex. Glob patterns are not regular expressions. * is not .*, ? does not mean "optional," and there are no groups. If you need real pattern matching on names, run re over p.name yourself.

rglob on a huge tree. Path.home().rglob("*") walks your entire home folder and can take a while. Point it at the folder you actually mean.

Case. Globbing is case-insensitive on Windows and case-sensitive on Linux and macOS. *.txt will not match NOTES.TXT on Linux.

Practice

Try each of these before you read the solution under it.

Write list_notes() that returns a sorted list of every note stem, current and archived, with no duplicates.
Write find_note(title) that returns the Path to a title's note whether it is current or archived, or None if there isn't one.
Print the finished books from the log that have no note anywhere. (You have the pieces from this chapter.)

Solutions

1. rglob already covers the archive, so one call does it.

from pathlib import Path

def list_notes():
    return sorted({p.stem for p in Path("notes").rglob("*.txt")})

for stem in list_notes():
    print(stem)

Output

klara-and-the-sun
one-hundred-years-of-solitude
piranesi
project-hail-mary
the-left-hand-of-darkness

2. Check the current folder first, then the archive, then give up.

from pathlib import Path

def find_note(title):
    stem = title.lower().replace(" ", "-")
    for folder in (Path("notes"), Path("notes/archive")):
        candidate = folder / f"{stem}.txt"
        if candidate.exists():
            return candidate
    return None

print(find_note("Piranesi").as_posix())
print(find_note("The Left Hand of Darkness").as_posix())
print(find_note("Dune"))

Output

notes/piranesi.txt
notes/archive/the-left-hand-of-darkness.txt
None

3. Reuse find_note. A book is finished when its last field is not empty.

from pathlib import Path

def find_note(title):
    stem = title.lower().replace(" ", "-")
    for folder in (Path("notes"), Path("notes/archive")):
        candidate = folder / f"{stem}.txt"
        if candidate.exists():
            return candidate
    return None

with open("reading-log.csv", encoding="utf-8") as f:
    next(f)
    for line in f:
        if line.rstrip("\n").endswith(","):
            continue  # still reading
        title = line.split(",")[0]
        if find_note(title) is None:
            print(title)

Output

"Rendezvous with Rama
The Peripheral
A Memory Called Empire

Where this leaves you

You can list a folder, match files by name pattern one level deep or all the way down, and shape the results with the usual tools. Combined with Chapter 5 you can now go from "somewhere in this folder tree" to a specific Path. Chapter 7 picks up the thread that has been dangling since Chapter 2: reading the reading log as actual columns, with the csv module.