Walk a directory tree and group files with identical contents, then make it safe for huge files, correct around symlinks, useful as a report, and configurable with ignore rules.

The five steps

Group files by content 10 min
Survive very large files 11 min
Do not chase symlinks 8 min
Turn it into a report 10 min
Ignore rules 9 min

You are given a starting directory. Find the files inside it that hold exactly the same bytes.

def find_duplicate_files(root_path: str) -> List[List[str]]

Rules:

Traverse the tree rooted at root_path, including nested subdirectories.
Group paths whose file contents are byte-for-byte identical.
Only return groups with at least two files. A file with unique contents is not reported.
Two empty files count as duplicates of each other.
Output is deterministic: each group is sorted, and the groups themselves are sorted by their first path.

# workspace/docs/spec.txt     -> "system design notes"
# workspace/archive/copy.txt  -> "system design notes"
# workspace/tmp/draft.txt     -> "draft v2"

find_duplicate_files("workspace")
# [["workspace/archive/copy.txt", "workspace/docs/spec.txt"]]

> Paths are built with os.path.join, so they use whatever separator the platform uses.

Duplicate File Finder