loopprep

Greedy Tokenizer

total 00:00
Hard45 minPremium Asked at Anthropic

    Scan text left to right emitting the longest matching vocabulary entry. Then expose the segmentation, make matching independent of vocabulary size, decode back to text, and collapse runs of unknown characters.

    The five steps
    1. Longest match wins 9 min
    2. Show the segmentation 7 min
    3. Stop scanning the vocabulary 12 min
    4. Decode 8 min
    5. Collapse unknown runs 9 min