loopprep

Same-Host Web Crawler

total 00:00
Hard50 minPremium Asked at AnthropicOpenAI

    Crawl a site through a link provider that costs something every time you call it: collect the reachable pages, respect a page budget and a depth limit, survive a flaky provider, and answer shortest-path questions without fetching the whole site.

    The five steps
    1. Everything on this host 8 min
    2. A page budget 9 min
    3. How far from home 9 min
    4. A provider that fails 11 min
    5. How do I get there from here 11 min