About

Our crawler

If you run a site we visit, this page tells you exactly what we do and how to make us stop.

How to recognise us

worldbody.org/0.1 (+https://worldbody.org/about/crawler; [email protected])

What we do

• We read robots.txt before anything else on a host, and we honour Disallow with full wildcard semantics — we do not widen a narrow rule into a broad one.
• We honour Crawl-delay. Where a publisher states one, it overrides our own configured rate, always downwards.
• We honour Retry-After on 429 and 503, and back off exponentially on top of it.
• We send conditional requests, so an unchanged page costs you a 304.
• Each source has a hard daily request cap enforced in code, claimed before the request is made rather than counted after it.
• For security posture we make one GET per host per cycle. That is a page view, not a scan.

What we never do

• We do not scan, probe, fuzz, or test any host for vulnerabilities.
• We do not enumerate object storage or fetch objects from it.
• We do not attempt to reach anything that requires authentication, and we do not work around an access control that answers 403.
• We do not republish images. Where we read image metadata, we record the metadata fields and discard the file.

How to stop us

Disallow our user-agent string in your robots.txt and we will stop on our next pass. If you need us to stop sooner, or to remove something already published, write to [email protected] and we will act on it without argument.