About
Our crawler
If you run a site we visit, this page tells you exactly what we do and how to make us stop.
How to recognise us
worldbody.org/0.1 (+https://worldbody.org/about/crawler; [email protected])
What we do
- We read
robots.txtbefore anything else on a host, and we honourDisallowwith full wildcard semantics — we do not widen a narrow rule into a broad one. - We honour
Crawl-delay. Where a publisher states one, it overrides our own configured rate, always downwards. - We honour
Retry-Afteron 429 and 503, and back off exponentially on top of it. - We send conditional requests, so an unchanged page costs you a 304.
- Each source has a hard daily request cap enforced in code, claimed before the request is made rather than counted after it.
- For security posture we make one GET per host per cycle. That is a page view, not a scan.
What we never do
- We do not scan, probe, fuzz, or test any host for vulnerabilities.
- We do not enumerate object storage or fetch objects from it.
- We do not attempt to reach anything that requires authentication, and we do not work around an access control that answers 403.
- We do not republish images. Where we read image metadata, we record the metadata fields and discard the file.
How to stop us
Disallow our user-agent string in your robots.txt and we will
stop on our next pass. If you need us to stop sooner, or to remove something
already published, write to
[email protected]
and we will act on it without argument.
Where this page comes from
- Certificate Transparency logs — First-party measurement (Data Torturing Solutions) · source
- European Parliament Open Data Portal — European Commission reuse notice (Decision 2011/833/EU) · source
- HTTP response headers — First-party measurement (Data Torturing Solutions)
- Machine-written summary of public-domain sources — First-party measurement (Data Torturing Solutions) · source
- Permanent Missions to the United Nations (the Blue Book) — UN document (public domain, ST/AI/189/Add.9/Rev.2) · source
- Public DNS — First-party measurement (Data Torturing Solutions)
- Public object-store index — Third-party index (facts only, no redistribution) · source
- Published image metadata — First-party measurement (Data Torturing Solutions)
- SERP API (Serper) — Third-party index (facts only, no redistribution) · source
- Third-party page found by search — Unassessed
- UN General Assembly ideal point estimates — CC0 1.0 · source
- UN General Debate Corpus (Jankin, Baturo & Dasandi) — CC0 1.0 · source
- UN Meetings Coverage and Press Releases — UN document (public domain, ST/AI/189/Add.9/Rev.2) · source
- UN Peace and Security Data Hub — un.org website terms (all rights reserved) · source
- UN Research Guides (Dag Hammarskjold Library) — un.org website terms (all rights reserved) · source
- UN Statistics Division SDG Indicators API — CC BY 4.0 · source
- UN World Population Prospects — un.org website terms (all rights reserved) · source
- UNHCR Refugee Data Finder — Unassessed · source
- United Nations Digital Library — UN document (public domain, ST/AI/189/Add.9/Rev.2) · source
- United Nations Security Council Consolidated List — UN document (public domain, ST/AI/189/Add.9/Rev.2) · source
- United Nations Treaty Collection — UN document (public domain, ST/AI/189/Add.9/Rev.2) · source
- United Nations website — un.org website terms (all rights reserved) · source
- Wikidata — CC0 1.0 · source
- Wikimedia Commons — Unassessed · source
-
Wikipedia — CC BY-SA 4.0 · source
This page reproduces text under CC BY-SA 4.0 and is itself available under the same licence. - World Bank Open Data — CC BY 4.0 · source
- worldbody.org (derived from the corpus) — First-party measurement (Data Torturing Solutions) · source