SubprocessorBot
If you found this in your server logs, this page explains what we were doing and how to stop us.
What it does
It looks for the page where your company publishes its sub-processor list — the suppliers you use to handle your customers' data. That list is a disclosure most companies are required to make and to keep current, and we build a dated, public record of what each one said and when it changed.
We read the same page any visitor can. Nothing behind a login, nothing that isn't linked, and nothing that isn't already published.
How it identifies itself
SubprocessorBot/1.0 (+https://subprocessor.org/bot)
How to stop it
Put this in your robots.txt and we will not read your site again. We honour it including where it costs us the page.
User-agent: SubprocessorBot Disallow: /
If we have already published a page about your company and you want it taken down or corrected, use the form on that page. Every request is recorded whether or not we agree with it, and you will hear back.
How often, and how hard
- One request at a time per site, with a gap between them — never a burst.
- Once a week for most companies. Daily only where somebody is relying on that page being current.
- A 429 or 503 stands us down for an hour, or for whatever your Retry-After says.
- Repeated failures back a site off rather than retrying it nightly forever.
Who to ask
Write to [email protected] and a person will answer. How the whole thing works covers the method and what we will not claim.