diff options
Diffstat (limited to 'bskyweb/static')
| -rw-r--r-- | bskyweb/static/robots.txt | 10 |
1 files changed, 9 insertions, 1 deletions
diff --git a/bskyweb/static/robots.txt b/bskyweb/static/robots.txt index d3475984e..4f8510d18 100644 --- a/bskyweb/static/robots.txt +++ b/bskyweb/static/robots.txt @@ -1 +1,9 @@ -# hello friends! +# Hello Friends! +# If you are considering bulk or automated crawling, you may want to look in +# to our protocol (API), including a firehose of updates. See: https://atproto.com/ + +# By default, may crawl anything on this domain. HTTP 429 ("backoff") status +# codes are used for rate-limiting. Up to a handful concurrent requests should +# be ok. +User-Agent: * +Allow: / |