diff options
Diffstat (limited to 'bskyweb/embedr-static/robots.txt')
| -rw-r--r-- | bskyweb/embedr-static/robots.txt | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/bskyweb/embedr-static/robots.txt b/bskyweb/embedr-static/robots.txt new file mode 100644 index 000000000..4f8510d18 --- /dev/null +++ b/bskyweb/embedr-static/robots.txt @@ -0,0 +1,9 @@ +# Hello Friends! +# If you are considering bulk or automated crawling, you may want to look in +# to our protocol (API), including a firehose of updates. See: https://atproto.com/ + +# By default, may crawl anything on this domain. HTTP 429 ("backoff") status +# codes are used for rate-limiting. Up to a handful concurrent requests should +# be ok. +User-Agent: * +Allow: / |