diff options
| author | bnewbold <bnewbold@robocracy.org> | 2024-04-13 12:20:06 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-04-13 12:20:06 -0700 |
| commit | 58842d03a95af014cb44c3495d109e3bb6731fde (patch) | |
| tree | 08c27ccc28e05235e02440d9584788c78d654bc7 /bskyweb/embedr-static/robots.txt | |
| parent | 196dd3a8abdc4ebdd0a73c5f6afe2acca38d8efc (diff) | |
| download | voidsky-58842d03a95af014cb44c3495d109e3bb6731fde.tar.zst | |
rebased embedr (#3511)
* skeleton of embedr service, based on bskyweb * embedr container setup * builds on this branch * actual routes * fix embedr go:embed * tweak embedr dockerfile * progress on embedr * fix path params * tweaks to build process * try to get embedr dockerfile to install embed deps * build this branch * updates to match sam's output HTML * try to unbreak embedr dockerfile * small embedr tweak * docker hack * get embed.js copied over to embedr * don't x-frame-options for embed.bsky.app * bskyembed: remove a console.log * use html/template for golang snippet generation * simplify embedr API fetches * missing file * Rm console.log fully --------- Co-authored-by: Dan Abramov <dan.abramov@gmail.com>
Diffstat (limited to 'bskyweb/embedr-static/robots.txt')
| -rw-r--r-- | bskyweb/embedr-static/robots.txt | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/bskyweb/embedr-static/robots.txt b/bskyweb/embedr-static/robots.txt new file mode 100644 index 000000000..4f8510d18 --- /dev/null +++ b/bskyweb/embedr-static/robots.txt @@ -0,0 +1,9 @@ +# Hello Friends! +# If you are considering bulk or automated crawling, you may want to look in +# to our protocol (API), including a firehose of updates. See: https://atproto.com/ + +# By default, may crawl anything on this domain. HTTP 429 ("backoff") status +# codes are used for rate-limiting. Up to a handful concurrent requests should +# be ok. +User-Agent: * +Allow: / |