katolaz
/
cgit-70
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
a gopher interface forked from the popular cgit
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1228 Commits
3 Branches
43 Tags
1.9 MiB
Tag: Branch: Tree: 36b1d78923
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '36b1d78923'
${ noResults }
cgit-70/robots.txt

4 lines
47 B
Raw Normal View History Unescape

robots.txt: disallow access to snapshots My dmesg is filled with the oom killer bringing down processes while the Bingbot downloads every snapshot for every commit of the Linux kernel in tar.xz format. Sure, I should be running with memory limits, and now I'm using cgroups, but a more general solution is to prevent crawlers from wasting resources like that in the first place. Suggested-by: Natanael Copa <ncopa@alpinelinux.org> Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
12 years ago
User-agent: *
Disallow: /*/snapshot/*
Allow: /