Burrow-The-Burrows

A Gopher burrower in a shell script. By using burrow and a bit of plumbing you can get all the links in a Gopher MENU, recursively visit all the available subdirs, and create a directed graph of the visited selectors.

burrow takes as input a gopher identifier, as generated by url_to_id, which is considered a gophermap, and provides on stdout the list of menu selectors found in that document. burrow will also dump on stderr the list of all the edges (to any kind of selector) found in that page, in the format:

src_SHA256 dst_SHA256

where src_SHA256 is the SHA256 of the source selector (the current document), while dst_SHA256 is the destination selector (the pointed document).

To start a crawl, one can do something like:

	$ ./url_to_id  gopher://your.gopher.url/ > ids
	$ tail -f ids | parallel -j2 './burrow {}' 2>> graph.txt | tee -a ids >/dev/null &

Notice that burrow will create a certain number of folders in the current directory, used to keep track of the selectors that have been already retrieved.