Search notes:
robots.txt
Links
The Web Robots Pages
:
about robots.txt
The
robotstxt github repository
contains
Google's
robots.txt parser and matcher as a C++ library
Index