Add a robots.txt file to stop search engines bothering our scrapers

and api.

docs/robots.txt

@@ -0,0 +1,4 @@
+User-agent: *
+Disallow: /cgi-bin/
+Disallow: /scrapers/
+Disallow: /api.php