Add a robots.txt file to stop search engines bothering our scrapers · ad6091868e - planningalerts - git.adrianshort.org

Add a robots.txt file to stop search engines bothering our scrapers

and api.

import/raw

duncan.parkes 17 years ago

parent

277d029525

commit

ad6091868e

1 changed files with 4 additions and 0 deletions

Unified View

							
								+ 4
								
								- 0
							
trunk/docs/robots.txtView File
								
																@@ -0,0 +1,4 @@
															
																User-agent: *
															
																Disallow: /cgi-bin/
															
																Disallow: /scrapers/
															
																Disallow: /api.php