adrianshort
/
planningalerts
mirror of https://github.com/adrianshort/planningalerts.git
1
0
Fork 0
Code Issues 0 Releases 0 Wiki Activity
Browse Source

Add a robots.txt file to stop search engines bothering our scrapers 

and api.
master
duncan.parkes 17 years ago
parent
5ba4a625c7
commit
26d13fc88a
1 changed files with 4 additions and 0 deletions
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. +4
    -0
      docs/robots.txt

+ 4
- 0
docs/robots.txt View File

@@ -0,0 +1,4 @@
User-agent: *
Disallow: /cgi-bin/
Disallow: /scrapers/
Disallow: /api.php

Write Preview
Loading…
Cancel
Save