adrianshort
/
planningalerts
mirror of https://github.com/adrianshort/planningalerts.git
1
0
Fork 0
Code Issues 0 Releases 0 Wiki Activity
Browse Source

Add a robots.txt file to stop search engines bothering our scrapers 

and api.
import/raw
duncan.parkes 17 years ago
parent
277d029525
commit
ad6091868e
1 changed files with 4 additions and 0 deletions
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. +4
    -0
      trunk/docs/robots.txt

+ 4
- 0
trunk/docs/robots.txt View File

@@ -0,0 +1,4 @@
User-agent: *
Disallow: /cgi-bin/
Disallow: /scrapers/
Disallow: /api.php

Write Preview
Loading…
Cancel
Save