planningalerts/wiki/ScraperDevelopment.wiki

= Scrapers being worked on =

  * Bridgnorth (Duncan) - AcolNet. implemented, but returns error 400 when accessed by script
  * Cambridgeshire (Duncan) - AcolNet. no advanced search page - will need some work
  * Canterbury (Duncan) - AcolNet. results table is different from usual format
  * Guildford (Duncan) - AcolNet, but odd.
  * Isle of Man (Dan)
  * Surrey Heath (Duncan) - AcolNet - The problem here looks like it might be cookie and javascript related.
  * Wychavon (Duncan) - an AcolNet site, but slightly odd.

= Deployed by Duncan 07-08/10/2007 =

  * Bassetlaw
  * Bracknell Forest
  * Broads Authority
  * Bromley (changed this to my python scraper, which seems to be working)
  * Caradon
  * Chelmsford
  * Chiltern
  * Craven
  * Doncaster (now fixed, so enabled again)
  * Dundee
  * Eastleigh
  * Eden
  * Fenland
  * Gloucestershire (don't be fooled into thinking there are no applications: see eg 20/8/2007)
  * London Borough of Sutton

= Working and ready to deploy =
  * East Devon (db not empty - see eg 11/7/2007) - downloads empty cgi file
  * East Dorset - downloads empty cgi file
  * Gateshead (not timing out today) - downloads empty cgi file
  * Havant - not in SitesToGenerate.csv
  * Hinckley and Bosworth - downloads empty cgi file
  * Luton
  * Mansfield
  * Moray
  * Newcastle (not timing out today)
  * Newcastle-under-Lyme (not timing out today)
  * New Forest DC
  * North Hertfordshire (oops... I'd commented out the class it all depends on)
  * North Wiltshire (not timing out today)
  * Oldham
  * Perthshire
  * Renfrewshire
  * South Lakeland
  * Stratford
  * Suffolk Coastal
  * Teignbridge (This never times out for me... must investigate what goes wrong on the server)
  * Watford
  * Welwyn Hatfield
  * Winchester
  * Wyre Forest

Other things to do:
  * Balby needs renaming to Blaby.
  * Rename "District of Easington" to "Easington".
  * Rename "London Borough off Ealing" to "London Borough of Ealing".
  * Why does Hertsmere appear in the list of councils we've done twice?


= Scrapes done, but sites/scrapers broken =
  * Bury (script returns nothing)
  * Dacorum (always returns no applications)
  * Derby (no postcodes)
  * Durham (Duncan) - works in browser - empty results for script
  * Hambleton (Duncan) - works for browser - empty results for script
  * South Bedfordshire - timing out
  * Wolverhampton (empty database)