planningalerts/wiki/ScraperDevelopment.wiki

= Scrapers being worked on =

  * Bridgnorth (Duncan) - AcolNet. implemented, but returns error 400 when accessed by script
  * Cambridgeshire (Duncan) - AcolNet. no advanced search page - will need some work
  * Canterbury (Duncan) - AcolNet. results table is different from usual format
  * Guildford (Duncan) - AcolNet, but odd.
  * Isle of Man (Dan)
  * Surrey Heath (Duncan) - AcolNet - The problem here looks like it might be cookie and javascript related.
  * Wychavon (Duncan) - an AcolNet site, but slightly odd.

= Deployed by Duncan 07-08/10/2007 =

  * Bassetlaw
  * Bracknell Forest
  * Broads Authority
  * Bromley (changed this to the python scraper, which seems to be working)
  * Caradon
  * Chelmsford
  * Chiltern
  * Craven
  * Doncaster (now fixed, so enabled again)
  * Dundee
  * Eastleigh
  * Eden
  * Fenland
  * Gloucestershire (don't be fooled into thinking there are no applications: see eg 20/8/2007)
  * London Borough of Sutton
  * Mansfield
  * New Forest DC
  * North Wiltshire
  * Oldham
  * South Lakeland
  * Watford
  * Welwyn Hatfield
  * Wyre Forest

Other stuff:
  * Renamed "District of Easington" to "Easington District Council"
  * Renamed "London Borough off Ealing" to "London Borough of Ealing".
  * Renamed Balby to Blaby.
  * Removed duplicate of Hertsmere with name "East Hertsmere".


= Working and ready to deploy =
  * East Devon (db not empty - see eg 11/7/2007) - downloads empty cgi file
  * East Dorset - downloads empty cgi file
  * Gateshead (not timing out today) - downloads empty cgi file
  * Havant - not in SitesToGenerate.csv
  * Hinckley and Bosworth - downloads empty cgi file
  * Luton - downloads empty cgi file
  * Moray - downloads empty cgi file
  * Newcastle (not timing out today) - downloads empty cgi file
  * Newcastle-under-Lyme (not timing out today) - downloads empty cgi file
  * Perthshire - downloads empty cgi file
  * Renfrewshire - downloads empty cgi file
  * South Bedfordshire - downloads empty cgi file
  * Stratford - internal server error
  * Teignbridge (This never times out for me... must investigate what goes wrong on the server) - downloads empty cgi file
  * Winchester - downloads empty cgi file

= Scrapes done, but sites/scrapers broken =
  * Bury (script returns nothing)
  * Dacorum (always returns no applications)
  * Derby (no postcodes)
  * Durham (Duncan) - works in browser - empty results for script
  * Hambleton (Duncan) - works for browser - empty results for script
  * South Bedfordshire - timing out
  * Wolverhampton (empty database)