The database table which says which scrapers are where will now be filled in automatically, 
which should result in rather less in the way of manual editing errors.

I've also redone the python PublicAccess scraper and set all the PublicAccess sites to use 
it (removing the PHP PublicAccess scrapers).

Make display method on a planningapplication work out the postcode if it isn't set.

Factor out CookieAddingHTTPRedirectHandler.

back to them once you have the cookie, you're fine...

Alter PlanningUtils to CDATA everything, scrapping the xmlquote function.

accessible with a referer.

Also adding the pycurl scraper for Westminster, just in case it is useful to remind us how to do stuff later.

The info url situation here is not really good enough. 
All we get is a page with the last 7 days apps on it with no info urls.
I'm using that page as the info url for the moment, but it will obviously
be no use after seven days...

have had to give the search url in all the info urls. Comment url works fine for each app.

Add scrapers for Merton and Conwy (PlanningExplorer)

This site is absolutely rubbish: 

1) It's impossible to link to the info page - needs a cookie. I've put a link to the search page for the moment.
2) Many applications have no description. I've put the text "Description Missing" when this happens.

we don't have postcodes.

Encode all output as UTF-8.

We still need to sort out comment urls here.

We already have scrapers for Islington and East Herts, but I'm adding the new ones as well - we can then see if we 
get the same results :-)

* Blackburn
    * Camden - done (has OSGB)
    * Charnwood (done, but site has a problem with its info links)
    * Crewe and Nantwich
    * East Staffordshire
    * Epping Forest
    * Hackney - done
    * Kennet
    * Lincoln (also has WAM)
    * Liverpool
    * South Shropshire
    * South Tyneside
    * Stockport
    * Tamworth

I've included Camden, as the scraper works (their site is down today).
The scraper for Charnwood is fine (both the search results page and the info links fail to display in a browser, but 
we are getting all the info out fine, and the links are correct. This appears to be a missing template on their 
site).

I've included several authorities for which the scrapes work fine, but which don't have any postcodes. Some of these 
have OSGB coords for each app, which we could maybe use instead.

    * Broadland (no postcodes)
    * Forest Heath (no postcodes)
    * South Norfolk (appears to have WAM as well) (No postcodes - but has OSGB coords)
    * Trafford (no postcodes)
    * West Oxfordshire (no postcodes)

There are a few PlanningExplorer sites that are currently down, and which I haven't been able to do yet:

    * Merton (site down)
    * Shrewsbury (site is broken)
    * Swansea (site is broken)
    * Waltham Forest (currently giving an error)

This includes:
Cumbria
Lincolnshire
West Sussex

It should also include:
Dorset
Somerset

but they are both down.

All of these sites are county councils which are only responsible for a small number of apps, many of which are not 
at sites which have postcodes, so don't expect too much from these scrapers!