A Ruby gem to get planning applications data from UK council websites.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

README.md 2.5 KiB

6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970
  1. # UK Planning Scraper
  2. **PRE-ALPHA: Only works with Idox and Northgate sites and spews a lot of stuff to STDOUT. Not for production use.**
  3. This gem scrapes planning applications data from UK council/local planning authority websites, eg Westminster City Council. Data is returned as an array of hashes, one hash for each planning application.
  4. This scraper gem doesn't use a database. Storing the output is up to you. It's just a convenient way to get the data.
  5. Currently this only works for some Idox sites. The ultimate aim is to provide a consistent interface in a single gem for all variants of all planning systems: Idox Public Access, Northgate Planning Explorer, OcellaWeb, and all the one-off systems.
  6. This project is not affiliated with any organisation.
  7. ## Installation
  8. Add this line to your application's Gemfile:
  9. ```ruby
  10. gem 'uk_planning_scraper', :git => 'https://github.com/adrianshort/uk_planning_scraper/'
  11. ```
  12. And then execute:
  13. $ bundle install
  14. Or install it yourself as:
  15. $ gem install specific_install
  16. $ gem specific_install adrianshort/uk_planning_scraper
  17. ## Usage
  18. ```ruby
  19. require 'uk_planning_scraper'
  20. require 'date'
  21. require 'pp'
  22. # change this to the URL of the advanced search page for the council you want
  23. url = 'https://planning.anytown.gov.uk/online-applications/search.do?action=advanced'
  24. options = {
  25. delay: 10, # seconds between scrape requests; optional, defaults to 10
  26. }
  27. params = {
  28. validated_from: Date.today - 30, # Must be a Date object; optional
  29. validated_to: Date.today, # Must be a Date object; optional
  30. description: 'keywords to search for', # Optional
  31. }
  32. apps = UKPlanningScraper.search(url, params, options)
  33. pp apps
  34. ```
  35. Try [ScraperWiki](https://github.com/openaustralia/scraperwiki-ruby) if you want a quick and easy way to throw the results into an SQLite database:
  36. ```ruby
  37. require 'scraperwiki' # Must be installed, of course
  38. ScraperWiki.save_sqlite([:council_reference], apps)
  39. ```
  40. ## Development
  41. After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
  42. To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
  43. ## Contributing
  44. Bug reports and pull requests are welcome on GitHub at https://github.com/adrianshort/uk_planning_scraper.