A Ruby gem to get planning applications data from UK council websites.
Du kannst nicht mehr als 25 Themen auswählen Themen müssen entweder mit einem Buchstaben oder einer Ziffer beginnen. Sie können Bindestriche („-“) enthalten und bis zu 35 Zeichen lang sein.

vor 6 Jahren
vor 6 Jahren
vor 6 Jahren
vor 6 Jahren
vor 6 Jahren
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970
  1. # UK Planning Scraper
  2. **PRE-ALPHA: Only works with Idox and Northgate sites and spews a lot of stuff to STDOUT. Not for production use.**
  3. This gem scrapes planning applications data from UK council/local planning authority websites, eg Westminster City Council. Data is returned as an array of hashes, one hash for each planning application.
  4. This scraper gem doesn't use a database. Storing the output is up to you. It's just a convenient way to get the data.
  5. Currently this only works for some Idox sites. The ultimate aim is to provide a consistent interface in a single gem for all variants of all planning systems: Idox Public Access, Northgate Planning Explorer, OcellaWeb, and all the one-off systems.
  6. This project is not affiliated with any organisation.
  7. ## Installation
  8. Add this line to your application's Gemfile:
  9. ```ruby
  10. gem 'uk_planning_scraper', :git => 'https://github.com/adrianshort/uk_planning_scraper/'
  11. ```
  12. And then execute:
  13. $ bundle install
  14. Or install it yourself as:
  15. $ gem install specific_install
  16. $ gem specific_install adrianshort/uk_planning_scraper
  17. ## Usage
  18. ```ruby
  19. require 'uk_planning_scraper'
  20. require 'date'
  21. require 'pp'
  22. # change this to the URL of the advanced search page for the council you want
  23. url = 'https://planning.anytown.gov.uk/online-applications/search.do?action=advanced'
  24. options = {
  25. delay: 10, # seconds between scrape requests; optional, defaults to 10
  26. }
  27. params = {
  28. validated_from: Date.today - 30, # Must be a Date object; optional
  29. validated_to: Date.today, # Must be a Date object; optional
  30. description: 'keywords to search for', # Optional
  31. }
  32. apps = UKPlanningScraper.search(url, params, options)
  33. pp apps
  34. ```
  35. Try [ScraperWiki](https://github.com/openaustralia/scraperwiki-ruby) if you want a quick and easy way to throw the results into an SQLite database:
  36. ```ruby
  37. require 'scraperwiki' # Must be installed, of course
  38. ScraperWiki.save_sqlite([:council_reference], apps)
  39. ```
  40. ## Development
  41. After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
  42. To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
  43. ## Contributing
  44. Bug reports and pull requests are welcome on GitHub at https://github.com/adrianshort/uk_planning_scraper.