Scrapes planning applications data for InLinkUK from BT kiosks.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

34 regels
689 B

  1. require 'uk_planning_scraper'
  2. require 'scraperwiki'
  3. require 'date'
  4. require 'time'
  5. require 'csv'
  6. councils = []
  7. CSV.foreach('councils.csv') do |line|
  8. councils << { name: line[0], url: line[1] } unless line[0][0] == '#'
  9. end
  10. params = {
  11. validated_from: Date.today - ENV['MORPH_DAYS'].to_i,
  12. validated_to: Date.today,
  13. description: 'inlink',
  14. }
  15. councils.each do |council|
  16. apps = UKPlanningScraper.search(council[:url], params)
  17. apps.map! do |app|
  18. app.merge(
  19. {
  20. la_name: council[:name],
  21. updated_at: Time.now
  22. }
  23. )
  24. end
  25. ScraperWiki.save_sqlite([:council_reference, :la_name], apps, 'applications')
  26. puts "#{council[:name]}: #{apps.size}"
  27. end