Merton Council planning applications
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Adrian Short e0b63f5b72
Fix site URL
4 years ago
.gitignore Add scraper code 5 years ago
Gemfile Add scraper code 5 years ago
Gemfile.lock Add scraper code 5 years ago
README.md Configure the search query through env vars 5 years ago
parser.rb Add scraper code 5 years ago
scraper.rb Fix site URL 4 years ago

README.md

Merton Council planning applications scraper

This scrapes planning applications data from Merton Council’s planning database website and puts it in an SQLite database.

Merton Council runs Northgate Planning Explorer.

This scraper is designed to run once per 24 hours.

It runs on Morph. To get started see the documentation.

Schema

The schema is based on the core elements from planningalerts.org.au.

Installation

$ git clone https://github.com/adrianshort/merton-planning-applications.git
$ cd merton-planning-applications
$ bundle

Configuration

According to the principle of one codebase, many deploys, this scraper is configured using environment variables rather than by editing constants in the code.

| Name | Purpose | Default | Required? | | --- | --- | --- | | MORPH_DELAY | Minimum delay in seconds between HTTP requests to the server. | 10 | No | | MORPH_USER_AGENT | User agent string sent as an HTTP request header. | None | Yes | | MORPH_LOG_LEVEL | Controls the level of detail in the output logs according to Ruby’s Logger class constants. | 1 (Logger::INFO) | No | | MORPH_DAYS | | Number of days to scrape | Only if MORPH_MONTHS is unset | | MORPH_MONTHS | Number of months to scrape | None | Only if MORPH_DAYS is unset | | MORPH_STATUS | Only scrape applications with this status code. | None | No |

## Running

$ bundle exec ruby scraper.rb

Logging

Log messages are written unbuffered to STDOUT. You can redirect them to a file or the log drain of your choice.

$ bundle exec ruby scraper.rb >> log.txt

Morph.io will only show the first 10,000 lines of log output. This constraint doesn’t apply when running elsewhere, eg on your local machine.

Similar projects

Tags

  • Merton
  • Merton Council
  • London
  • UK
  • localgov
  • localgovdigital
  • opendata
  • Morph
  • ScraperWiki
  • planning
  • Planning Alerts
  • plantech
  • civictech

## Author

By Adrian Short.

This project is not by or affiliated with Merton Council.