Merton Council planning applications scraper

This scrapes planning applications data from Merton Council’s planning database website and puts it in an SQLite database.

Merton Council runs Northgate Planning Explorer.

This scraper is designed to run once per 24 hours.

It runs on Morph. To get started see the documentation.

Schema

The schema is based on the core elements from planningalerts.org.au.

Installation

$ git clone https://github.com/adrianshort/merton-planning-applications.git
$ cd merton-planning-applications
$ bundle

Configuration

According to the principle of one codebase, many deploys, this scraper is configured using environment variables rather than by editing constants in the code.

| Name | Purpose | Default | Required? | | --- | --- | --- | | MORPH_DELAY | Minimum delay in seconds between HTTP requests to the server. | 10 | No | | MORPH_USER_AGENT | User agent string sent as an HTTP request header. | None | Yes | | MORPH_LOG_LEVEL | Controls the level of detail in the output logs according to Ruby’s Logger class constants. | 1 (Logger::INFO) | No | | MORPH_DAYS | | Number of days to scrape | Only if MORPH_MONTHS is unset | | MORPH_MONTHS | Number of months to scrape | None | Only if MORPH_DAYS is unset | | MORPH_STATUS | Only scrape applications with this status code. | None | No |

## Running

$ bundle exec ruby scraper.rb

Logging

Log messages are written unbuffered to STDOUT. You can redirect them to a file or the log drain of your choice.

$ bundle exec ruby scraper.rb >> log.txt

Morph.io will only show the first 10,000 lines of log output. This constraint doesn’t apply when running elsewhere, eg on your local machine.

Similar projects

• maxharlow/scrape-planning-northgate (Node)
• adrianshort/planningalerts, especially the Python scrapers for Northgate Planning Explorer - not by me, just a copy of this project’s codebase

Tags

• Merton
• Merton Council
• London
• UK
• localgov
• localgovdigital
• opendata
• Morph
• ScraperWiki
• planning
• Planning Alerts
• plantech
• civictech

## Author

By Adrian Short.

This project is not by or affiliated with Merton Council.