Pārlūkot izejas kodu

Rename env vars per Morph spec

Must start with MORPH_
master
Adrian Short pirms 6 gadiem
vecāks
revīzija
496fcaf391
2 mainītis faili ar 6 papildinājumiem un 6 dzēšanām
  1. +3
    -3
      README.md
  2. +3
    -3
      scraper.rb

+ 3
- 3
README.md Parādīt failu

@@ -25,9 +25,9 @@ According to the principle of _one codebase, many deploys_, this scraper is [con

|Name|Purpose|Default|Required?|
|------------------|-----------------------------------------|----------|
|SCRAPER_DELAY |Minimum delay in seconds between HTTP requests to the server.|10|No|
|SCRAPER_USER_AGENT|User agent string sent as an HTTP request header.|_None_|Yes|
|SCRAPER_LOG_LEVEL |Controls the level of detail in the output logs according to [Ruby's `Logger` class](https://ruby-doc.org/stdlib-2.1.0/libdoc/logger/rdoc/Logger.html) constants.|1 _(Logger::INFO)_|No|
|MORPH_DELAY |Minimum delay in seconds between HTTP requests to the server.|10|No|
|MORPH_USER_AGENT|User agent string sent as an HTTP request header.|_None_|Yes|
|MORPH_LOG_LEVEL |Controls the level of detail in the output logs according to [Ruby's `Logger` class](https://ruby-doc.org/stdlib-2.1.0/libdoc/logger/rdoc/Logger.html) constants.|1 _(Logger::INFO)_|No|

## Running



+ 3
- 3
scraper.rb Parādīt failu

@@ -17,13 +17,13 @@ def crawl_delay
sleep DELAY_S
end

DELAY_S = ENV['SCRAPER_DELAY'].to_f || 10 # seconds. Conservatively slow by default. Scrapes approx 360 pages per hour.
USER_AGENT = ENV['SCRAPER_USER_AGENT']
DELAY_S = ENV['MORPH_DELAY'].to_f || 10 # seconds. Conservatively slow by default. Scrapes approx 360 pages per hour.
USER_AGENT = ENV['MORPH_USER_AGENT']
DATE_REGEX = /\d{2}-\d{2}-\d{4}/

$stdout.sync = true # Flush output buffer after every write so log messages appear immediately.
logger = Logger.new($stdout)
logger.level = ENV['SCRAPER_LOG_LEVEL'].to_i || Logger::INFO
logger.level = ENV['MORPH_LOG_LEVEL'].to_i || Logger::INFO
logger.info "Scraper starts. Let's do this."
logger.info "Delay between requests is #{DELAY_S} seconds."
logger.info "User agent is: #{USER_AGENT}"


Notiek ielāde…
Atcelt
Saglabāt