There is a lot of repeated code across each county's scraper and parser scripts. And lots of code that should be repeated but isn't (like retrying on transient network failures). Additionally, there are a handful of counties which use shared systems (e.g., https://www.mptsweb.net/).
Ideally, a county script would instantiate a class with a few variables (CSV file location, URL template, etc) and define a parse_html() function and call a method which takes care of everything else.
I'm working on this as part of Placer (#17). I'm creating this issue to track and discuss the work.
@typpo One question I have so far is related to my Placer work. You recommend the geojson script step. It seems like it'd be easier to do this in python (with, e.g., pyshp) to minimize the number of steps that someone has to follow. Have you found that the geojson script is better for one reason or another?
There is a lot of repeated code across each county's scraper and parser scripts. And lots of code that should be repeated but isn't (like retrying on transient network failures). Additionally, there are a handful of counties which use shared systems (e.g., https://www.mptsweb.net/).
Ideally, a county script would instantiate a class with a few variables (CSV file location, URL template, etc) and define a parse_html() function and call a method which takes care of everything else.
I'm working on this as part of Placer (#17). I'm creating this issue to track and discuss the work.
@typpo One question I have so far is related to my Placer work. You recommend the geojson script step. It seems like it'd be easier to do this in python (with, e.g., pyshp) to minimize the number of steps that someone has to follow. Have you found that the geojson script is better for one reason or another?