Fix download-osm and remove osmborder

[ZeLonewolf]

Fixes #461

This PR improves error handling in the download-osm script by replacing the deprecated asyncio.wait() function with asyncio.gather() and adding proper exception handling. It also updates the catalog search code and makes the Docker container work again.

Additionally, this PR completely removes osmborder, which is no longer used after openmaptiles/openmaptiles#1213

[TomPohys]

Thank you for the awesome work. Could you please test it with OpenMapTiles? I am getting some errors, e.g.

make -> bash: line 1: build/openmaptiles.tm2source/data.yml: No such file or directory

./quickstart.sh --empty / make download area=europe/albania -> PermissionError: [Errno 13] Permission denied: '/usr/src/app/cache'

Thank you!

[ZeLonewolf]

@TomPohys - fixed errors and now works as tested with OpenMapTiles & unit tests pass.
Generated Rhode Island extract with this PR:

I used the following docker config to test this from openmaptiles:

 #    image: "openmaptiles/openmaptiles-tools:${TOOLS_VERSION}"
    build:
      context: ../openmaptiles-tools
      dockerfile: Dockerfile

[nitrag]

@TomPohys can you please re-review?

[TomPohys]

Thank you for the update. Unfortunately, I am still getting an access error for /cache. I think that it could be solved by setting not /cache but /usr/src/app/cache as it is defined in IMPOSM_CACHE_DIR.

openmaptiles-tools/Dockerfile

Line 128 in 5a13cc1

RUN mkdir -p /cache /import /mapping /usr/src/app/data /usr/src/app/build/openmaptiles.tm2source

openmaptiles-tools/Dockerfile

Line 132 in 5a13cc1

RUN chmod 777 /cache /import /mapping /usr/src/app/data /usr/src/app/build/openmaptiles.tm2source

Also, during searching for the data source, I am getting an file_len=None, which raises an error and does not let break the loop (also, there is not self.is_guessing twice ).

openmaptiles-tools/bin/download-osm

Line 628 in 5a13cc1

if not self.is_guessing or src.file_len is not None or not self.is_guessing:

log

./quickstart.sh --empty
...
...
Downloading albania from any source...
Using docker compose V2 (docker compose)
Downloading albania into data/albania.osm.pbf from any source
Unable to auto-detect the source for 'albania'
DEBUG: AreaParamParser.parse called with area_id="albania"
DEBUG: Available sources: {list(self.sources.keys())}
DEBUG: is_guessing={self.is_guessing}
Area source has not been specified. Auto-detecting...
DEBUG: Trying source "url" of type UrlSrc
DEBUG: Source "url" search returned: None
DEBUG: Trying source "geofabrik" of type Geofabrik
DEBUG: Geofabrik.search called with area_id="albania", is_guessing=True
DEBUG: AreaSource.get_catalog called for {self.name}
DEBUG: Cache directory: {cache_dir}, list_path: {list_path}
DEBUG: No cached catalog found
Downloading the list of available Geofabrik areas...
DEBUG: Fetched catalog data ({len(data)} chars)
DEBUG: Cached catalog data to {list_path}
DEBUG: Parsed catalog with {len(catalog)} entries, {len(warnings)} warnings
DEBUG: Catalog loaded with {len(catalog)} entries
DEBUG: Regex search for "albania" found 1 matches: ['europe/albania']
DEBUG: Single match found: "europe/albania"

DEBUG: Source "geofabrik" search returned: SearchResult(source=Source(name='europe/albania', url='https://download.geofabrik.de/europe/albania-latest.osm.pbf', timestamp=None, mirror=None, url_hash='https://download.geofabrik.de/europe/albania-latest.osm.pbf.md5', hash=None, file_len=None), repl_url='https://download.geofabrik.de/europe/albania-updates', state_url='https://download.geofabrik.de/europe/albania.state.txt', bbox='18.89648,39.62738,21.06285,42.66313')

DEBUG: Loading source metadata for europe/albania from https://download.geofabrik.de/europe/albania-latest.osm.pbf
DEBUG: Making HEAD request to {self.url}
DEBUG: HEAD response status: {resp.status}
DEBUG: HEAD response headers: {dict(resp.headers)}
DEBUG: No Content-Length header found

DEBUG: Source metadata loaded - file_len=None, hash=1da832dcb9442e9df2240eef3ba7f594

DEBUG: Skipping source "geofabrik" - in guessing mode but no file metadata

DEBUG: Trying source "bbbike" of type Bbbike
DEBUG: AreaSource.get_catalog called for {self.name}
DEBUG: Cache directory: {cache_dir}, list_path: {list_path}
DEBUG: No cached catalog found
Downloading the list of available Bbbike areas...
DEBUG: Fetched catalog data ({len(data)} chars)
DEBUG: Cached catalog data to {list_path}
DEBUG: Parsed catalog with {len(catalog)} entries, {len(warnings)} warnings
DEBUG: Source "bbbike" search returned: None
DEBUG: Trying source "osmfr" of type Osmfr
DEBUG: Source "osmfr" search returned: None
DEBUG: No sources succeeded, raising SystemExit

make: *** [Makefile:373: download] Error 1

Exclude the if not self.is_guessing or src.file_len: condition, the quickstart runs OK.

Thank you for looking into it!

[ZeLonewolf]

Thanks @TomPohys. Can you describe your test approach so that I can make sure I'm replicating it correctly?

[TomPohys]

Sorry, I should have written it much earlier.

I adjust the __init__.py file e.g. on __version__ = '7.2.0'

openmaptiles-tools/openmaptiles/__init__.py

Line 1 in a4ba8bc

__version__ = '0.0.0'

Then I built dockers by:

• make build-docker
• make build-import-data
• make build-generate-vectortiles
• make build-postgis

Then I edit the .env in OMT to match the version in __init__.py (7.2.0).

And then run classic tasks in OMT (quickstart, manual walkthrough, tileserver with OSM style,...)

I do not use make build-postgis-preloaded as it is match to the "latest", and just running ./quickstart.sh --empty which import data from import-data image.

I am using classic Ubuntu 24.04 with Docker v28.4.0 and Docker Compose v2.39.2 and with postinstall Linux steps

[ZeLonewolf]

@TomPohys:

Using your method, I can't figure out how to get the built docker containers to be recognized by openmaptiles:

This is starting from a clean docker slate (all image purged) and then build in openmaptiles tools with make build-docker && make build-import-data && make build-generate-vectortiles && make build-postgis.

Do you have any suggestions?

[TomPohys]

I just tested with 7.3.0 (in OMT-T I used commit d3f24b1 with only change in __init__.py)

I built the omt-t with your command (`make build-docker && make build-import-data && ...). It was built without errors.

$ docker  images | grep openmaptiles
openmaptiles/generate-vectortiles   7.3.0     c6f6fc61ea32   9 seconds ago   295MB
openmaptiles/openmaptiles-tools     7.3.0     ef3e3bf4b563   6 minutes ago   1.47GB
openmaptiles/openmaptiles-tools     7.2.0     5039c11165c5   12 hours ago    1.47GB
openmaptiles/import-data            7.2.0     67dc38c4156a   23 hours ago    1.79GB
openmaptiles/import-data            7.3.0     67dc38c4156a   23 hours ago    1.79GB
openmaptiles/postgis                7.2.0     dfea03037e7d   23 hours ago    1.05GB
openmaptiles/postgis                7.3.0     dfea03037e7d   23 hours ago    1.05GB

and my only change in OMT is in .env

Ohh.. I just saw, that you are using Docker compose v1.

But during the quickstart it is also raised an error, but it ignores it and it is continue.

[+] Pulling 4/4
 ✘ generate-vectortiles Error manifest for openmaptiles/generate-vectortiles:7.3.0 not found: manifest unknown: manifest unknow...                             1.3s 
 ✘ openmaptiles-tools Error   manifest for openmaptiles/openmaptiles-tools:7.3.0 not found: manifest unknown: manifest unknown                                 1.3s 
 ✘ postgres Error             manifest for openmaptiles/postgis:7.3.0 not found: manifest unknown: manifest unknown                                            1.3s 
 ✘ import-data Error          manifest for openmaptiles/import-data:7.3.0 not found: manifest unknown: manifest unknown                                        1.3s 
 

But you could try $ NO_REFRESH=1 ./quickstart.sh --empty to skip trying update docker images from docker hub.

[ZeLonewolf]

@TomPohys thanks for the testing tips. I was able to fix the two issues you noted and this should be good to go now!

[TomPohys]

Thanks for the great work. I tested it, and after the install unzip it will be ready to be merged. Then I will release a new version of OMT-T and update the OMT.

[ZeLonewolf]

I have not tested the addition of unzip yet, but it seems low-risk to add it.

[ZeLonewolf]

Tested and confirmed this works! Good to merge.

[TomPohys]

🚀 Awesome! Thank you!