Consider including OpenAerialMap /meta and STAC services in the index

[andrewharvey]

Integrating https://api.openaerialmap.org/meta was previously discussed in #509 and the OpenAerialMap Mosaic was added in #1649.

I've been thinking about providing a middle ground between "include all individual image sets in ELI" and "only include the mosaic in ELI" for imagery providers that allow it.

For example with OpenAerialMap we could include a single entry pointing to the endpoint https://api.openaerialmap.org/meta and specifying the protocol eg OAM_META used to query the index.

Another protocol would add is STAC https://stacspec.org/en/about/stac-spec/ used by LINZ.

ELI could even be implemented as a STAC service or OAM /meta service for clients who don't wish to download the whole ELI index and instead want to query and endpoint for imagery at the given location.

The benefits of including these in the index rather than in each editor means we keep the index central to all editors who choose to use it, continue to validate licensing and OSM compatibility for inclusion.

cc @grischard @simonpoole @tordans @cicku

[1ec5]

As more imagery becomes available, ELI needs to become an “catalog of catalogs”. Otherwise, the maintenance burden for some imagery sources is too high, so we don’t bother and OSM misses out.

A prominent example is Esri Wayback Imagery (#1364). It should’ve replaced Esri Clarity Beta by now (#2752), but we wouldn’t be able to maintain hundreds of layers (and counting) all by ourselves. Instead, Rapid and iD have their own custom implementations. OpenHistoricalMap/issues#806 tracks unifying these implementations, but other editors still wouldn’t benefit from them unless it comes from ELI. There’s also an increasing amount of catalogs and individual sources that use IIIF, which similarly requires working around ELI: OpenHistoricalMap/issues#768.

The OSMUS Imagery Working Group has discussed STAC frequently, since its first meetings. Originally it was an idea for replacing ELI outright with off-the-shelf STAC-compliant software, but I think folks are instead working on a different concept. These efforts shouldn’t stop us from collecting metadata about imagery catalogs in this repository. To maintain backwards compatibility, we could start a new catalogs/ folder alongside sources/ and fill it with JSON or GeoJSON files about the sources that come to mind. If a better solution arises, we can migrate the catalog metadata to it.

[andrewharvey]

These efforts shouldn’t stop us from collecting metadata about imagery catalogs in this repository. To maintain backwards compatibility, we could start a new catalogs/ folder alongside sources/ and fill it with JSON or GeoJSON files about the sources that come to mind. If a better solution arises, we can migrate the catalog metadata to it.

That sounds good. It's whether we process those catalogs into our build outputs, so downsteam users of ELI don't need to do anything special, or we leave it to them to fetch the layers from the catalog.

[k-yle]

linking to https://github.com/moregeo-it/stac-index (stacindex.org), which collects a list of STAC catalogs. unclear what license their database uses.

[simonpoole]

We already have a similar situation with WMS and WMTS server entries, we need to find a balance between providing easy/direct access to the most useful sources, and providing access to non-core and back catalogue entries potentially via an index provided by the service.

My main issue is that we can't guarantee that all contents of such a service are suitably licenced and we would have to rely on the applications checking the licence information, which I'm sure every single one does (not).

As a tendency I would find it more important to extend the schema to cover more ways tiles are being provisioned (pmtiles for example) and different tile contents, see my tile_type extension #1074

PS: and maybe fix some schema issues at the same time #1901

[1ec5]

My main issue is that we can't guarantee that all contents of such a service are suitably licenced and we would have to rely on the applications checking the licence information, which I'm sure every single one does (not).

We check the copyright status of a source at the time we put it into the index but don’t automate that in any way. It sounds like you’re concerned about potentially mixed-license catalogs? I agree that would be an issue, but couldn’t we establish a rule that a catalog can only be listed if it comes with such guarantees? This would essentially limit the catalog feature to single-publisher catalogs, which I think would be fine.

For example, the Esri Wayback catalog is completely licensed for use in open data projects like OSM, including any of the items within. If for some reason they were to add a new item to this catalog under different terms, it would be akin to one of the existing mosaic layers introducing different license terms for their imagery in Asia versus everywhere else, contradicting the metadata in this index. (I worry about that whenever the ostensible USGS topo layer adds coverage outside the U.S.)

[andrewharvey]

Yes we should only include catalogs where we are sure all the layers are and would be covered under a compatible license.

We already have a similar situation with WMS and WMTS server entries, we need to find a balance between providing easy/direct access to the most useful sources, and providing access to non-core and back catalogue entries potentially via an index provided by the service.

Perhaps we could have better categorisation of the sources into "core" and "secondary"? So editors could provide a different user experience for each?

Core sources would be kept minimal to the most likely to be used sources, determined by clarity, currency, accuracy, with secondary everything else that might be useful in some instances but not generally. We want mappers to swap between the core sources, eg. to check the most recent imagery to make sure thing are current, while swapping back to the imagery with the best accuracy for tracing. If the sources list is full of "secondary" sources users won't be able to find the best core ones.

The secondary sources could be hidden under a "more" list, although I guess the "historicalmap" category kind of does this already, and iD already excludes those sources.

[andrewharvey]

#2869 is a sketch of an OpenAerialMap catalogue

[simonpoole]

It is unclear to me what the point of the OAM entry would be, it isn't a protocol with widespread use, the only popular instance has been nearly abandoned half a dozen times, nothing about it is stable, and an entry would only be useful for apps that have implemented querying the OAM catalogue (and therefore already know about it). The last time I checked the API docs had little to nothing to do with what the API actually implemented, and in general HOT doesn't seem even a tiny bit interested in evolving a app ecosystem around OAM or committing to continued availability of the service.

This is all very different than STAC.

Just for the record: Vespucci has had OAM integration since February 2018.

[grischard]

I would agree with Simon on this. If OAM is a one-off format, the editors can add it themselves, or work together on an OAM to STAC converter.

[andrewharvey]

To me ELI also acts as a list of approved imagery sources, which only contains sources we can use for mapping, includes the documentation about why we can use it for mapping and the PR process provides checks to ensure we do a licensing check.

Including the OAM catalog means we include the usage documentation for the source and editor software know that since it's in the index it's safe to present to users. If editors implement sources on their own, that licensing check and documentation becomes more fragmented.

I don't think it matters that the API might not be considered stable, or that the service itself might not be. Editors incorporating it could decide what level of reliance to place on the API being stable.

This OAM catalog json entry doesn't need to do anything right now, it's just there to indicate that the catalog exists and is compatible for use in OSM mapping.

[andrewharvey]

Another catalog entry could be Esri World Imagery Wayback. If we decide it's not ideal to add each individual entry into the ELI as their own source, we should include it as a catalog.

We probably need an alternative distribution to the index.json/geojson/xml files that's geographically indexed, so editors can make a single request for imagery at a given geographic hash, and all the source layers including catalog items can be returned for the given location only.

[1ec5]

If editors implement sources on their own, that licensing check and documentation becomes more fragmented.

Multiple editors have already had to integrate Esri Wayback independently. This is inevitable given different languages and architectures, but it would be nice if the implementations could be more data-driven based on a shared source of metadata like ELI. As it is, the implementations are quite obscure. Any change to something as minor as an icon will need to be replicated across each implementation. Also, quality assurance such as reachability and license checking is out of scope for these client projects, whereas ELI even tries to automate these aspects to some extent.

We probably need an alternative distribution to the index.json/geojson/xml files that's geographically indexed, so editors can make a single request for imagery at a given geographic hash, and all the source layers including catalog items can be returned for the given location only.

For Esri Wayback, both geographic indexing and vintage lookup (#909) currently rely on an API that’s unique to this service. Rapid has integrated the API for more precise geographic indexing, whereas iD just assumes each layer is global, which is technically true. Esri Wayback is just a special case of the feed feature in ArcGIS, though I don’t know of any other ArcGIS feed that ELI would look to include.

We could nudge both OAM and Esri to provide a STAC-compatible endpoint, so that we don’t have to think about three different catalog types. In the meantime, the extra types would be less of a burden on ELI itself than on clients (who have the option of ignoring these types).

[simonpoole]

My other issue with including STAC catalogues (BTW see https://radiantearth.github.io/stac-browser/#/external/data.geo.admin.ch/api/stac/v1/ for an example of why I believe it is problematic, all of the sources there are "free to view" but good luck finding out if they could actually be used in/for OSM without specialist knowledge) is that it is literally just a catalogue providing no guarantees as to the individual collections/items being technically useful in a typical OSM context. What that might be is naturally a matter of debate, but it is likely not to include that ability to process multi-GB COGs in any of 100s of projections, as just a random example of why this might be of very limited usefulness.

As far as I can tell the only thing that would be immediately applicable to the typical OSM environment would be catalogues that provide data for the web-map-links extension of the standard https://github.com/stac-extensions/web-map-links, but does anybody do that?

PS: STAC plus web-map-links might be a way to provide ELI via an API without reinventing the wheel.

[andrewharvey]

My other issue with including STAC catalogues (BTW see https://radiantearth.github.io/stac-browser/#/external/data.geo.admin.ch/api/stac/v1/ for an example of why I believe it is problematic, all of the sources there are "free to view" but good luck finding out if they could actually be used in/for OSM without specialist knowledge) is that it is literally just a catalogue providing no guarantees as to the individual collections/items being technically useful in a typical OSM context.

We don't need to add every STAC endpoint in existence, only those where we want all the whole collection available to mappers and we are sure that all the items are compatible. For example LINZ aerial imagery.