Helper to split data into limited OSM `user/preferences` "tags"

[tordans]

Context

I have a couple of things that I would like to nudge forward related to shared settings between editors. I wrote about this here https://github.com/tordans/editor-settings-schema

This idea is stuck ATM due to the technical limitations that the user/preferences endpoint has in terms of character limits.

The PR openstreetmap/openstreetmap-website#6399 would resolve this, but so far there was no indication that this might get merged.

There are a couples of apps that could use this data store to share data, some are listed in openstreetmap/openstreetmap-website#6297

Idea

The idea now is, not to wait for the changes to the API but instead work around it with a standard to split the data into the smaller segments that we now have. Ideally, this would happen in a library like this so users can use the same logic easily. But it needs to be simple enough to be re-build in Swift etc. for other apps.

In https://github.com/tordans/editor-settings-schema?tab=readme-ov-file#osm-api my idea was use the API in a array-like manner. However that can still cause errors of too long strings (in both cases, the longer limits as well as today).

So a different idea is to JSON-stringify the string and then split it ab 200 chars (bytes?) and then put it together again to parse it.
We could go further and zip the data or something like that but this will make it harder for other languages to re-implement the logic.

---

Any ides on this?
And is this something that could happen in this project? Or should I create a separate packages for it?

[k-yle]

it's unfortunate that we need such a complex solution. Is there any reason why openstreetmap/openstreetmap-website#6399 is only increasing it to 10k instead of 1 million or 10 million?

---

assuming that there is a need to store content >10000 characters, i have a few thoughts on the JSON idea.

For the rest of this comment, lets assume the limit is 9 characters, not 255 or 10000. An example json object would be split into 3 "chunks" like this:

ESS:*:0:   {"abc":12
ESS:*:1:   3,"xyz":4
ESS:*:2:   56}

and your proposed keys would be ESS:*:0, ESS:*:1, and ESS:*:2.

problem

If one chunk fails to save, then the entire data is corrupted.

lets say we change "xyz": 456 to "xyz": "hello". We need to write to 2 of the 3 chunks:

ESS:*:0:   {"abc":12        # no change
ESS:*:1:   3,"xyz":"        # changed
ESS:*:2:   hello"}          # changed

If 1 of the 2 write operations fails, then the JSON structure will have a syntax error, because it's in a half-updated state. Therefore all the data is corrupted.

• Failed fetch requests are common when using a phone in a moving vehicle , e.g. in a bus, u-bahn, etc. where cellphone service drops out and recovers within a few seconds.
• It could also happen when the OSM API is overloaded and rejects every second request. this has happened every day this week.

JSON is a non-concatenative language, so there's no easy way to optimise this.

suggestion

since the index approach violations most of the ACID principles, i would suggest slightly different logic:

• Instead of using numeric indexs for your keys, use a short hash of the chunk's content.
• Introduce a special key like :root which contains metadata about the chunks.

The same example in this proposed format:

ESS:*:root:       f6ffd6b3,e4e20734,5c12b5b8
ESS:*:f6ffd6b3:   {"abc":12
ESS:*:e4e20734:   3,"xyz":4
ESS:*:5c12b5b8:   56}

Now there's no more risk of corruption.

• All the new chunks are written first, and won't override the old chunks.
• The :root chunk is updated next, since it's just a pointer to the other keys.
• The old chunks are deleted last (just to cleanup old space).

With this approach, any API request can fail without corrupting the data.
This doesn't solve the atomicity issue, but it's still an improvement.

If the max key length is 255, this gives you ~7.9kB of space¹. With a maxlength of 10,000 chars, you get 12MB.

---

but: with all that said, do we really want this much extra complexity just to work around a limitation that could be solved by openstreetmap/openstreetmap-website#6399 - can we increase the limit to 1000000 characters? that's enough to store the entire id-tagging-schema with no compression

Footnotes

1. limit = 255; hash_length = 8; floor(limit / hash_length) * limit ↩

[tordans]

Thanks a lot. Will update my PR to 1_000_000 and see how it goes.

[tordans]

Will update my PR to 1_000_000 and see how it goes.

I just updated my PR openstreetmap/openstreetmap-website#6399 (comment)
Lets wait for feedback over there…