Serialization error due to unknown language(s)

[MCOfficer]

If DeepL detects a language the library doesn't know about, it might respond like this:

{
  "translations" : [ {
    "text" : "[omitted]",
    "detected_source_language" : "CEB",
    "tag_handling_version" : "V1"
  } ]
}

Which causes the following error:

InvalidResponse(
    "convert json bytes to Rust type: error decoding response body",
)

The custom error message at

deepl-rs/src/lang.rs

Line 149 in eda1cbf

format!("invalid language code {lang}. This is an internal issue with the lib, please open issue")

is missing.

---

Taking a step back, can the serialization in general be improved to be less prone to errors?
Maybe an unknown language could serialize to something like Lang::UNKNOWN(String)? Or would using an ISO-693 crate be better, to cover all possible codes?

[Avimitin]

I understand your reason on how to treat the serialization, but the original intent of covering all possible language codes was to ensure downstream projects can confidently select source and destination languages. The core philosophy is that this crate should handle the complexity of mapping various strings to an enum.

While using Unknown as a catch-all or strictly following ISO-693 has merits, my concern is that it shifts the conversion workload onto downstream projects (For example, if we apply this idea, downstream project will have to write their match control flow to handle string conversion or language code that DeepL does not support.) . Instead, I believe the best approach would be improving the error messages and adding automation to handle new language varieties more easily.

I am currently busy on my job, I will try to take some time looking on this issue.

[MCOfficer]

Thank you for elaborating. I can kind of see how there are two use cases at odds with each other.

On the one hand, I agree that ideally, the crate should provide type-safe mappings.
On the other hand, we are using DeepL to translate XMLs, which can at times contain little to no actual text. If DeepL wrongly recognizes them as an obscure language, translation becomes straight up impossible. And unless you know the input language and provide it in the query, there's virtually no way to fix it.

What are your throughts on hiding this kind of "best effort" serialization behind a parameter or feature flag? That way the default safe mapping is retained, but when things break, users can opt into a workaround until the crate is updated.

Instead, I believe the best approach would be improving the error messages and adding automation to handle new language varieties more easily.

I did briefly check the API, and I didn't see any way to get all languages (apart from scraping the docs). /v2/languages only seems to return a subset, CEB for instance is not among them :/

If time is an issue for you, I might be able to PR whatever we come up with.

[Avimitin]

Thanks for your further explanation. I totally understand your use case.

What are your throughts on hiding this kind of "best effort" serialization behind a parameter or feature flag?

If we are taking a step back, then I prefer use Lang::Unknown to help catch cases that are not handled. With further consideration of my current situation, I think I don't have the ability to update language code in time, so maybe this is the best way to minimize code changes and help downstream project continuing working without waiting this crate to update.

If time is an issue for you, I might be able to PR whatever we come up with.

PR is always welcome! I really appreciate it if you have time working on this issue.