Exploring Cost-Effective Translation Alternatives Using LLM Models

[AioPhos]

We are currently using DeepL for our translation needs, but I've been thinking about more cost-effective solutions that could potentially be free or significantly cheaper. I'd like to explore the possibility of using Large Language Models (LLMs) like Llama 3.2 90B vision model or other models with OpenAI-compatible API integration for our translation tasks.

Here's what I'm proposing:

Alternative LLM Models: Instead of relying on DeepL, we could use LLMs like Llama 3.2 90B or other models that support text-to-image translation. These models might offer similar or even better translation capabilities at a lower cost or even for free.

OpenAI-Compatible API Integration: Many LLMs are compatible with the OpenAI API, which means we can easily switch providers by changing the API endpoint, API key, and model name. This flexibility allows us to explore various models without significant changes to our existing infrastructure.

Free Alternatives: There are several hosts that offer free access to their LLM models through their APIs. By signing up with these hosts, we could potentially translate text for pictures at no cost, which would be a significant saving compared to our current DeepL expenses. Providers like Mistral AI, SambaNova, Grok, and many others offer free access to their models through their APIs.

Feasibility and Implementation: I'd like to investigate whether implementing this solution is feasible. We need to assess the quality of translations from these alternative models, the ease of integration, and the overall cost-effectiveness compared to DeepL.

Next Steps:

Research available LLM models with OpenAI-compatible API integration.
Identify hosts offering free access to these models, including providers like Mistral AI, SambaNova, Grok, and others.
Test the translation quality and performance of these models against our current solution.
Evaluate the potential cost savings and feasibility of implementation.
I believe this approach could revolutionize our translation process, making it more efficient and cost-effective. I'd love to hear your thoughts on this and whether you think it's worth exploring further.

Let's discuss the potential benefits and challenges of moving away from DeepL to LLM-based translation solutions.

[DCY1117]

Sorry for the late reply.

The current workflow is as follows:

• Text Segmentation: We begin by detecting the text in images, which also allows us to capture text outside the bubbles. The output is a text segmentation layer. The downside is that there are quite a few false positives, but we can reduce them using the "Modify text boxes" feature in the GUI.

• Text Block Detection: After obtaining the segmentation layer, we dilate it to group together text that is close to one another, forming blocks. I know there is still room for improvement for this part.

• Text Recognition: We use Manga-OCR to convert the text in each block into digital text for further processing.

• Image Inpainting: Llama inpainting is employed to restore the blocks of text for later insertion.

• Text Translation and Injection: Currently, DeepL is used to translate the text, and the translated content is then injected back into its corresponding text block in the image.

Your idea to explore vision-based translation solutions directly is great because it would allow the model to also capture the context of the images when translating which would fix ambiguity in pronouns in japanese translation and other things that are hard to translate without context, but we would need to verify if the translation is good and whether the models detect all the text correctly. Essentially, this approach could eliminate the need for text recognition and translation separately.

With this method, we'd have the translated text for each block, assuming the model is accurate and got all the block of text. Then, we would only need to determine which text belongs to which block and inject it accordingly.

However, most models, like Llama 3.2 90B, are too large to run locally. Additionally, I don’t think there’s a viable free option for using APIs effectively. That said, the idea is solid and could be implemented as an additional feature to use vision models for more accurate text translation. There are also smaller models available that could be explored to try out this feature or or "Identify hosts offering free access to these models, including providers like Mistral AI, SambaNova, Grok, and others.".

So basically we would need to get the text and match it to each block, since japanese text is read from right to left, we could just order the blocks and then match it accordingly. Since we are using vision models, we could just make an bounding box for each block with a number, and then use the model to locate those block and translate it, in this way, I think we could get more precise translation and also match the blocks of text. If you find another way of translating the text and also locating the block of text, that would be perfect, since right now the text segmentation part gives a lot of false positives.

We could also use more powerful and curated models such as the one in https://github.com/zyddnys/manga-image-translator and a workflow like https://github.com/dmMaze/comic-text-detector and implement the aformentioned approach for translation.

Given this workflow how would you implement it, or is there a better solution? or replacement for the text segmentation part?

You are also welcome to contribute 👍