convert dump.txt to kmc database

[sashulkaSh]

Hello!
I need to normalize my kmers from two samples before comparing them (in R), but then I need to intersect these datasets. This "kmc_tools simple intersect" operation only works with the kmc database, not with dump.txt, right? How can I convert dump.txt to the kmc database?

[sashulkaSh]

or would it be easier to do the intersection also in R? if there is no way to do the conversion
But my files are over 100GB X_X

[marekkokot]

Hi, there is no direct conversion, unfortunately.
You could translate your dump.txt to fasta and run kmc on this, but to keep the counters you would need to duplicate k-mers.
If you dont need counts its simpler. Not sure what the wole pipeline is so not sure what is the best advice.
What I think may help is to tell you that when you dump kmc database to txt using kmc_tools you can provide -s option. With this you will have sorted k-mers, and for sorted k-mers intersection does not need to load everything into memory, but instead you can just stream/scan both files, and you should be able to perform this even in R. This is not the fastes solution, but may work for your case. If you provide more detailes I may be able to help more. For example what is your input, how you run kmc, what actually you want to achieve.

[sashulkaSh]

Hello! Thanks for your answer. But I need to first normalize the k-mers (this is only possible for dump.txt data, not for the kmc database) and then perform the intersection (this is only possible for the kmc database).

[marekkokot]

I see, I assume your normalized counts are real numbers (likely in range [0,1]).
Currently, kmc and kmc_tools work with integer counters (we had real numbers for quality-based counts, but I don't think they were used, and maintaining them in the code was quite complex, so we removed them). So, at this stage, it is not possible to use real numbers as counters. I think, for your use case, the simplest approach, at least for now, is to dump with -s and merge as I described in the previous response.