Skip to content

dupd takes way too much memory #53

@hydrargyrum

Description

@hydrargyrum

i'm running dupd scan on several directories with a lot of duplicates, dupd took 7gb of RAM (!) while an equivalent program, rdfind only used 24mb of RAM on the exact same directories.
I don't know how dupd works but it puts way too much stuff in RAM, especially considering it uses sqlite, it could offload a lot to sqlite.
My system went OOM several times before I could make dupd run without halting the whole system, especially considering RAM prices today.

here's peak-mem output for rdfind:

Peak memory usage: 21.4 MB (RSS) / 24.4 MB (VSZ)

and dupd on the same scanned dirs:

Peak memory usage: 7.1 GB (RSS) / 8.2 GB (VSZ)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions