reduce disk usage so we can halve or quarter server cost #1459
Description
Description
We are paying for wasted disk space, as most of the server usage is from scrapers that are no longer functional with large historical databases.
Describe the solution you'd like
I would like to reduce the Linode 32 GB Memory, 640 GB storage server to half (16 GB Memory, 320 GB storage) or quarter (8 GB Memory, 160 GB Storage) of its current size and thus cost.
Cost Saving:
- USD 1,152 per year if we drop down to 16 GB Memory, 320 GB storage (need to recover 178 GB of disk space)
- USD 1,728 per year if we drop down to 8 GB Memory, 160 GB Storage (need to recover 290 GB of disk space)
Additional context
Only 120 data.sqlite files have been updated in the last year
Current disk usage is 402 GB, with the following highlights:
- 132 GB /var/lib/mysql/morph (126 GB from log_lines)
- 117 GB out of 119 GB used by data.sqlite from 9 users, and most are not active scrapers
- 91 GB used by /var/lib/docker, 85 GB of it from overlay2 (81% of files over a year old), 6 GB from containers dir - consider a periodic docker cleanup?
- 19 GB used by /backups/mysql
Related issues and ideas:
- Clear out records with date-scraped < 30 days ago (planning alerts only looks at last 5 days)
- Vacume DB after deleting records to recover disk space
- clear outdated run details to reduce DB size to allow space for sql backups #1373
- Need to add cleanup as a cron task, consider stricter cleanup of log_lines
- Turn off scrapers that fail for 6 months continually to save resources #1380
- Should turn off scrapers that have not updated data.sqlite for 14 months? (this can be added to the email they get, handles scrapers that scrape seasonal data)
- Should also remove data.sqlite that havn't been updated for > 15 months? (again this can be added to the email they get, includes scrapers that are turned off - we could clear out their repo, the data.sqlite, the docker image)
- Allow admin to easily see historical logs rather than just the last one #1381