Skip to content

Can we remove the local DB and operate directly on the CSV + local assets? #145

@jonborchardt

Description

@jonborchardt

Summary
Much of the slowness and instability appears tied to the local DB layer. Requesting an option (or migration) to run without the DB: read/write directly to the CSV and local asset files.

Context / Problem

  • Current local DB introduces performance overhead and occasional failures.
  • For my workflows, the CSV already represents the source of truth for records; assets live on disk.
  • The DB adds complexity (schema/versioning, sync, caching) that doesn’t seem to pay off for small–medium projects.

Proposal

  • Provide a “DB-less mode” that:
    • Reads records directly from the CSV.
    • Writes changes atomically back to the CSV (append+rename pattern or temp file with checksum).
    • Resolves assets by path on disk (no DB indirection).
    • Uses in-memory indices per session only (rebuild quickly at startup).
  • Keep existing DB mode as optional for very large datasets, if needed.

Questions

  • Are there specific features that hard-require the DB (e.g., complex queries, transactions across multiple tables, offline diff/merge)?
  • What’s the minimum needed indexing to keep UI snappy without persisting a DB?
  • Can we gate this behind a project setting: “Storage backend: CSV (direct) | Local DB”?

Acceptance Criteria

  • Opening, editing, searching, and exporting a project with N≈50k rows works without the DB enabled.
  • Cold start index build on 50k rows completes within a reasonable baseline.
  • All CRUD flows read/write only the CSV and file system assets.
  • No data loss in the face of app crash during write (verified with fault injection).
  • A project-level toggle (and migration path) is documented and tested.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions