Skip to content

Add delete+insert incremental materialization strategy #80

@chris-celerdata

Description

@chris-celerdata

Description

Currently, dbt-starrocks supports ['append', 'insert_overwrite', 'dynamic_overwrite', 'microbatch']. Another extremely common strategy is delete+insert. This strategy modifies rows by first deleting records that match specified conditions, then inserting the new/updated records. Support for row-level deletion does exist.

Proposed Implementation

Native StarRocks Tables

The delete+insert strategy can be implemented immediately for native StarRocks tables using the existing DELETE statement syntax:

DELETE FROM target_table WHERE <unique_key_condition>;
INSERT INTO target_table SELECT * FROM temp_table;

External Tables

Implementation for external tables requires upstream changes in StarRocks core to support row-level modifications. For example, in Iceberg V2, delete operations are supported through position deletes and equality deletes, but StarRocks would need to expose this functionality.

Acceptance Criteria

  • Add delete+insert to supported incremental strategies for native StarRocks tables
  • Implement logic to generate appropriate DELETE and INSERT statements based on unique_key configuration
  • Add tests covering edge cases (null keys, multiple unique keys, etc.)

Additional Notes

Another strategy is merge, but that would require MERGE INTO or UPDATE to be supported, which does not look like a planned feature in StarRocks core. Therefore, this issue will only focus on delete+insert.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions