Skip to content

Plan for materializing tables #10

@chanwutk

Description

@chanwutk

Materializing table transformations helps improving runtime performance if the transformations are to be execute multiple times.
Some transformations on a table also change table schema based on content in the table.

When to materialize

  • When user observe the table: We already compute all the transformations that builds up the table. We should materialize it, unless the user only observe a part of the table. Ex: table.print({limit: 2})
  • When user say so: We should allow users to control when a table can materialized on demand.
  • When transforming with a verb that requires the knowledge of the data countent: For example, pivot needs to know all the possible values of the pivoting key column before calculating the schema of the output table.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions