Skip to content

Commit c5145db

Browse files
PAenugulacopybara-github
authored andcommitted
Add ElasticIterDatasetIterator to handle scaling up and down between checkpoints.
* Allows users to keep their pipelines elastic and restore from a checkpoint with variable amount of hosts Limitations * Does not guarantee determinism between scaling * The limit of parallelism is the number of shards * Currently doesn't support multiprocessing PiperOrigin-RevId: 864910611
1 parent 5bad1a2 commit c5145db

4 files changed

Lines changed: 580 additions & 22 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ changes. Best viewed [here](https://google-grain.readthedocs.io/en/latest/change
6868
* Exposes `SharedMemoryArrayMetadata` in a public API as a metadata descriptor
6969
for `SharedMemoryArray`.
7070
* `ParquetIterDataset` can read from multiple string paths interleaving reads.
71+
* Add `ElasticIterDatasetIterator` for scaling up and down the number of shards between checkpoints.
7172

7273
* Breaking changes:
7374
* Custom implementations of `RandomAccessDataSource` should accept `int`

grain/_src/python/dataset/BUILD

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,7 @@ py_library(
166166
":dataset",
167167
"//grain/_src/core:sharding",
168168
"//grain/_src/python:options",
169+
"//grain/_src/python/dataset/transformations:zip",
169170
],
170171
)
171172

0 commit comments

Comments
 (0)