Commit c5145db
Add
* Allows users to keep their pipelines elastic and restore from a checkpoint with variable amount of hosts
Limitations
* Does not guarantee determinism between scaling
* The limit of parallelism is the number of shards
* Currently doesn't support multiprocessing
PiperOrigin-RevId: 864910611ElasticIterDatasetIterator to handle scaling up and down between checkpoints.1 parent 5bad1a2 commit c5145db
4 files changed
Lines changed: 580 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| 71 | + | |
71 | 72 | | |
72 | 73 | | |
73 | 74 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
166 | 166 | | |
167 | 167 | | |
168 | 168 | | |
| 169 | + | |
169 | 170 | | |
170 | 171 | | |
171 | 172 | | |
| |||
0 commit comments