Skip to content

Commit 65a896b

Browse files
authored
Fix #37894: Update yaml doc for composite transform changes (#37901)
* Fix #37833: Use named logger instead of root logger in transforms/util.py Replace root logger calls (logging.info, logging.warning, etc.) with a module-level named logger (_LOGGER = logging.getLogger(__name__)) in apache_beam.transforms.util. This allows sdk_harness_log_level_overrides to properly control log levels for this module. * Update yaml-schema.md with implicit input chaining documentation
1 parent ca34216 commit 65a896b

File tree

1 file changed

+48
-0
lines changed

1 file changed

+48
-0
lines changed

website/www/site/content/en/documentation/sdks/yaml-schema.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,54 @@ pipeline:
110110
elements: []
111111
```
112112

113+
## Implicit Input Chaining for Composite Transforms
114+
115+
When using `type: composite`, you can now omit explicit `input` specifications on
116+
sub-transforms. The sub-transforms will automatically chain together, similar to
117+
how `type: chain` works.
118+
119+
For example, this pipeline:
120+
121+
```yaml
122+
pipeline:
123+
type: composite
124+
input: SomeInput
125+
transforms:
126+
- type: MapToFields
127+
config:
128+
language: python
129+
fields:
130+
x: "element.x * 2"
131+
- type: Filter
132+
config:
133+
language: python
134+
keep: "element.x > 5"
135+
```
136+
137+
Is equivalent to:
138+
139+
```yaml
140+
pipeline:
141+
type: composite
142+
input: SomeInput
143+
transforms:
144+
- type: MapToFields
145+
input: SomeInput
146+
config:
147+
language: python
148+
fields:
149+
x: "element.x * 2"
150+
- type: Filter
151+
input: MapToFields
152+
config:
153+
language: python
154+
keep: "element.x > 5"
155+
```
156+
157+
Note: Implicit chaining only works when:
158+
1. The composite transform has an input (e.g., `input: SomeInput`)
159+
2. Sub-transforms have no explicit `input` or `output` specifications
160+
113161
WARNING: If a transform doesn't have the error_handling configuration available
114162
and a user chooses to use this optional output_schema feature, any failures
115163
found will result in the entire pipeline failing. If the user would still like

0 commit comments

Comments
 (0)