Don't output aggregated data for Early Release data runs - #443
Merged
Merged
Conversation
grgmiller
commented
Aug 9, 2026
| # incomplete (analysis of 2024 data revealed that the EIa-923 generation fuel | ||
| # table is missing about 6-7% of records representing about 2% of the total | ||
| # generation and fuel). | ||
| if year <= latest_validated_year: |
Collaborator
Author
There was a problem hiding this comment.
I don't know why the diff captured it like this, but all I did was add this if statement and then move lines 660-672 to lines 627-639
grgmiller
marked this pull request as ready for review
August 11, 2026 23:14
sarahchute
approved these changes
Aug 12, 2026
grgmiller
added a commit
that referenced
this pull request
Aug 12, 2026
* fix misaligned timestamps * update docs * shape using tz offset * fix BA-level aggregation * fix bug with national average profiles * Skip aggregated outputs for early release data (#443)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The EIA early release data notes:
I was curious specifically how incomplete the early release data was, and I used Claude to help implement a new analysis notebook to evaluate the data completeness of the monthly and early release versions of 2024 EIA-923 data relative to the final release 2024 data. The top level findings:
The early release completeness is actually pretty good, so we could potentially debate whether releasing aggregated power sector data would be appropriate. However, our current use case for completing the early release now is primarily for plant-level data, so there's no immediate need to push us to release this data as part of an intermediate data release.
What the code is doing
The primary code change is to skip calculation and output of all of the power sector and carbon accounting results if the pipeline is being run for the early release year.
Testing
Ran the entire pipeline
Where to look
data_pipeline.py: it is a singleif:statement, which indents all of the (unchanged) code below it. I also moved the plant attributes table output before the output since we still want to output that, and it was not modified after that point.Review estimate
<5 min.
Future work
Consider whether early release aggregated results would be appropriate.
Checklist
ruff