refactor(store): extract _item_to_finding, fix TikTok data loss#74
Open
phjlljp wants to merge 1 commit intomvanhorn:mainfrom
Open
refactor(store): extract _item_to_finding, fix TikTok data loss#74phjlljp wants to merge 1 commit intomvanhorn:mainfrom
phjlljp wants to merge 1 commit intomvanhorn:mainfrom
Conversation
Replace 8 nearly-identical for-loops in the --store persistence path with a single _item_to_finding() dispatch function. This also: - Fixes a bug where TikTok findings were silently dropped from SQLite storage (the original loops never included deduped_tiktok) - Adds TruthSocial to the persist path (also previously missing) - Simplifies Engagement.to_dict() using dataclasses.asdict() so new fields are automatically included without manual None-checks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
chidev
added a commit
to chidev/last30days-skill
that referenced
this pull request
Mar 25, 2026
7 tasks
Owner
|
See my reply on #76 covering all three of your PRs. Can't commit to anything right now with the v3.0 refactor underway, but I'll consider each one once it lands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bug fix: TikTok and TruthSocial findings silently lost in
--storemodeWhen a user runs
--storeto persist research findings to SQLite, the code atlast30days.py:1941-2021iterates over each source's deduped items and builds afindingslist forstore_mod.store_findings(). There are 8 separate for-loops, one per source — but TikTok and TruthSocial were never included. Their deduped items (deduped_tiktok,deduped_ts) simply aren't iterated over, so they're silently dropped.This means any user relying on
--storefor watchlist/briefing workflows (the variants/ use case) has been accumulating an incomplete dataset — TikTok and TruthSocial results appear in the CLI output but never make it to the database.Root cause: The 8 for-loops were likely copy-pasted as new sources were added, and TikTok/TruthSocial were missed when they were introduced. The repetitive structure made it easy to overlook.
Refactor: Extract
_item_to_finding()dispatch functionTo prevent this class of bug from recurring, this PR replaces the 8 for-loops with:
_item_to_finding(item)function that dispatches onisinstance()to map any schema item type to the flat dict format expected by the store modulefindings = [_item_to_finding(item) for item in all_deduped]whereall_dedupedexplicitly concatenates all 10 source listsAdding a new source now requires adding one
elifbranch to_item_to_finding()and onelist()entry toall_deduped— rather than copy-pasting an entire for-loop block. The fallbackelsebranch also handles unknown item types gracefully.Cleanup:
Engagement.to_dict()simplifiedReplaces 11 manual
if self.field is not Nonechecks with{k: v for k, v in asdict(self).items() if v is not None}. This means new engagement fields (likeshares,volume,liquiditywhich were added after the original code) are automatically included in serialization without requiring a corresponding manual check. Theasdict()call is safe here becauseEngagementis a flat dataclass with only scalar fields.Test plan
_item_to_finding()Engagement.to_dict()output is identical before/after for all field combinations (all-None, single field, mixed sources, zero values, all fields populated)elsefallback in_item_to_finding()handles unexpected item types without crashing🤖 Generated with Claude Code