Skip to content

feat: allow grouping models in report output via flag/toggle#117

Draft
cantalupo555 wants to merge 1 commit intojunhoyeo:mainfrom
cantalupo555:fix/model-aggregation-by-model-id
Draft

feat: allow grouping models in report output via flag/toggle#117
cantalupo555 wants to merge 1 commit intojunhoyeo:mainfrom
cantalupo555:fix/model-aggregation-by-model-id

Conversation

@cantalupo555
Copy link
Copy Markdown
Contributor

Summary

  • Remove provider_id from model aggregation key to prevent duplicate entries when the same model appears with different provider names

Problem

When a provider changes its name (e.g., zai-coding-planzhipuai-coding-plan for Zhipu AI), the same model (GLM-4.7) appears as two separate entries in the report because the aggregation key included the provider_id.

Before (duplicated):

Model Provider Cost
glm-4.7 zai-coding-plan $8.92
glm-4.7 zhipuai-coding-plan $0.15

After (consolidated):

Model Cost
glm-4.7 $9.07

Changes

Changed aggregation key from source:provider_id:model_id to source:model_id in 3 locations:

  • get_model_report()
  • finalize_report()
  • finalize_report_and_graph()

This is safe because:

  1. Cost calculation happens per-message BEFORE aggregation
  2. Pricing always uses the official provider price (not intermediary)
  3. The provider field is retained in ModelUsage struct for reference

Remove provider_id from model aggregation key to prevent duplicate
entries when the same model appears with different provider names
(e.g., zai-coding-plan vs zhipuai-coding-plan for GLM-4.7).
@vercel
Copy link
Copy Markdown
Contributor

vercel bot commented Jan 15, 2026

@cantalupo555 is attempting to deploy a commit to the Inevitable Team on Vercel.

A member of the Team first needs to authorize it.

@cantalupo555
Copy link
Copy Markdown
Contributor Author

Same model (glm-4.7) was being split across providers. Now correctly aggregated:

{
  "source": "opencode",
  "model": "glm-4.7",
  "provider": "zai-coding-plan",
  "input": 11704248,
  "output": 114649,
  "cacheRead": 14947523,
  "cacheWrite": 0,
  "reasoning": 507,
  "messageCount": 445,
  "cost": 8.92011953
}
{
  "source": "opencode",
  "model": "glm-4.7",
  "provider": "zhipuai-coding-plan",
  "input": 214600,
  "output": 2234,
  "cacheRead": 103025,
  "cacheWrite": 0,
  "reasoning": 1233,
  "messageCount": 5,
  "cost": 0.14772015
}

After

{
  "source": "opencode",
  "model": "glm-4.7",
  "provider": "zai-coding-plan",
  "input": 11918848,
  "output": 116883,
  "cacheRead": 15050548,
  "cacheWrite": 0,
  "reasoning": 1740,
  "messageCount": 450,
  "cost": 9.067839679999997
}

@junhoyeo junhoyeo self-requested a review January 15, 2026 05:39
@junhoyeo junhoyeo self-assigned this Jan 24, 2026
junhoyeo
junhoyeo previously approved these changes Jan 24, 2026
@junhoyeo junhoyeo self-requested a review January 24, 2026 11:10
@junhoyeo
Copy link
Copy Markdown
Owner

Hey @cantalupo555, thanks for taking the time to dig into this and submit a PR! I appreciate the effort.

After thinking about this more carefully, I don't think I can merge this change. Here's why:

The provider dimension is intentionally part of the aggregation key.

The same model accessed through different providers can have significantly different pricing. For example:

  • claude-3-5-sonnet via Anthropic direct: $3/$15 per 1M tokens
  • claude-3-5-sonnet via AWS Bedrock: different pricing structure
  • claude-3-5-sonnet via Azure: yet another pricing

If we remove provider_id from the key, these would be incorrectly merged — and users would lose visibility into where their costs are actually coming from.

Users with multiple providers need per-provider breakdowns.

Some power users intentionally route the same model through different providers (for redundancy, rate limits, or cost optimization). They rely on the per-provider breakdown to understand their usage patterns and costs across each provider. Merging these would remove valuable insights they depend on.

Thanks again for the contribution! 🙏

@junhoyeo junhoyeo closed this Jan 26, 2026
@cantalupo555
Copy link
Copy Markdown
Contributor Author

cantalupo555 commented Jan 26, 2026

Hey @junhoyeo, thanks for the review!

After updating OpenCode and logging back into my subscription (Z.AI Coding Plan), I noticed that it had a different name, even though it was the same provider.

I agree we should keep provider_id in the aggregation key — even when prices are the same, the per‑provider breakdown is still useful for visibility/convenience.

So the “different pricing per provider” argument is less applicable here; provider_id is mainly used for breakdown.

The cost of OpenCode is recalculated only by model_id, not by provider_id, using lookup. In other words, today provider_id is mainly used for breakdown/visibility, not for pricing.

Or the example you mentioned above, those providers currently show the same pricing (screenshot below). I’m not claiming pricing is always identical across providers — just that in this OpenCode case, pricing variance isn’t the issue.
image

The issue that motivated the PR is that OpenCode is now showing different IDs for the same provider (zai-coding-plan and zhipuai-coding-plan), which splits the same model in the report. This change comes from the models.dev catalog, not from the auth code.
Screenshot From 2026-01-26 11-57-49

So I’d propose reopening the PR as draft with a safer approach: add a “group by model” view option (without changing the internal key). This could be:

  • a CLI flag (--group-by-model) for the “models” output
  • and a TUI hotkey toggle to quickly switch views.

References (OpenCode repo):

  • packages/opencode/src/provider/models.ts (fetches https://models.dev/api.json + cache in ~/.cache/opencode/models.json)
  • packages/opencode/test/tool/fixtures/models-api.json (contains both IDs)
    (others in the OpenCode repo if needed)

Catalog IDs:

If this makes sense, could you reopen the PR as draft so I can update it with the grouping option (flag + TUI toggle) purely at the presentation layer?

@junhoyeo junhoyeo reopened this Jan 26, 2026
@junhoyeo
Copy link
Copy Markdown
Owner

junhoyeo commented Jan 26, 2026

Makes sense! @cantalupo555
I also think making a group-by-model feature only in the display layer is a great idea, maybe we would like to add a new flag to the configuration file as well?

@cantalupo555 cantalupo555 marked this pull request as draft January 26, 2026 15:20
@cantalupo555 cantalupo555 changed the title fix: aggregate models by source and model_id only feat: allow grouping models in report output via flag/toggle Jan 26, 2026
@junhoyeo junhoyeo removed their assignment Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants