Skip to content

docs: japan cloud - #2809

Merged
jannikmaierhoefer merged 26 commits into
mainfrom
pr/japan-updates
Apr 26, 2026
Merged

docs: japan cloud #2809
jannikmaierhoefer merged 26 commits into
mainfrom
pr/japan-updates

Conversation

@clemra

@clemra clemra commented Apr 14, 2026

Copy link
Copy Markdown
Member

No description provided.

@vercel

vercel Bot commented Apr 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
langfuse-docs Ready Ready Preview, Comment Apr 26, 2026 11:58pm

Request Review

@github-actions

Copy link
Copy Markdown

@claude review

@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. docs labels Apr 14, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — straightforward documentation update adding the Japan (beta) region to the data regions table and subprocessors list.

Extended reasoning...

Overview

This PR adds documentation for the new Japan (beta) cloud region across two security documentation files: content/security/data-regions.mdx and content/security/subprocessors.mdx. Changes include adding the Japan region row to the regions table, a URL example, updating the introductory text, and a new subprocessors section mirroring the EU region structure with AWS/Clickhouse hosted in Japan.

Security Risks

No security risks. These are purely documentation changes with no code logic, authentication, or data-handling changes.

Level of Scrutiny

Low scrutiny required. The changes follow the exact same pattern used for the existing US, EU, and HIPAA regions. The subprocessor table for Japan correctly shows AWS and Clickhouse in Japan and the remaining processors in EU, consistent with the explanatory text.

Other Factors

No bugs were found. The Vercel preview deployed successfully. The changes are internally consistent and follow established documentation conventions.

@jannikmaierhoefer jannikmaierhoefer changed the title docs: japan cloud subprocessors docs: japan cloud Apr 24, 2026
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Apr 24, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Straightforward docs + region-selector config addition for the new Japan cloud region; the only finding is a minor labeling-consistency nit already posted inline.

Extended reasoning...

Overview

This PR adds the new Japan cloud region (jp.cloud.langfuse.com, AWS ap-northeast-1/Tokyo) to four places: the cloudRegions map and selector order in lib/cloud-regions.ts, the region card config in app/cloud/[[...path]]/page.tsx, a new row + bullet in content/security/data-regions.mdx, and a new 'Langfuse Cloud: Japan' subprocessors section + an intro-sentence tweak in content/security/subprocessors.mdx. All changes follow the existing patterns established for the US/EU/HIPAA regions.

Security risks

None meaningful. The code change is a new entry in a static region-config map and a new card in a client-side selector; the URL is a fixed https endpoint with no user-controlled interpolation, and the existing buildCloudRedirectUrl/stripControlChars logic handles it the same way it handles the other regions. The MDX changes are pure content. Subprocessors disclosures are compliance-relevant but this is a documentation addition, not a policy change.

Level of scrutiny

Low. This is a docs-and-config PR that mirrors existing patterns exactly — no logic changes, no new abstractions, no auth/crypto touchpoints, and no CODEOWNER-sensitive code. Mechanical additions to parallel structures.

Other factors

The bug hunter flagged a real but minor inconsistency: cloudRegions.jp.label and the MDX bullet list say 'Japan (beta)', but the /cloud selector card, the data-regions table row, and the subprocessors heading all say plain 'Japan'. That's worth the author picking one form — but it's a labeling nit, not a correctness issue, and the inline comment already surfaces the specific locations. No other issues found.

Comment thread lib/cloud-regions.ts
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Apr 24, 2026
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

Made-with: Cursor

# Conflicts:
#	components/ToAppButton.tsx
Comment thread cookbook/otel_integration_mlflow.ipynb
Comment thread content/integrations/native/opentelemetry.mdx
Comment thread md-override/pricing.md
Comment thread app/cloud/[[...path]]/page.tsx
Comment thread content/security/dpa.mdx
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Apr 24, 2026
Comment thread components/japan/JapanLanding.tsx
Comment thread content/marketing/meta.json
Comment thread components/japan/JapanLanding.tsx
Comment thread components/japan/JapanLanding.tsx Outdated
Comment thread components/japan/JapanLanding.tsx
clemra and others added 3 commits April 26, 2026 13:32
Clarify at the top of the v4 data migration cookbook that migrated observations are newly ingested in the destination project and therefore incur ingestion-based charges.

Made-with: Cursor
Refine the Langfuse Cloud Japan changelog copy to streamline positioning, simplify metadata, and clarify migration guidance for larger observation volumes.

Made-with: Cursor
Comment thread content/guides/cookbook/example_data_migration.mdx Outdated
Comment thread app/japan/page.tsx
Correct the Langfuse Fast Grant monthly credit value on the research page from USD 499 to the actual USD 200 amount.

Made-with: Cursor
Comment thread content/guides/cookbook/example_data_migration.mdx
Comment thread .langfuse/current-session.json
Comment thread content/security/data-regions.mdx
Comment thread components/japan/JapanLanding.tsx Outdated
Comment thread content/guides/cookbook/example_data_migration.mdx Outdated
Comment thread content/changelog/2026-04-27-langfuse-cloud-japan.mdx
Comment thread components/japan/JapanLanding.tsx
Comment thread content/guides/cookbook/example_data_migration-jp.mdx
Comment thread content/guides/cookbook/example_data_migration.mdx
Comment thread public/images/japan-logos/sumup.svg
Comment thread content/guides/cookbook/example_data_migration.mdx
Comment thread cookbook/example_data_migration-jp.ipynb Outdated
Comment thread content/blog/2026-04-27-langfuse-cloud-japan.mdx
@jannikmaierhoefer
jannikmaierhoefer merged commit 40c0823 into main Apr 26, 2026
13 checks passed
@jannikmaierhoefer
jannikmaierhoefer deleted the pr/japan-updates branch April 26, 2026 23:58
Comment on lines 1 to 16
---
title: Migrating Data Between Langfuse Projects (Python SDK v4)
sidebarTitle: Data Migration Script
description: Migrate observations, scores, prompts, datasets, custom models, score configs, and annotation queues between Langfuse projects using the Python SDK v4.
description: Migrate observations, scores, prompts, datasets, custom models, and score configs between Langfuse projects using the Python SDK v4.
---

# Migrating Data Between Langfuse Projects (Python SDK v4)

This notebook migrates data from one Langfuse project to another using the **Langfuse Python SDK v4**.
This notebook migrates data from one Langfuse project to another using the **Langfuse Python SDK v4**.

<Callout type="warning">
Migrating data into a Langfuse project creates new ingestion events in the destination project. This migration therefore incurs Langfuse ingestion charges based on the number of observations transferred.
</Callout>

Common use cases:
- Migrating between cloud regions (US ↔ EU ↔ JP ↔ HIPAA)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Section 3 (Prompts) at example_data_migration.mdx:358-397 unconditionally calls dst.api.prompts.create(request=req) for every source (name, version) pair with no destination-side idempotency check. Every other section in the script has one (Section 1: existing_by_name dedup; Section 2: catches 409/"already exists"; Section 4: deterministic OTel IDs; Section 5: preserves score_id=s.id; Section 6: datasets.get(...) before create). On retry — which the launch changelog explicitly invites for >1M observation migrations — the destination accumulates duplicate prompt versions, and Langfuse's automatic label-move semantics silently shift production (and other labels) onto the duplicates, breaking apps pinned to a specific version. Same fix needed in example_data_migration-jp.mdx and both .ipynb siblings: at the start of Section 3, paginate destination prompts and skip (source_name, source_version) pairs whose commit_message already records that source version (the script already writes "Original version: N" into commit_message).

Extended reasoning...

What the bug is

Section 3 of the new migration cookbook (content/guides/cookbook/example_data_migration.mdx:358-397, mirrored in example_data_migration-jp.mdx and both paired .ipynb files) iterates every source (name, version) pair and calls:

with_retries(lambda: dst.api.prompts.create(request=req))

with no destination-side check. The Langfuse prompts API does not accept a target version on create — it auto-assigns max(existing) + 1 per name — so re-running the script enumerates the same source pairs and creates new destination versions with sequential numbers each time.

Why the asymmetry matters

Every other section of the same script has explicit idempotency:

  • Section 1 (score configs, line 223): existing_by_name dedup, increments skipped on collision.
  • Section 2 (custom models, line 305-306): catches 409 / "already exists" as skipped.
  • Section 4 (observations, lines 139-150): deterministic to_otel_trace_id / to_otel_span_id derivation, so re-runs land on the same destination IDs.
  • Section 5 (scores, line 766): preserves score_id=s.id with an inline comment about idempotent re-runs.
  • Section 6 (datasets, line 838): calls dst.api.datasets.get(...) before create_dataset; items preserve source IDs.

Section 3 is the only outlier. The cookbook description says "The destination assigns fresh version numbers; the original source version is recorded in commit_message so it stays traceable" — nothing warns the reader that re-running is unsafe.

Step-by-step proof

  1. First run. Source has prompt my_prompt v1 (label production), v2. Section 3 calls prompts.create() for both. Destination assigns sequential versions: my_prompt v1, v2 (with production on v1 to mirror source).
  2. Transient failure (rate limit, network blip, partial migration of >1M observations). User re-runs the script with the same credentials.
  3. Section 3 enumerates the same source pairs. The destination, having no concept of source-version equivalence, assigns the next sequential numbers — my_prompt v3 (content of source v1), v4 (content of source v2).
  4. commit_message records "Original version: 1." on both destination v1 and destination v3, so by inspection nothing flags the duplication. Destination quietly accumulates redundant prompt versions on every retry.

Compounding label side-effect

Langfuse documents that labels are unique per prompt name: creating a new version with label production automatically removes that label from the prior version. So the retry doesn't just create v3, v4 — it actively shifts production from destination v1 to destination v3 (a duplicate of the same content). Apps pinned to a specific destination version number now diverge from what production resolves to. Same applies to any other moveable label (e.g., latest, staging).

Why this matters at launch

The migration cookbook is the canonical path linked from the new launch changelog (content/changelog/2026-04-27-langfuse-cloud-japan.mdx:23) and from the /japan landing page's Migration CTA. The launch changelog explicitly invites users with >1M observations to reach out — those large migrations are precisely the runs most likely to need a retry after partial failure.

Fix

At the start of Section 3, paginate destination prompts and build a set of (name, original_source_version) already migrated by reading the commit_message field (the script already writes "Original version: N" there). Skip any source pair whose mapping is already present:

import re
_OV = re.compile(r"Original version: (\d+)")

migrated_pairs = set()
page = 1
while True:
    resp = with_retries(lambda: dst.api.prompts.list(page=page, limit=100))
    if not resp.data:
        break
    for meta in resp.data:
        for v in meta.versions:
            p = with_retries(lambda: dst.api.prompts.get(quote(meta.name, safe=''), version=v))
            m = _OV.search(p.commit_message or '')
            if m:
                migrated_pairs.add((meta.name, int(m.group(1))))
    if page >= getattr(resp.meta, 'total_pages', page):
        break
    page += 1

# … then skip in the main loop:
if (name, version) in migrated_pairs:
    skipped += 1
    continue

Same change is needed in all four authoring artifacts: example_data_migration.mdx, example_data_migration-jp.mdx, cookbook/example_data_migration.ipynb, cookbook/example_data_migration-jp.ipynb.

Comment on lines 213 to 218
## 1. Score Configs

Score configs must be migrated first so that scores and annotation queues created in the destination can reference them by ID. We keep a `config_id_map` to remap references in later sections.
Score configs must be migrated first so that scores created in the destination can reference them by ID. We keep a `config_id_map` to remap references in later sections.



Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Section 1's destination dedup map at example_data_migration.mdx:223 (and JP sibling example_data_migration-jp.mdx:216, plus the matching .ipynb cells) builds existing_by_name from a single un-paginated dst.api.score_configs.get(limit=100).data call — every other listing call in the script paginates via while True. If the destination project already has more than 100 score configs (realistic for any production team running LLM-as-Judge for a while), dedup silently misses entries past page 1: source configs whose names collide with un-fetched destination configs are sent through score_configs.create and either accumulate as duplicates or get caught by the bare except, leaving config_id_map[cfg.id] unpopulated — which then causes Section 5 to write the raw source config_id UUID into destination scores via config_id_map.get(s.config_id, s.config_id), producing dangling references. Fix: replace the single-page fetch with a paginated loop mirroring the source-side pattern at lines 227–263.

Extended reasoning...

What the bug is

In Section 1 of the migration cookbook, existing_by_name (the destination-side dedup map for score configs) is built from a single un-paginated REST call:

existing_by_name = {c.name: c for c in dst.api.score_configs.get(limit=100).data}

This is the only listing call in the entire script that does not paginate. The very next block at lines 227–263 correctly iterates source configs page-by-page via while True with a getattr(resp.meta, "total_pages", page) check, and the rest of the script (prompts, observations, scores, datasets) all paginate too. The author clearly knew the right pattern; this single line is the outlier.

Why this matters

If the destination project already has more than 100 score configs — realistic for any team running LLM-as-Judge in production for a while, since each judge prompt + score type typically gets its own config — existing_by_name is silently incomplete. For any source config whose name matches a destination config that did not land on page 1:

  1. if cfg.name in existing_by_name evaluates false.
  2. dst.api.score_configs.create(...) is called.
  3. Two failure modes follow, depending on how the server handles duplicate-name configs:
    • Server allows duplicate names (Langfuse score configs do not currently enforce a unique-name constraint): a duplicate config is silently created, leaving the destination with two configs that share a name and have different IDs.
    • Server rejects with 409: the bare except Exception at lines 258–260 catches it, increments failed, and continues — but config_id_map[cfg.id] is never populated.
  4. In Section 5, scores are migrated with new_cfg_id = config_id_map.get(s.config_id, s.config_id) if s.config_id else None (line 757). When the map entry is missing, the fallback writes the raw source UUID into the destination score, creating a dangling config_id reference to a config that does not exist in the destination project.

So the user ends up with either silent duplicate score configs in the destination, or scores that point at non-existent config UUIDs — depending on the server-side behavior for duplicate names.

Step-by-step proof

  1. Destination project already has 150 score configs (e.g., from a prior LLM-as-Judge setup or a previous migration run).
  2. User runs this cookbook against that destination.
  3. Line 223: dst.api.score_configs.get(limit=100).data returns the first 100. existing_by_name has 100 entries.
  4. The remaining 50 destination configs are not in the map.
  5. Source has a config named factuality_v2 that exists on the destination but did not land in page 1.
  6. Loop iteration: cfg.name = "factuality_v2", cfg.name in existing_by_name is False (it would have been True if pagination worked).
  7. dst.api.score_configs.create(name="factuality_v2", ...) is called. Either a duplicate is silently created, or a 409 is caught at lines 258–260.
  8. config_id_map["<source-config-id>"] is left unset (in the 409 path).
  9. Section 5 migrates a score that referenced factuality_v2. new_cfg_id = config_id_map.get(source_config_id, source_config_id) returns the raw source UUID.
  10. The destination accepts the score with a config_id pointing at a UUID that does not exist in this project. Dangling reference.

Why this slips past review

The source-side loop two lines below paginates correctly via while True with a total_pages check, so anyone reviewing the diff naturally assumes the destination fetch does too. The mismatch is easy to miss because limit=100 reads as a per-page parameter rather than a hard cap.

How to fix

Replace the single-page fetch with a paginated loop, mirroring the existing source-side pattern:

existing_by_name: Dict[str, Any] = {}
page = 1
while True:
    resp = with_retries(lambda: dst.api.score_configs.get(page=page, limit=100))
    if not resp.data:
        break
    for c in resp.data:
        existing_by_name[c.name] = c
    if page >= getattr(resp.meta, "total_pages", page):
        break
    page += 1

Same fix needed in:

  • content/guides/cookbook/example_data_migration.mdx:223
  • content/guides/cookbook/example_data_migration-jp.mdx:216
  • The corresponding cells in cookbook/example_data_migration.ipynb and cookbook/example_data_migration-jp.ipynb (the .mdx files are generated from the .ipynb siblings, so the notebooks are the canonical edit point).

Severity

normal — silent data-fidelity regression on a launch-day cookbook. Trigger requires the destination project to already hold more than 100 score configs, which is realistic for any production team using LLM-as-Judge. The cookbook is the canonical migration path linked from the new launch changelog (content/changelog/2026-04-27-langfuse-cloud-japan.mdx:23) and the /japan landing page Migration CTA (components/japan/JapanLanding.tsx:841).

Comment on lines +50 to +53
.japan-corners.no-tl::before { -webkit-mask-position: 50% 150%, bottom left, bottom right, top right; mask-position: 50% 150%, bottom left, bottom right, top right; }
.japan-corners.no-tr::before { -webkit-mask-position: top left, bottom left, bottom right, 50% 150%; mask-position: top left, bottom left, bottom right, 50% 150%; }
.japan-corners.no-bl::before { -webkit-mask-position: top left, 50% 150%, bottom right, top right; mask-position: top left, 50% 150%, bottom right, top right; }
.japan-corners.no-br::before { -webkit-mask-position: top left, bottom left, 50% 150%, top right; mask-position: top left, bottom left, 50% 150%, top right; }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The four .japan-corners.no-tl/no-tr/no-bl/no-br::before rules at components/japan/styles.tsx:50-53 all have identical specificity (0,0,2,1) and each replaces the entire mask-position 4-value list, so when multiple no-* classes are applied only the last in source order (no-br) wins. On the new /japan launch page this leaves stray 8x8 corner brackets at the seams: JapanLanding.tsx:82 (no-tl no-tr no-bl no-br) renders three of four brackets, :60 (no-bl no-br) leaves bottom-left visible, and :139 (no-tl no-tr) leaves top-left visible. Fix by stacking modifiers as composite selectors (e.g. .no-tl.no-tr, .no-bl.no-br, .no-tl.no-tr.no-bl.no-br) so each combination has a unique selector, or by rewriting per-layer using individual mask properties.

Extended reasoning...

What the bug is

components/japan/styles.tsx defines four corner-bracket modifier rules at lines 50-53:

.japan-corners.no-tl::before { mask-position: 50% 150%, bottom left, bottom right, top right; }
.japan-corners.no-tr::before { mask-position: top left, bottom left, bottom right, 50% 150%; }
.japan-corners.no-bl::before { mask-position: top left, 50% 150%, bottom right, top right; }
.japan-corners.no-br::before { mask-position: top left, bottom left, 50% 150%, top right; }

All four selectors have specificity (0,0,2,1) — two classes plus one pseudo-element — so any combination of them ties under CSS cascade rules. Each rule sets the full 4-layer mask-position list, not just the one position it intends to hide. When two or more apply to the same element, the rule defined latest in source order replaces the entire list, restoring all of the earlier rule's hidden positions to their visible defaults.

Why existing code does not prevent it

The base .japan-corners::before rule at lines 21-49 sets the four mask layers and the four positions in a single 4-value list. There is no per-layer rule using mask-position-x/mask-position-y or individual mask-image layers, so any modifier has to override the whole 4-value list at once.

Step-by-step proof (the most prominent case, JapanLanding.tsx:82)

  1. The middle hero box at components/japan/JapanLanding.tsx:82 has className={${cornerBoxBase} no-tl no-tr no-bl no-br ...}. The author's intent is clearly "all four corner brackets hidden" — the box sits between the top hero strip (line 60, no-bl no-br) and the stats strip (line 113-115) to read as one continuous bracketed column.
  2. CSS resolves the four matching rules at styles.tsx:50-53 in source order. All have specificity (0,0,2,1).
  3. .no-tl::before sets mask-position: 50% 150%, bottom left, bottom right, top right. Then .no-tr::before overwrites the entire list with top left, bottom left, bottom right, 50% 150% — the top-left position is back to visible. Then .no-bl::before overwrites again, restoring top-right to visible. Finally .no-br::before writes top left, bottom left, 50% 150%, top right — the final state.
  4. Final effective mask-position: top left, bottom left, 50% 150%, top right → only the bottom-right bracket is hidden. The other three (top-left, top-right, bottom-left) are still painted.
  5. Visually, this means the middle column of the hero stack shows three of four small 8×8 bracket marks at exactly the points the author meant to hide, breaking the intended "one continuous bracketed column" effect.

The same logic applies to :60 (no-bl no-br) — only no-br wins, bottom-left stays visible — and to :139 (no-tl no-tr) — only no-tr wins, top-left stays visible.

Impact

Purely visual, no runtime or compile breakage, but on the marquee /japan launch page that the new sitewide banner at components/layout/Banner.tsx:10-15 funnels traffic toward. A reader scanning the hero will see stray bracket marks at internal seams that read as misaligned UI rather than design intent.

Fix

Either of these is well-scoped to this PR:

  1. Composite class selectors so each combination actually used in markup gets its own unique selector with a non-tied specificity:
    .japan-corners.no-bl.no-br::before { mask-position: top left, 50% 150%, 50% 150%, top right; }
    .japan-corners.no-tl.no-tr::before { mask-position: 50% 150%, bottom left, bottom right, 50% 150%; }
    .japan-corners.no-tl.no-tr.no-bl.no-br::before { mask-position: 50% 150%, 50% 150%, 50% 150%, 50% 150%; }
  2. Per-layer rules that change only one layer per modifier, e.g. by splitting the four masks into individually addressable layers with separate mask-position-x/mask-position-y properties, so each no-* rule only writes the layer it owns.

Severity: nit — purely visual on a marketing surface, no functional regression. Worth fixing before launch since /japan is the destination of the new launch banner and the broken corners are most prominent on the central hero box (line 82) that ships with all four modifiers.

@claude claude Bot mentioned this pull request May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants