Skip to content

[Detail Bug] Test harness: compare_streams reports wrong line number when one output stream ends early #604

Description

@detail-app

Detail Bug Report

https://app.detail.dev/org_89d327b3-b883-4365-b6a3-46b6701342a9/bugs/bug_e789b0b3-2212-41e4-b4af-e42805c579f2

Introduced in #542 by @sgbalogh on Jun 19, 2026

Summary

  • Context: When compare_streams detects that stream A has fewer lines than stream B, it reports "differs at line N" where N is the line count from stream A.
  • Bug: When stream A ends before stream B, the reported line number points to the last matching line (stream A’s line count) rather than the comparison round where the mismatch is detected.
  • Actual vs. expected: Actual: message like stdout differs at line 3 even though line 3 matched in both streams; the real mismatch is the next comparison round where stream B still has content. Expected: the line number should correspond to the comparison round where the mismatch is detected (e.g., differs at line 4 when B has an extra 4th line).
  • Impact: Misleading error messages during determinism/stream comparisons; developers may check the reported line and find it matches, wasting time before realizing the issue is one stream having extra content.

Code with Bug

// sim/src/meta.rs
if read_a > 0 {
    lines += 1; // <-- BUG 🔴 counts only stream A lines; under-reports mismatch position when A ends first
}

Explanation

compare_streams increments lines only when read_a > 0 (i.e., only when stream A produces a line). If stream A ends early but stream B continues, the mismatch is detected on the next comparison round where read_a == 0 and read_b > 0, but lines is not incremented for that round. As a result, the error message uses a line number that refers to the previous (matching) line.

This also produces asymmetric reporting: comparing (A shorter vs B longer) reports a different line number than the reversed comparison, even though the mismatch occurs on the same comparison round.

Recommended Fix

// sim/src/meta.rs
if read_a > 0 || read_b > 0 {
    lines += 1;
}

History

This bug was introduced in commit b00f2c8. The initial implementation incremented lines only when stream A advanced (if read_a > 0 { lines += 1 }), which causes misleading line numbers when stream A ends before stream B.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions