v0.6.5

CLI hygiene + store-aware doctor

Patch release hardening the evidence-preservation consumer surface. No verdict, perturbation, invariant, or spec-language change.

Changed

• Read-only commands stay off the model stack (#84, #87) — doctor, verify, replay, inspect, diff, history, timeline, matrix, export no longer import litellm. Lazy command dispatch + PEP 562 deferral of LiteLLMAdapter; a subprocess-per-module meta-guard enforces it and forbids read-only commands from re-importing the verdict resolver.
• Store-aware doctor (#85) — resolves the store scheme --store-path selects, reports registered backends, fails (exit 3) when none is registered, and write-probes only the built-in SQLite store (plugin stores reported, never constructed).

Documentation & Tests

• Evidence-protocol + architecture doc-freshness tripwires (#90, #92, #94); consumer-side verdict-map coverage guard; agent-context correction (#86).

Full changelog: https://github.com/ericckzhou/falsifyai/blob/main/CHANGELOG.md

v0.6.4

paraphrase validity fix (reject lossy rewrites under --nli)

Patch release closing a generation-layer self-falsification (case study 06).

The paraphrase validity gate used embedding cosine similarity — which preserves topic but not task completeness — so an llm_rewrite that deleted a task's grounding while keeping its vocabulary passed the gate, drove the model to refuse, and the refusal scored as a stable failure: a manufactured CONSISTENTLY_WRONG @ 0.00 over a correct model. The BidirectionalNLIValidator (plan.md §9.3) — entailment in both directions — now rejects such lossy rewrites under --nli.

Completes the self-falsification trilogy: 03 interpretation · 05 presentation · 06 generation.

--nli-less runs are byte-identical; resolver untouched. Full notes in CHANGELOG.

v0.6.3 - confidence-label inversion fix + store-plugin plumbing

Patch release. Headline: the presentation-layer confidence-label inversion (case study 05) is fixed across every consumer surface (run/replay/inspect render 'stability floor:'; history drops the redundant unlabeled number; matrix/timeline audited clean). Resolver and stored artifacts byte-identical. Also ships the additive falsifyai.stores plugin group. See CHANGELOG.md [0.6.3].

v0.6.2

schema_match JSON-extraction fix

Patch release. Hardens the schema_match invariant against a false structural failure surfaced by dogfooding case study 03 — correct JSON wrapped in a markdown fence or embedded in prose was being scored as a shape failure. No new fields, verdicts, or spec-language changes; extraction never relaxes the strict schema check.

Fixed

• schema_match extracts JSON before validating. Previously ran json.loads on the entire model output, so fenced (json … ) or prose-embedded JSON failed. It now extracts the JSON value (whole string → first fenced block → first balanced {/[ via raw_decode) before the unchanged strict schema check. Finding 3 of case study 03.

Also in this release (docs)

• Case study 03 — self-falsification (evaluator false positive)
• Case study 04 — overconfident negation (genuine CONSISTENTLY_WRONG)

Full changelog: https://github.com/ericckzhou/falsifyai/blob/main/CHANGELOG.md

0.6.1

Hallucination-oracle NEUTRAL=abstain fix

Patch release. Corrects a false-positive in the NLI HallucinationOracle surfaced by dogfooding the probe-03 "confidently wrong" bake-off — where all five candidate model outputs were in fact correct, yet one was being scored as a confident falsehood. No new fields, verdicts, or spec-language changes; default (non---nli) behavior is byte-identical to 0.6.0.

Fixed

• HallucinationOracle treats NLI NEUTRAL as abstain, not wrong. An output the NLI backend can neither entail nor contradict against the ground truth (relation NEUTRAL) is unsupported, not false — it no longer contributes a spurious CONSISTENTLY_WRONG signal. Previously a correct answer whose phrasing diverged from the reference enough to read as NEUTRAL was mislabeled a hallucination. The oracle now fires only on genuine CONTRADICTION. Regression tests pin the NEUTRAL→abstain boundary. See docs/case-studies/probe-03/RESULTS.md (Finding 1) for the discovery and replay artifact.

Full changelog: https://github.com/ericckzhou/falsifyai/blob/v0.6.1/CHANGELOG.md

0.6.0

Semantic-judgment depth (NLI + full 8-verdict resolver)

Semantic-judgment depth. Deepens the oracle layer with natural-language inference (NLI) and completes the 8-verdict taxonomy the 5-verdict MVP deferred. The NLI machinery is an opt-in extra (pip install "falsifyai[nli]"); the default install and the 5-verdict behavior are unchanged, so existing specs and replay artifacts read identically. The four new verdicts are reachable only when grounding context and/or the NLI oracles are supplied.

Added

• NLI backend primitive — NLIBackend Protocol with bidirectional entailment/contradiction scoring. MockNLIBackend (deterministic, dependency-free) backs tests and default behavior; TransformersNLIBackend ships behind the opt-in [nli] extra, lazy-loaded so the model downloads only on the first classify() call, never at construction.
• Semantic oracles — GroundingOracle (answer supported by provided context → INFORMATION_PRESENT), HallucinationOracle (confident claim contradicted by ground truth → CONSISTENTLY_WRONG), and ContradictionOracle (self-inconsistency across the output set, with a vs-reference path). Aggregation helpers reduce per-output NLI labels to a single oracle signal.
• Full 8-verdict resolver — adds INFORMATION_PRESENT, INFORMATION_NULL, ADVERSARIALLY_VULNERABLE, and AMBIGUOUS to the prior five, completing the 2-D verdict space. RAG-style grounding context is carried on OracleContext; failure-shape classification feeds the new branches. CLI exit codes map all eight verdicts.
• falsifyai run --nli — opt-in flag that constructs the NLI backend and activates the semantic oracles for a run. Purely additive: it adds grounding/hallucination evidence and can surface a richer verdict, but never turns a passing case into a failing one on its own.

Notes

• Resolver verdict-branch count moved 5 → 9 (the four new verdict classes), still guarded by tests/meta/test_resolver_branch_count.py. This is the planned completion of the taxonomy, not resolver inflation — oracles continue to pre-arbitrate before the resolver.
• Default (no --nli) runs require no new dependencies and produce the same verdicts as 0.5.0; the heavyweight transformers + torch stack is pulled only by the [nli] extra.

0.5.0

Capability-breadth track. Closes the Phase 1 capability gaps the artifact-infrastructure track (0.2–0.4) skipped: the semantic-judgment (oracle) layer, byte-level adversarial perturbation, structural assertion, extensibility, and cross-run/cross-model analytics. Spec language is a superset of 0.4.0 (new perturbation/invariant/plugin spec types); the 5-verdict set and replay format are unchanged (a new invalid_eval_count field on SessionVerdict defaults for backward-compatible reads).

Added

• unicode perturbation family (ADVERSARIAL category) — visually-identical, byte-different input: invisible space variants (incl. U+202F), zero-width characters, Cyrillic/Greek homoglyphs. The generation-side complement to case study 01; FalsifyAI now generates the failure it could previously only detect.
• schema_match invariant — strict structural assertion that output is valid JSON conforming to a declared schema (top-level type, required keys, typed properties), over stdlib json with no new runtime dependency.
• Oracle layer — Oracle Protocol + OracleVerdict + OracleContext (the semantic-judgment surface), and a real ConsistencyOracle (ground-truth contradiction + optional embedding-agreement signal).
• MetaOracle — the sole, rigorous source of INVALID_EVAL: invariant degeneration (an invariant failing >95% of outputs including the clean baseline) and oracle conflict. Guarded by a resolver branch-count meta-test so oracles pre-arbitrate rather than inflating the verdict resolver.
• Entry-point plugin system — perturbations and invariants are extensible without forking via the falsifyai.perturbations / falsifyai.invariants entry-point groups and a generic {type: plugin, name, params} spec; built-ins are registered through the same mechanism (dogfooded).
• falsifyai matrix — cross-model reliability profiles: N sessions × perturbation families, each cell the model's worst-case stability in that family.
• falsifyai timeline — longitudinal robustness trend for one case (chronological stability_ci_low sparkline) with regression detection; exit 5 on a verdict-class downgrade.
• falsifyai minimize — minimal-falsifier search: the smallest perturbation strength that flips a case out of STABLE.

Notes

• Resolver verdict-branch count moved 4 → 5 (the new INVALID_EVAL class), guarded by tests/meta/test_resolver_branch_count.py. Adding an oracle must not grow it.
• Consumer surfaces (matrix, timeline) are forbidden from importing the resolver (enforced by meta-tests); minimize is an orchestrator and legitimately resolves.

v0.4.0

0.4.0 — Artifact-infrastructure track complete

Artifact-infrastructure track complete (3 of 3 locked items shipped).
The locked sequence verify → export --bundle → embedded CLI invocation
is now closed. After v0.4.0, the artifact answers four questions without
external bookkeeping:

Question	Source
What happened	case results + verdict
How it was evaluated	materialized spec + invariants
What was exported	bundle manifest + bundle_id
What exact command produced it	new cli_invocation

What's new

• Persisted cli_invocation on ReplayArtifact — descriptive procedural
  provenance. CliInvocation is a frozen dataclass with two fields: argv
  (normalized — argv[0] canonicalized to "falsifyai" regardless of entry
  path) and falsifyai_version (runtime version at capture time). Captured
  exactly once at entry to cmd_run; read-only consumer surfaces never stamp
  invocation.

• Bundle README "Generated by" section now renders — the conditional
  render path PR-32 added speculatively lights up automatically. Includes an
  explicit semantic-boundary disclaimer: records what command produced the
  artifact, not a guarantee that re-running will produce identical outputs.
  Replay-determinism guarantees still live in materialized_hash and
  bundle_id.

Backward compatibility

• Pre-v0.4.0 artifacts carry cli_invocation = None and load cleanly
• verify does NOT gain a 9th check (would break existing artifacts)
• Bundle format is unchanged; the new field is an additive optional on
  artifact.json

Install

pip install falsifyai==0.4.0

Full diff

v0.3.0...v0.4.0

v0.3.0

0.3.0 — Artifact-infrastructure track

Artifact-infrastructure track (2 of 3 locked items shipped). New verify and
export --bundle consumer surfaces, plus diff sharpening for CI gating.
EU AI Act Annex IV compliance mapping documented. Case study 02 adds a
methodologically restrained second exemplar.

Spec language and verdict semantics remain unchanged from 0.1.0; every new
surface is a reader of preserved evidence, never a producer of new verdicts.

What's new

• falsifyai diff sharpening — --strict (exits 5 on same-verdict
  confidence drop ≥ 0.10; exits 6 on candidate falsifiability < 0.50) and
  --show-timeline (per-row direction markers). Default output stays
  byte-identical to v0.2.0 (fixture-enforced).
• falsifyai verify <session_id> — 8-check artifact integrity validation.
  Materialized-hash recomputation, session-verdict roll-up consistency, CI
  bound ordering, falsifiability score range. Exit 7 on failure.
• falsifyai export <session_id> --bundle <output>.fai.zip — deterministic
  content-addressed portable evidence bundle. Manifest carries bundle_id
  (sha256 over canonical manifest) + per-file SHA256s + reserved
  attestations: [] / signature_slots: [] for future signing.
• Case study 02 — Resolver arbitration: boundary shift without verdict
  shift. Demonstrates drift that pass/fail evaluators would miss.
• EU AI Act Annex IV compliance mapping — docs/COMPLIANCE.md maps each
  §2(g) requirement to a specific ReplayArtifact field or CLI command.

Exit codes

Code	Meaning
0	SUCCESS
5	REGRESSION (diff verdict-class downgrade or --strict confidence drop)
6	LOW_FALSIFIABILITY (diff --strict)
7	INTEGRITY_FAILURE (verify — new in 0.3.0)

Install

pip install falsifyai==0.3.0

Full diff

v0.2.0...v0.3.0

0.2.0

Phase 1 first wave. Three new consumer surfaces, one new perturbation family, the project's first canonical case study, and automated PyPI publishing — all shipped together as 0.2.0.

The artifact format, the spec language, and the resolver behavior for run / replay / diff are unchanged from 0.1.0. Every new surface is a reader of preserved evidence, never a producer of new verdicts.

Highlights

• falsifyai inspect <session_id> — per-case deep-dive over a stored session's preserved evidence. Surfaces every perturbed input, output, and invariant judgment. --case <case_id> expands one case; --full disables truncation. Pure consumer surface — the artifact already contained the data.

• falsifyai history <case_id> — temporal view of one case across saved sessions. Newest-first, one row per session, showing verdict + CI + worst perturbation family. Reads case.verdict from preserved artifacts; never re-resolves, never aggregates, never infers trends.

• paraphrase perturbation family — LLM-driven semantic-preserving rewrites with embedding-similarity validity gating. First semantic pressure axis, orthogonal to the existing character-level families (typo_noise, casing_variant). Configurable per-spec (count, similarity_threshold, max_attempts, optional model override).

• Canonical case study: Invisible character substitution — worked tour of the evidence infrastructure over real preserved artifacts from the Phase 0 validation campaign. Cross-model contains-contract brittleness as thesis (history); Pair 3 model-migration regression (U+202F substitution between "30" and "days") as vivid concrete proof (diff + inspect). Ships with a bundled ReplayStore — every command shown reproduces verbatim.

• Automated PyPI publishing via Trusted Publisher (OIDC) — .github/workflows/publish.yml fires on v* tag push. No long-lived API tokens in the repo. This release was published via that pipeline.

Install

pip install falsifyai==0.2.0

# For the semantic_equivalence invariant and paraphrase validity gating:
pip install "falsifyai[semantic]==0.2.0"