Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
205 lines (137 loc) · 8.76 KB

ARCHITECTURE.md

File metadata and controls

205 lines (137 loc) · 8.76 KB
Raw

Evidence Review Assistant Architecture

README · Demo guide · Build Week changelog

Scope

The Evidence Review Assistant is an isolated Build Week extension layered beside the Crypto Intelligence Terminal. It reviews bounded evidence and returns explanatory data. It is not imported by the baseline terminal and has no route to order placement, position management, exchange APIs, production databases, or Telegram delivery.

System flow

flowchart TD
    A["Synthetic TradeEvidenceInput"] --> B["Pydantic schema validation"]
    B --> C["Semantic validation"]
    C --> D["Deterministic normalization"]
    D --> E["Rule engine"]
    E --> F{"Provider configuration"}
    F -->|"MOCK"| G["MockReviewProvider"]
    F -->|"OPENAI"| H["OpenAIProvider"]
    H --> I["Bounded prompt builder"]
    I --> J["Responses API structured output"]
    J --> K["Pydantic and citation validation"]
    G --> L["EvidenceReviewService"]
    K --> L
    L --> M["EvidenceReviewOutput"]
Loading

The service owns orchestration and final output assembly. Providers contribute only interpretation items.

Evidence pipeline

1. Input contract

TradeEvidenceInput is a strict Pydantic model. Unknown fields are rejected. It captures:

  • trade identity and direction;
  • entry, exit, stop, target, and duration;
  • market regime and technical context;
  • execution measurements such as planned entry, actual entry, slippage, MFE, and MAE;
  • bounded derivatives context;
  • evidence quality, completeness, provenance, and stable evidence IDs.

P4 submission behavior remains synthetic-only: synthetic must be true and provenance must equal OPENAI_BUILD_WEEK_SYNTHETIC_FIXTURE.

2. Semantic validation

evidence_normalizer.py rejects structurally valid but impossible evidence, including:

  • non-positive prices;
  • exit times before entry times;
  • contradictory holding duration;
  • invalid long/short stop and target geometry;
  • out-of-range RSI, ADX, choppiness, completeness, and quality values; and
  • negative excursion magnitudes.

Semantic failures occur before either provider is called.

3. Normalization

The normalizer standardizes symbols, directions, timeframes, timestamps, and categorical values. It calculates bounded derived values such as duration, slippage, important-field completeness, and a canonical JSON representation used for stable review IDs.

It preserves original values separately and does not fabricate missing evidence.

Rule engine

review_rules.py is deterministic and network-free. It produces distinct collections for:

  • observed facts;
  • rule-based inferences;
  • supporting evidence;
  • opposing evidence;
  • execution-quality observations; and
  • evidence gaps.

Implemented rules cover immediate stop behavior, planned-versus-actual entry degradation, target-to-stop relationships, direction-versus-trend consistency, ADX trend strength, choppy conditions, breakout/retest evidence, favorable excursion, adverse excursion, and low completeness.

Every ReviewItem includes a statement, origin, cited evidence IDs, and uncertainty from 0 to 1. Rule language is intentionally cautious and does not assert causation.

Provider abstraction

BaseReviewProvider is a structural protocol:

name: str
model_identifier: str
interpret(NormalizedEvidence, RuleEvaluation) -> list[ReviewItem]

EvidenceReviewService depends on this interface. provider_config.py selects one of two implemented values:

  • MOCK
  • OPENAI

Provider selection defaults to MOCK and reads no API credential.

MockReviewProvider

The mock provider is deterministic and performs no network activity. It uses only normalized evidence and deterministic rule output. Its business content is stable for identical input, and its items use ReviewOrigin.MOCK_PROVIDER.

The mock provider is the default for tests, documentation examples, and the standalone API application.

OpenAIProvider

The OpenAI provider uses the installed Python SDK and the Responses API.

Configuration lifecycle

  • OPENAI_MODEL is read when the provider is instantiated.
  • OPENAI_TIMEOUT is read and validated when the provider is instantiated.
  • OPENAI_API_KEY is read only when the first OpenAI review actually needs to create the lazy client.
  • Imports and normal application startup do not require a credential.

One internal model default is defined in openai_provider.py; an environment value overrides it. Timeout must be a positive number and defaults to 30 seconds.

Request boundary

prompt_builder.py sends only an explicit subset of normalized evidence plus observed facts, rule summaries, completeness, and gaps. It does not send original values, canonical JSON, application state, database content, Telegram content, or execution instructions.

The request:

  • calls client.responses.parse;
  • supplies a Pydantic text_format;
  • requests no tools;
  • sets store=False;
  • applies a bounded output-token limit and timeout; and
  • instructs the model to return only structured JSON, cite known evidence, avoid advice, avoid causation, and perform no action.

Response validation

The SDK response must parse into OpenAIInterpretationResponse. The provider then verifies that every cited evidence ID was present in the rule/observation input. The model does not supply provenance; the application assigns ReviewOrigin.OPENAI_PROVIDER after validation.

Timeout, authentication, rate-limit, invalid-response, configuration, and unexpected failures become safe ProviderFailure categories. SDK response bodies and exception details are not exposed through the review output.

Review output flow

EvidenceReviewService creates the stable review ID before provider execution, measures provider duration, and maps the result into EvidenceReviewOutput.

The output separates:

  • observed_facts;
  • rule_based_inferences; and
  • provider_interpretations.

It also contains supporting/opposing evidence, execution quality, likely factors, evidence gaps, confidence, alternative explanations, next-review actions, provider/model identifiers, schema version, generation time, and a safety statement.

Insufficient evidence bypasses the provider. Provider failure returns PROVIDER_ERROR with zero confidence and no provider interpretations; it does not mutate any external state.

Logging model

The service logs one safe execution record containing:

  • provider name;
  • model name;
  • stable review ID;
  • execution time in milliseconds; and
  • success or failure.

It never logs API keys, prompts, normalized evidence, response content, trade IDs, symbols, or SDK exception details.

Safety model

The safety design uses multiple independent boundaries:

  1. Input boundary: strict schema, semantic validation, synthetic provenance.
  2. Inference boundary: observations, rules, and provider interpretations remain distinguishable.
  3. Citation boundary: provider citations must reference known evidence IDs.
  4. Provider boundary: a narrow interface returns review items only.
  5. Request boundary: bounded input, no tools, structured output, no storage.
  6. Failure boundary: provider exceptions become controlled review failure state.
  7. Import boundary: the extension imports no backend module.
  8. Test boundary: OpenAI success and failure paths use injected fake SDK clients.

Why the assistant cannot execute trades

The assistant cannot execute trades because execution capability is absent by construction:

  • the provider interface has only interpret;
  • the Responses API request supplies no tools or function handlers;
  • the extension imports no terminal backend, exchange client, database writer, Telegram module, or execution engine;
  • EvidenceReviewService returns a Pydantic review object and performs no command dispatch;
  • the standalone API exposes review, schema, health, and allowlisted sample routes only; and
  • isolation tests statically scan imports and verify operation when socket construction is denied for the mock path.

Adding execution would require new imports, interfaces, tools, tests, permissions, and production wiring outside the approved architecture.

Implemented, limited, and future

Implemented

  • deterministic normalization and rules;
  • mock and OpenAI provider abstraction;
  • lazy OpenAI client and structured Responses API path;
  • safe configuration, timeout, failure mapping, and logging;
  • six synthetic scenarios and offline unit tests.

Current limitations

  • synthetic-only evidence intake;
  • no dashboard integration or persistence;
  • live provider availability depends on separately supplied runtime credentials and model access;
  • automated tests do not send live requests.

Future work

Dashboard integration, persistent review history, authentication, deployment, broader evaluation data, and human-feedback workflows are not implemented.