feat: real-time console UX redesign with phase banners, verbose flags, and structured output#111
Open
ElNiak wants to merge 14 commits intoproductionfrom
Open
feat: real-time console UX redesign with phase banners, verbose flags, and structured output#111ElNiak wants to merge 14 commits intoproductionfrom
ElNiak wants to merge 14 commits intoproductionfrom
Conversation
…d classes Track 1 - God class splits: - Split DockerBuilder into platform_detection, buildx_operations, image_operations - Split MetricsCollector into metric_types, metric_analysis - Split ExperimentManager into experiment_phases, experiment_cleanup Track 2 - Event/observer simplification: - Replace ~130 event subclasses with factory classmethods on 8 base classes - Simplify ITypedObserver from 881 to ~80 lines with dynamic dispatch - Remove all backward-compat alias functions from events - Clean up events/__init__.py from 404 to ~100 lines - Update all emitters to use factory methods directly Track 3 - Validator cleanup: - Remove dead validator exports from pydantic_factories.py - Simplify validators/__init__.py (remove __getattr__ machinery) Net reduction: ~5,600 lines across 40 files. 870 tests passing.
…, and mixin inlining Wire hidden core modules to CLI (report, build-metrics, debug commands), delete orphaned tools and unused mixins, inline single-use mixins into their consumers, replace empty protocol placeholders with intermediate base classes (ClientServerProtocolBase, PeerToPeerProtocolBase). Net: -1,746 LOC, 13 files deleted, 5 created, 10 new CLI commands.
Add LogContext (contextvars-based) and StructuredJsonFormatter that replaces flat text file logs with structured JSONL. Every log record now carries experiment_id, test_id, service_id, phase context automatically via contextvars — zero changes to call sites needed. - New: log_context.py (LogContext dataclass + context manager) - New: structured_formatter.py (JSONL logging.Formatter) - Modified: logger_factory.py (structured file handler replaces text) - Modified: global_config.py (removed debug_file_logging, added structured_log_file) - Removed: debug_file_logging references from configs and experiment_manager
Wire LogContext into experiment lifecycle boundaries so that every log record and event emitted during a phase/test/service carries the relevant identifiers (experiment_id, test_id, service_id, phase). Context propagation locations: - ExperimentManager.initialize_experiments: experiment_id + phase - ExperimentManager.run_tests: experiment_id + test_execution phase - ExperimentManager.run_tests per-test loop: test_id - ExperimentManager.cleanup: experiment_id + cleanup phase - TestCase.run: test_id (via manual __enter__/__exit__) - ServiceManagementMixin: service_id + phase for setup_testers, setup_implementations, prepare_services, teardown_services New EventStreamRecorder observer: - Appends every event to the same structured.jsonl used by StructuredJsonFormatter, interleaving events and log records chronologically in a single timeline file - Uses "source": "event" and "level": "EVENT" to distinguish from logging records - Reads LogContext for experiment/test/service/phase fields - Thread-safe file writes via threading.Lock - Registered as a global observer in ExperimentObserverMixin with low priority (10) so business observers run first Also: - Register EventStreamRecorder in observer factory as "event_stream" - Add deprecation note to StorageObserver about separate JSONL files now being redundant with structured.jsonl (kept for backward compat) - 12 new unit tests covering schema, context propagation, deduplication, thread safety, error handling
Streaming JSONL log query engine (LogQueryEngine, LogFilter) that reads structured.jsonl files without loading them fully into memory. Supports filtering by level, service, test, phase, time range, correlation ID, message regex, and source. CLI commands: panther logs query <dir> --level/--service/--test/--phase/... panther logs errors <dir> (shorthand for --level ERROR,CRITICAL) Both commands support --json (default JSONL) and --human (table) output. 35 unit tests cover all filter combinations, file discovery, malformed input handling, and the count() aggregation.
Create an incremental, thread-safe output index that tracks every file produced during an experiment and serializes it as output_index.json. - New: panther/core/outputs/output_index.py with OutputIndexBuilder, FileEntry, and detect_type_and_format helper - Integrate into ExperimentManager: init at startup, register known files (config, reports, metrics, structured log), flush at cleanup - Extend OutputAggregator with optional output_index_builder param to register collected environment outputs in the manifest - 35 unit tests covering serialization, auto-detection, dedup, thread safety, and aggregator integration
…mands Add TimelineRenderer for chronological swimlane display of structured log entries, and ArtifactBrowser for listing/filtering experiment output files from output_index.json (with directory walk fallback). Register both as `panther report timeline` and `panther report artifacts` CLI subcommands with filtering by test, service, time range, phase, and artifact type.
…matching Add declarative FailurePattern library (8 built-in patterns derived from ErrorCategory) and RootCauseAnalyzer that reads structured.jsonl via LogQueryEngine to produce ranked root-cause findings with log excerpts and actionable suggestions. Extend ExperimentReporter with optional RCA and artifact inventory sections in both JSON and Markdown reports (gracefully skipped when structured.jsonl is absent). Add `panther report diagnose` CLI subcommand with --json/--human output.
Replace the raw colorlog/logging.Formatter in LoggerFactory with a new ConsoleFormatter that provides: - Short HH:MM:SS timestamps (no date for live runs) - Phase/service context from LogContext: (phase|service_id) prefix - Abbreviated module names (strips panther.core. prefix) - Static banner() method for phase separators via click.echo() Wire phase banners into ExperimentManager at each transition: initialization, plugin loading, test execution, and cleanup.
Add LoggerFactory.set_console_level() classmethod that updates all StreamHandler instances (root + named loggers) to a given level while keeping FileHandlers at DEBUG for structured JSONL output. Wire --verbose (DEBUG) and --debug (TRACE) Click options through the run command into ExperimentManager via a new console_level parameter, giving CLI flags final precedence over YAML config. Includes 8 unit tests covering level updates, FileHandler preservation, TRACE support, and idempotency.
Verify format_test_header() and format_test_result() output formatting, emoji control, timing precision, and padding behavior.
Add format_experiment_summary() that displays a structured banner after all tests complete, showing pass/fail counts with percentage, duration, output directory, failed test details, and next-step CLI commands. Replaces the simple one-line completion message.
DockerOutputParser now adapts to console verbosity: concise one-liner step labels by default, full instruction text when --verbose/--debug is active. Download/extract progress is suppressed in concise mode. Build start, completion (with duration), and cache-skip messages use click.echo() for consistent structured output.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
✅ PR Validation Passed Your changes look good! The quick validation checks have passed:
The full CI pipeline will run when this PR is merged or when targeting the main branches. |
Contributor
There was a problem hiding this comment.
Pull request overview
Redesigns PANTHER’s live console/logging UX by introducing context-aware console formatting, unified structured JSONL logging (logs + events), verbosity controls, and new post-run reporting/query tooling built on the structured log stream.
Changes:
- Added structured logging primitives (
LogContext,StructuredJsonFormatter,EventStreamRecorder) and wired them intoLoggerFactory/ experiment observer setup. - Introduced console UX improvements via
ConsoleFormatterand CLI verbosity flags (--verbose,--debug) plus new CLI commands for log querying and debugging. - Added reporting utilities (log query engine, timeline renderer, root-cause analyzer, artifact browser) and output indexing support.
Reviewed changes
Copilot reviewed 79 out of 128 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| panther/plugins/environments/environment_plugin_mixin.py | Removed environment plugin mixin (common init/state helpers). |
| panther/core/utils/structured_formatter.py | Adds JSONL formatter for log records with context fields. |
| panther/core/utils/logging_mixin.py | Refactors feature-aware logging into LoggerMixin and adds helper methods. |
| panther/core/utils/logger_factory.py | Wires ConsoleFormatter, structured JSONL file output, and console-level override. |
| panther/core/utils/log_context.py | Adds contextvars-based LogContext and log_context() manager. |
| panther/core/utils/feature_logger_mixin.py | Replaces mixin class with get_feature_logger() helper. |
| panther/core/utils/console_formatter.py | Adds context-aware console formatter and banner printing helper. |
| panther/core/utils/init.py | Updates utils public API exports for new logging utilities. |
| panther/core/test_cases/test_case_impl.py | Pushes per-test LogContext during test execution. |
| panther/core/test_cases/mixins/service_management.py | Adds service/phase LogContext around service setup/teardown calls. |
| panther/core/reporting/timeline_renderer.py | Adds timeline renderer based on structured JSONL queries. |
| panther/core/reporting/root_cause_analyzer.py | Adds RCA engine using failure patterns + structured log queries. |
| panther/core/reporting/log_query_engine.py | Adds streaming JSONL query/filter engine. |
| panther/core/reporting/failure_patterns.py | Adds built-in failure-pattern catalog for RCA. |
| panther/core/reporting/experiment_reporter.py | Integrates optional RCA + artifact inventory into reports. |
| panther/core/reporting/artifact_browser.py | Adds artifact browsing via output index or directory walk fallback. |
| panther/core/reporting/init.py | Exposes new reporting utilities in package API. |
| panther/core/outputs/output_index.py | Adds thread-safe output manifest builder (output_index.json). |
| panther/core/outputs/output_aggregator.py | Optionally registers collected outputs into the output index. |
| panther/core/outputs/init.py | Exports OutputIndexBuilder. |
| panther/core/observer/workflow/init.py | Minor docstring cleanup. |
| panther/core/observer/impl/storage_observer.py | Updates typed handlers to BaseEvent and documents structured.jsonl coexistence. |
| panther/core/observer/impl/state_observer.py | Updates typed handlers to accept BaseEvent. |
| panther/core/observer/impl/metrics_observer.py | Updates typed handlers and imports to align with event refactor. |
| panther/core/observer/impl/logger_observer.py | Updates typed handlers to accept BaseEvent. |
| panther/core/observer/impl/experiment_observer.py | Switches event dispatching to (entity_type, name) tuple routing. |
| panther/core/observer/impl/event_stream_recorder.py | Adds observer recording events into structured JSONL stream. |
| panther/core/observer/impl/command_audit_observer.py | Refactors event typing; changes supported event types registration behavior. |
| panther/core/observer/impl/init.py | Exports EventStreamRecorder. |
| panther/core/observer/factory.py | Registers event_stream observer type. |
| panther/core/metrics/resource_monitor.py | Docstring cleanup. |
| panther/core/metrics/metric_types.py | Adds shared metric dataclasses. |
| panther/core/metrics/enums.py | Docstring cleanup. |
| panther/core/experiment_observer.py | Registers EventStreamRecorder during experiment observer setup. |
| panther/core/events/test/emitter.py | Migrates test emitter to factory methods on TestEvent. |
| panther/core/events/test/init.py | Shrinks exports to consolidated event types. |
| panther/core/events/step/events.py | Replaces many step event subclasses with factory classmethods on StepEvent. |
| panther/core/events/step/emitter.py | Migrates step emitter to StepEvent factories. |
| panther/core/events/step/init.py | Shrinks exports to consolidated step event types. |
| panther/core/events/service/emitter.py | Migrates service emitter to ServiceEvent factories (keeps Docker build events). |
| panther/core/events/service/init.py | Shrinks exports to consolidated service event types. |
| panther/core/events/plugin/emitter.py | Migrates plugin emitter to PluginEvent factories. |
| panther/core/events/plugin/init.py | Shrinks exports to consolidated plugin event types. |
| panther/core/events/experiment/events.py | Migrates experiment events to factories; keeps a compatibility subclass. |
| panther/core/events/experiment/emitter.py | Migrates experiment emitter to ExperimentEvent factories. |
| panther/core/events/experiment/init.py | Shrinks exports to consolidated experiment event types. |
| panther/core/events/environment/emitter.py | Migrates environment emitter to EnvironmentEvent factories. |
| panther/core/events/environment/init.py | Shrinks exports to consolidated environment event types. |
| panther/core/events/assertion/events.py | Migrates assertion events to factory classmethods. |
| panther/core/events/assertion/emitter.py | Migrates assertion emitter to AssertionEvent factories. |
| panther/core/events/assertion/init.py | Shrinks exports to consolidated assertion event types. |
| panther/core/events/init.py | Updates root events API exports to consolidated types. |
| panther/core/docker_builder/plugin_mixin/service_manager_docker_mixin.py | Adds explicit console echoes for cached-image skips. |
| panther/core/docker_builder/platform_detection.py | Adds platform/buildx detection + build-mode validation mixin. |
| panther/core/command_processor/mixins/modification_mixin.py | Removes command modification mixin module. |
| panther/core/command_processor/mixins/init.py | Removes CommandModificationMixin export. |
| panther/core/command_processor/init.py | Updates docs/exports to reflect mixin removal. |
| panther/config/core/validators/init.py | Switches from lazy __getattr__ to explicit imports/exports. |
| panther/config/core/models/global_config.py | Replaces debug_file_logging with structured_log_file config. |
| panther/cli/core/main.py | Registers new CLI command groups (report/debug/logs/build-metrics). |
| panther/cli/commands/run.py | Adds --debug flag and forwards computed console level to manager. |
| panther/cli/commands/logs.py | Adds structured log querying CLI (logs query/errors). |
| panther/cli/commands/debug.py | Adds developer introspection CLI (debug events/observers). |
| panther/cli/commands/build_metrics.py | Adds CLI for viewing/exporting build/test metrics from JSONL storage. |
| experiment-config/base/experiment_config_example_minimal_docker_no_buildx.yaml | Removes deprecated debug_file_logging config key. |
| experiment-config/base/experiment_config_example_minimal_docker.yaml | Removes deprecated debug_file_logging config key. |
| experiment-config/base/experiment_config_example_minimal.yaml | Removes deprecated debug_file_logging config key. |
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+1
to
5
| """Logging mixin with feature-aware capabilities for PANTHER components.""" | ||
|
|
||
| from typing import Any, Dict, Optional, Union | ||
|
|
||
| """ |
Comment on lines
29
to
31
| def __init__(self, *args, **kwargs): | ||
| """Initialize mixin state.""" | ||
| super().__init__(*args, **kwargs) |
| if ctx.service_id: | ||
| parts.append(ctx.service_id) | ||
| if parts: | ||
| return f"({' | '.join(parts)}) " # note trailing space |
| """ | ||
| # Inject extra fields that the format string references | ||
| record.context_tag = _build_context_tag() # type: ignore[attr-defined] | ||
| record.short_module = _short_module(record.module) # type: ignore[attr-defined] |
Comment on lines
+196
to
+198
| # Push log context for this test; will be popped in the finally block | ||
| _ctx_token = log_context(test_id=self.test_name) | ||
| _ctx_token.__enter__() |
Comment on lines
+80
to
+83
| line = json.dumps(record, default=str) | ||
| with self._lock: | ||
| with open(self.output_path, "a", encoding="utf-8") as fh: | ||
| fh.write(line + "\n") |
Comment on lines
+173
to
+184
| event_filter = LogFilter(sources={"event"}) | ||
| for record in self._engine.query(event_filter): | ||
| message = record.get("message", "") | ||
| event_type = record.get("event_type", "") | ||
| # Include events that look error-related | ||
| if any( | ||
| kw in (message + " " + event_type).lower() | ||
| for kw in ("error", "fail", "crash", "timeout", "killed") | ||
| ): | ||
| # Avoid duplicates | ||
| if record not in error_records: | ||
| error_records.append(record) |
Comment on lines
+54
to
+56
| def get_supported_event_types(self) -> list: | ||
| """Return the list of event types this observer handles.""" | ||
| return [ | ||
| CommandGenerationStartedEvent, | ||
| CommandGeneratedEvent, | ||
| CommandModifiedEvent, | ||
| ConfigGeneratedEvent, | ||
| ] | ||
|
|
||
| def handle_command_generation_started( | ||
| self, event: CommandGenerationStartedEvent | ||
| ) -> None: | ||
| return [] |
| structured_formatter = StructuredJsonFormatter() | ||
| file_handler = cls._get_or_create_handler( | ||
| "file", logging.FileHandler(output_file, mode="a") | ||
| "file", logging.FileHandler(structured_path, mode="a") |
Comment on lines
+158
to
+163
| if self._should_use_buildx(): | ||
| # Buildx enabled - can cross-compile, use target platform | ||
| return self.get_target_platform() | ||
| else: | ||
| # Buildx disabled - standard Docker builds for native platform only | ||
| host_platform = self._get_host_platform() |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves the real-time console experience during
panther run:HH:MM:SStimestamps, phase/service context tags(initialization|picoquic), and visual phase banners using box-drawing characters--verbose/--debugflags: Runtime console verbosity control —--verboseshows DEBUG,--debugshows TRACE with full Docker outputclick.echo()— shows test headers, pass/fail with timing, and error hintsStep 5/12in normal mode, full commands in verbose mode, with build start/complete messagespanther report diagnose/panther logs errorsBuilds on PR #110 (structured JSONL logging).
Test plan
--verboseand--debugflags show additional detail