Skip to content

Latest commit

 

History

History
992 lines (773 loc) · 28.6 KB

TEST_CASES.md

File metadata and controls

992 lines (773 loc) · 28.6 KB

Test Cases for TrustGraph Microservices

This document provides comprehensive test cases for all TrustGraph microservices, organized by service category and following the testing strategy outlined in TEST_STRATEGY.md.

Table of Contents

  1. Text Completion Services
  2. Embeddings Services
  3. Storage Services
  4. Query Services
  5. Flow Processing
  6. Configuration Management
  7. Data Extraction Services
  8. Retrieval Services
  9. Integration Test Cases
  10. Error Handling Test Cases

Text Completion Services

OpenAI Text Completion (trustgraph.model.text_completion.openai)

Unit Tests

  • test_openai_processor_initialization

    • Test processor initialization with valid API key
    • Test processor initialization with invalid API key
    • Test processor initialization with default parameters
    • Test processor initialization with custom parameters (temperature, max_tokens)

  • test_openai_message_processing

    • Test successful text completion with simple prompt
    • Test text completion with complex multi-turn conversation
    • Test text completion with system message
    • Test text completion with custom temperature settings
    • Test text completion with max_tokens limit
    • Test text completion with streaming enabled/disabled

  • test_openai_error_handling

    • Test rate limit error handling and retry logic
    • Test API key authentication error
    • Test network timeout error handling
    • Test malformed response handling
    • Test token limit exceeded error
    • Test model not found error

  • test_openai_metrics_collection

    • Test token usage metrics collection
    • Test request duration metrics
    • Test error rate metrics
    • Test cost calculation metrics

Claude Text Completion (trustgraph.model.text_completion.claude)

Unit Tests

  • test_claude_processor_initialization

    • Test processor initialization with valid API key
    • Test processor initialization with different model versions
    • Test processor initialization with custom parameters

  • test_claude_message_processing

    • Test successful text completion with simple prompt
    • Test text completion with long context
    • Test text completion with structured output
    • Test text completion with function calling

  • test_claude_error_handling

    • Test rate limit error handling
    • Test content filtering error handling
    • Test API quota exceeded error
    • Test invalid model parameter error

Ollama Text Completion (trustgraph.model.text_completion.ollama)

Unit Tests

  • test_ollama_processor_initialization

    • Test processor initialization with local Ollama instance
    • Test processor initialization with remote Ollama instance
    • Test processor initialization with custom model

  • test_ollama_message_processing

    • Test successful text completion with local model
    • Test text completion with model loading
    • Test text completion with custom generation parameters
    • Test text completion with context window management

  • test_ollama_error_handling

    • Test connection refused error handling
    • Test model not available error
    • Test out of memory error handling
    • Test invalid model parameter error

Azure OpenAI Text Completion (trustgraph.model.text_completion.azure)

Unit Tests

  • test_azure_processor_initialization

    • Test processor initialization with Azure credentials
    • Test processor initialization with deployment name
    • Test processor initialization with API version

  • test_azure_message_processing

    • Test successful text completion with Azure endpoint
    • Test text completion with content filtering
    • Test text completion with regional deployment

  • test_azure_error_handling

    • Test Azure authentication error handling
    • Test deployment not found error
    • Test content filtering rejection error
    • Test quota exceeded error

Google Vertex AI Text Completion (trustgraph.model.text_completion.vertexai)

Unit Tests

  • test_vertexai_processor_initialization

    • Test processor initialization with GCP credentials
    • Test processor initialization with project ID and location
    • Test processor initialization with model selection (gemini-pro, gemini-ultra)
    • Test processor initialization with custom generation config

  • test_vertexai_message_processing

    • Test successful text completion with Gemini models
    • Test text completion with system instructions
    • Test text completion with safety settings
    • Test text completion with function calling
    • Test text completion with multi-turn conversation
    • Test text completion with streaming responses

  • test_vertexai_safety_filtering

    • Test safety filter configuration
    • Test blocked content handling
    • Test safety threshold adjustments
    • Test safety filter bypass scenarios

  • test_vertexai_error_handling

    • Test authentication error handling (service account, ADC)
    • Test quota exceeded error handling
    • Test model not found error handling
    • Test region availability error handling
    • Test safety filter rejection error handling
    • Test token limit exceeded error handling

  • test_vertexai_metrics_collection

    • Test token usage metrics collection
    • Test request duration metrics
    • Test safety filter metrics
    • Test cost calculation metrics per model type

Embeddings Services

Document Embeddings (trustgraph.embeddings.document_embeddings)

Unit Tests

  • test_document_embeddings_initialization

    • Test embeddings processor initialization with default model
    • Test embeddings processor initialization with custom model
    • Test embeddings processor initialization with batch size configuration

  • test_document_embeddings_processing

    • Test single document embedding generation
    • Test batch document embedding generation
    • Test empty document handling
    • Test very long document handling
    • Test document with special characters
    • Test document with multiple languages

  • test_document_embeddings_vector_operations

    • Test vector dimension consistency
    • Test vector normalization
    • Test similarity calculation
    • Test vector serialization/deserialization

Graph Embeddings (trustgraph.embeddings.graph_embeddings)

Unit Tests

  • test_graph_embeddings_initialization

    • Test graph embeddings processor initialization
    • Test initialization with custom embedding dimensions
    • Test initialization with different aggregation methods

  • test_graph_embeddings_processing

    • Test entity embedding generation
    • Test relationship embedding generation
    • Test subgraph embedding generation
    • Test dynamic graph embedding updates

  • test_graph_embeddings_aggregation

    • Test mean aggregation of entity embeddings
    • Test weighted aggregation of relationship embeddings
    • Test hierarchical embedding aggregation

Ollama Embeddings (trustgraph.embeddings.ollama)

Unit Tests

  • test_ollama_embeddings_initialization

    • Test Ollama embeddings processor initialization
    • Test initialization with custom embedding model
    • Test initialization with connection parameters

  • test_ollama_embeddings_processing

    • Test successful embedding generation
    • Test batch embedding processing
    • Test embedding caching
    • Test embedding model switching

  • test_ollama_embeddings_error_handling

    • Test connection error handling
    • Test model loading error handling
    • Test out of memory error handling

Storage Services

Document Embeddings Storage

Qdrant Storage (trustgraph.storage.doc_embeddings.qdrant)

Unit Tests

  • test_qdrant_storage_initialization

    • Test Qdrant client initialization with local instance
    • Test Qdrant client initialization with remote instance
    • Test Qdrant client initialization with authentication
    • Test collection creation and configuration

  • test_qdrant_storage_operations

    • Test single vector insertion
    • Test batch vector insertion
    • Test vector update operations
    • Test vector deletion operations
    • Test vector search operations
    • Test filtered search operations

  • test_qdrant_storage_error_handling

    • Test connection error handling
    • Test collection not found error
    • Test vector dimension mismatch error
    • Test storage quota exceeded error

Milvus Storage (trustgraph.storage.doc_embeddings.milvus)

Unit Tests

  • test_milvus_storage_initialization

    • Test Milvus client initialization
    • Test collection schema creation
    • Test index creation and configuration

  • test_milvus_storage_operations

    • Test entity insertion with metadata
    • Test bulk insertion operations
    • Test vector search with filters
    • Test hybrid search operations

  • test_milvus_storage_error_handling

    • Test connection timeout error
    • Test collection creation error
    • Test index building error
    • Test search timeout error

Graph Embeddings Storage

Qdrant Storage (trustgraph.storage.graph_embeddings.qdrant)

Unit Tests

  • test_qdrant_graph_storage_initialization

    • Test Qdrant client initialization for graph embeddings
    • Test collection creation with graph-specific schema
    • Test index configuration for entity and relationship embeddings

  • test_qdrant_graph_storage_operations

    • Test entity embedding insertion with metadata
    • Test relationship embedding insertion
    • Test subgraph embedding storage
    • Test batch insertion of graph embeddings
    • Test embedding updates and versioning

  • test_qdrant_graph_storage_queries

    • Test entity similarity search
    • Test relationship similarity search
    • Test subgraph similarity search
    • Test filtered search by graph properties
    • Test multi-vector search operations

  • test_qdrant_graph_storage_error_handling

    • Test connection error handling
    • Test collection not found error
    • Test vector dimension mismatch for graph embeddings
    • Test storage quota exceeded error

Milvus Storage (trustgraph.storage.graph_embeddings.milvus)

Unit Tests

  • test_milvus_graph_storage_initialization

    • Test Milvus client initialization for graph embeddings
    • Test collection schema creation for graph data
    • Test index creation for entity and relationship vectors

  • test_milvus_graph_storage_operations

    • Test entity embedding insertion with graph metadata
    • Test relationship embedding insertion
    • Test graph structure preservation
    • Test bulk graph embedding operations

  • test_milvus_graph_storage_error_handling

    • Test connection timeout error
    • Test graph schema validation error
    • Test index building error for graph embeddings
    • Test search timeout error

Graph Storage

Cassandra Storage (trustgraph.storage.triples.cassandra)

Unit Tests

  • test_cassandra_storage_initialization

    • Test Cassandra client initialization
    • Test keyspace creation and configuration
    • Test table schema creation

  • test_cassandra_storage_operations

    • Test triple insertion (subject, predicate, object)
    • Test batch triple insertion
    • Test triple querying by subject
    • Test triple querying by predicate
    • Test triple deletion operations

  • test_cassandra_storage_consistency

    • Test consistency level configuration
    • Test replication factor handling
    • Test partition key distribution

Neo4j Storage (trustgraph.storage.triples.neo4j)

Unit Tests

  • test_neo4j_storage_initialization

    • Test Neo4j driver initialization
    • Test database connection with authentication
    • Test constraint and index creation

  • test_neo4j_storage_operations

    • Test node creation and properties
    • Test relationship creation
    • Test graph traversal operations
    • Test transaction management

  • test_neo4j_storage_error_handling

    • Test connection pool exhaustion
    • Test transaction rollback scenarios
    • Test constraint violation handling

Query Services

Document Embeddings Query

Qdrant Query (trustgraph.query.doc_embeddings.qdrant)

Unit Tests

  • test_qdrant_query_initialization

    • Test query service initialization with collection
    • Test query service initialization with custom parameters

  • test_qdrant_query_operations

    • Test similarity search with single vector
    • Test similarity search with multiple vectors
    • Test filtered similarity search
    • Test ranked result retrieval
    • Test pagination support

  • test_qdrant_query_performance

    • Test query timeout handling
    • Test large result set handling
    • Test concurrent query handling

Milvus Query (trustgraph.query.doc_embeddings.milvus)

Unit Tests

  • test_milvus_query_initialization

    • Test query service initialization
    • Test index selection for queries

  • test_milvus_query_operations

    • Test vector similarity search
    • Test hybrid search with scalar filters
    • Test range search operations
    • Test top-k result retrieval

Graph Embeddings Query

Qdrant Query (trustgraph.query.graph_embeddings.qdrant)

Unit Tests

  • test_qdrant_graph_query_initialization

    • Test graph query service initialization with collection
    • Test graph query service initialization with custom parameters
    • Test entity and relationship collection configuration

  • test_qdrant_graph_query_operations

    • Test entity similarity search with single vector
    • Test relationship similarity search
    • Test subgraph pattern matching
    • Test multi-hop graph traversal queries
    • Test filtered graph similarity search
    • Test ranked graph result retrieval
    • Test graph query pagination

  • test_qdrant_graph_query_optimization

    • Test graph query performance optimization
    • Test graph query result caching
    • Test concurrent graph query handling
    • Test graph query timeout handling

  • test_qdrant_graph_query_error_handling

    • Test graph collection not found error
    • Test graph query timeout error
    • Test invalid graph query parameter error
    • Test graph result limit exceeded error

Milvus Query (trustgraph.query.graph_embeddings.milvus)

Unit Tests

  • test_milvus_graph_query_initialization

    • Test graph query service initialization
    • Test graph index selection for queries
    • Test graph collection configuration

  • test_milvus_graph_query_operations

    • Test entity vector similarity search
    • Test relationship vector similarity search
    • Test graph hybrid search with scalar filters
    • Test graph range search operations
    • Test top-k graph result retrieval
    • Test graph query result aggregation

  • test_milvus_graph_query_performance

    • Test graph query performance with large datasets
    • Test graph query optimization strategies
    • Test graph query result caching

  • test_milvus_graph_query_error_handling

    • Test graph connection timeout error
    • Test graph collection not found error
    • Test graph query syntax error
    • Test graph search timeout error

Graph Query

Cassandra Query (trustgraph.query.triples.cassandra)

Unit Tests

  • test_cassandra_query_initialization

    • Test query service initialization
    • Test prepared statement creation

  • test_cassandra_query_operations

    • Test subject-based triple retrieval
    • Test predicate-based triple retrieval
    • Test object-based triple retrieval
    • Test pattern-based triple matching
    • Test subgraph extraction

  • test_cassandra_query_optimization

    • Test query result caching
    • Test pagination for large result sets
    • Test query performance with indexes

Neo4j Query (trustgraph.query.triples.neo4j)

Unit Tests

  • test_neo4j_query_initialization

    • Test query service initialization
    • Test Cypher query preparation

  • test_neo4j_query_operations

    • Test node retrieval by properties
    • Test relationship traversal queries
    • Test shortest path queries
    • Test subgraph pattern matching
    • Test graph analytics queries

Flow Processing

Base Flow Processor (trustgraph.processing)

Unit Tests

  • test_flow_processor_initialization

    • Test processor initialization with specifications
    • Test consumer specification registration
    • Test producer specification registration
    • Test request-response specification registration

  • test_flow_processor_message_handling

    • Test message consumption from Pulsar
    • Test message processing pipeline
    • Test message production to Pulsar
    • Test message acknowledgment handling

  • test_flow_processor_error_handling

    • Test message processing error handling
    • Test dead letter queue handling
    • Test retry mechanism
    • Test circuit breaker pattern

  • test_flow_processor_metrics

    • Test processing time metrics
    • Test message throughput metrics
    • Test error rate metrics
    • Test queue depth metrics

Async Processor Base

Unit Tests

  • test_async_processor_initialization

    • Test async processor initialization
    • Test concurrency configuration
    • Test resource management

  • test_async_processor_concurrency

    • Test concurrent message processing
    • Test backpressure handling
    • Test resource pool management
    • Test graceful shutdown

Configuration Management

Configuration Service

Unit Tests

  • test_configuration_service_initialization

    • Test configuration service startup
    • Test Cassandra backend initialization
    • Test configuration schema creation

  • test_configuration_service_operations

    • Test configuration retrieval by service
    • Test configuration update operations
    • Test configuration validation
    • Test configuration versioning

  • test_configuration_service_caching

    • Test configuration caching mechanism
    • Test cache invalidation
    • Test cache consistency

  • test_configuration_service_error_handling

    • Test configuration not found error
    • Test configuration validation error
    • Test backend connection error

Flow Configuration

Unit Tests

  • test_flow_configuration_parsing

    • Test flow definition parsing from JSON
    • Test flow validation rules
    • Test flow dependency resolution

  • test_flow_configuration_deployment

    • Test flow deployment to services
    • Test flow lifecycle management
    • Test flow rollback operations

Data Extraction Services

Knowledge Graph Extraction

Topic Extraction (trustgraph.extract.kg.topics)

Unit Tests

  • test_topic_extraction_initialization

    • Test topic extractor initialization
    • Test LLM client configuration
    • Test extraction prompt configuration

  • test_topic_extraction_processing

    • Test topic extraction from text
    • Test topic deduplication
    • Test topic relevance scoring
    • Test topic hierarchy extraction

  • test_topic_extraction_error_handling

    • Test malformed text handling
    • Test empty text handling
    • Test extraction timeout handling

Relationship Extraction (trustgraph.extract.kg.relationships)

Unit Tests

  • test_relationship_extraction_initialization

    • Test relationship extractor initialization
    • Test relationship type configuration

  • test_relationship_extraction_processing

    • Test relationship extraction from text
    • Test relationship validation
    • Test relationship confidence scoring
    • Test relationship normalization

Definition Extraction (trustgraph.extract.kg.definitions)

Unit Tests

  • test_definition_extraction_initialization

    • Test definition extractor initialization
    • Test definition pattern configuration

  • test_definition_extraction_processing

    • Test definition extraction from text
    • Test definition quality assessment
    • Test definition standardization

Object Extraction

Row Extraction (trustgraph.extract.object.row)

Unit Tests

  • test_row_extraction_initialization

    • Test row extractor initialization
    • Test schema configuration

  • test_row_extraction_processing

    • Test structured data extraction
    • Test row validation
    • Test row normalization

Retrieval Services

GraphRAG Retrieval (trustgraph.retrieval.graph_rag)

Unit Tests

  • test_graph_rag_initialization

    • Test GraphRAG retrieval initialization
    • Test graph and vector store configuration
    • Test retrieval parameters configuration

  • test_graph_rag_processing

    • Test query processing and understanding
    • Test vector similarity search
    • Test graph traversal for context
    • Test context ranking and selection
    • Test response generation

  • test_graph_rag_optimization

    • Test query optimization
    • Test context size management
    • Test retrieval caching
    • Test performance monitoring

Document RAG Retrieval (trustgraph.retrieval.document_rag)

Unit Tests

  • test_document_rag_initialization

    • Test Document RAG retrieval initialization
    • Test document store configuration

  • test_document_rag_processing

    • Test document similarity search
    • Test document chunk retrieval
    • Test document ranking
    • Test context assembly

Integration Test Cases

End-to-End Flow Tests

Document Processing Flow

  • test_document_ingestion_flow

    • Test PDF document ingestion
    • Test text document ingestion
    • Test document chunking
    • Test embedding generation
    • Test storage operations

  • test_knowledge_graph_construction_flow

    • Test entity extraction
    • Test relationship extraction
    • Test graph construction
    • Test graph storage

Query Processing Flow

  • test_graphrag_query_flow

    • Test query input processing
    • Test vector similarity search
    • Test graph traversal
    • Test context assembly
    • Test response generation

  • test_agent_flow

    • Test agent query processing
    • Test ReAct reasoning cycle
    • Test tool usage
    • Test response formatting

Service Integration Tests

Storage Integration

  • test_vector_storage_integration

    • Test Qdrant integration with embeddings
    • Test Milvus integration with embeddings
    • Test storage consistency across services

  • test_graph_storage_integration

    • Test Cassandra integration with triples
    • Test Neo4j integration with graphs
    • Test cross-storage consistency

Model Integration

  • test_llm_integration
    • Test OpenAI integration
    • Test Claude integration
    • Test Ollama integration
    • Test model switching

Error Handling Test Cases

Network Error Handling

  • test_connection_timeout_handling

    • Test database connection timeouts
    • Test API connection timeouts
    • Test Pulsar connection timeouts

  • test_network_interruption_handling

    • Test network disconnection scenarios
    • Test network reconnection scenarios
    • Test partial network failures

Resource Error Handling

  • test_memory_exhaustion_handling

    • Test out of memory scenarios
    • Test memory leak detection
    • Test memory cleanup

  • test_disk_space_handling

    • Test disk full scenarios
    • Test storage cleanup
    • Test storage monitoring

Service Error Handling

  • test_service_unavailable_handling

    • Test external service unavailability
    • Test service degradation
    • Test service recovery

  • test_data_corruption_handling

    • Test corrupted message handling
    • Test invalid data detection
    • Test data recovery procedures

Rate Limiting Error Handling

  • test_api_rate_limit_handling

    • Test OpenAI rate limit scenarios
    • Test Claude rate limit scenarios
    • Test backoff strategies

  • test_resource_quota_handling

    • Test storage quota exceeded
    • Test compute quota exceeded
    • Test API quota exceeded

Performance Test Cases

Load Testing

  • test_concurrent_processing

    • Test concurrent message processing
    • Test concurrent database operations
    • Test concurrent API calls

  • test_throughput_limits

    • Test message processing throughput
    • Test storage operation throughput
    • Test query processing throughput

Stress Testing

  • test_high_volume_processing

    • Test processing large document sets
    • Test handling large knowledge graphs
    • Test processing high query volumes

  • test_resource_exhaustion

    • Test behavior under memory pressure
    • Test behavior under CPU pressure
    • Test behavior under network pressure

Scalability Testing

  • test_horizontal_scaling

    • Test service scaling behavior
    • Test load distribution
    • Test scaling bottlenecks

  • test_vertical_scaling

    • Test resource utilization scaling
    • Test performance scaling
    • Test cost scaling

Security Test Cases

Authentication and Authorization

  • test_api_key_validation

    • Test valid API key scenarios
    • Test invalid API key scenarios
    • Test expired API key scenarios

  • test_service_authentication

    • Test service-to-service authentication
    • Test authentication token validation
    • Test authentication failure handling

Data Protection

  • test_data_encryption

    • Test data encryption at rest
    • Test data encryption in transit
    • Test encryption key management

  • test_data_sanitization

    • Test input data sanitization
    • Test output data sanitization
    • Test sensitive data masking

Input Validation

  • test_input_validation

    • Test malformed input handling
    • Test injection attack prevention
    • Test input size limits

  • test_output_validation

    • Test output format validation
    • Test output content validation
    • Test output size limits

Monitoring and Observability Test Cases

Metrics Collection

  • test_prometheus_metrics

    • Test metrics collection and export
    • Test custom metrics registration
    • Test metrics aggregation

  • test_performance_metrics

    • Test latency metrics collection
    • Test throughput metrics collection
    • Test error rate metrics collection

Logging

  • test_structured_logging

    • Test log format consistency
    • Test log level configuration
    • Test log aggregation

  • test_error_logging

    • Test error log capture
    • Test error log correlation
    • Test error log analysis

Tracing

  • test_distributed_tracing

    • Test trace propagation
    • Test trace correlation
    • Test trace analysis

  • test_request_tracing

    • Test request lifecycle tracing
    • Test cross-service tracing
    • Test trace performance impact

Configuration Test Cases

Environment Configuration

  • test_environment_variables

    • Test environment variable loading
    • Test environment variable validation
    • Test environment variable defaults

  • test_configuration_files

    • Test configuration file loading
    • Test configuration file validation
    • Test configuration file precedence

Dynamic Configuration

  • test_configuration_updates

    • Test runtime configuration updates
    • Test configuration change propagation
    • Test configuration rollback

  • test_configuration_validation

    • Test configuration schema validation
    • Test configuration dependency validation
    • Test configuration constraint validation

Test Data and Fixtures

Test Data Generation

  • test_synthetic_data_generation

    • Test synthetic document generation
    • Test synthetic graph data generation
    • Test synthetic query generation

  • test_data_anonymization

    • Test personal data anonymization
    • Test sensitive data masking
    • Test data privacy compliance

Test Fixtures

  • test_fixture_management

    • Test fixture setup and teardown
    • Test fixture data consistency
    • Test fixture isolation

  • test_mock_data_quality

    • Test mock data realism
    • Test mock data coverage
    • Test mock data maintenance

Test Execution and Reporting

Test Execution

  • test_parallel_execution

    • Test parallel test execution
    • Test test isolation
    • Test resource contention

  • test_test_selection

    • Test tag-based test selection
    • Test conditional test execution
    • Test test prioritization

Test Reporting

  • test_coverage_reporting

    • Test code coverage measurement
    • Test branch coverage analysis
    • Test coverage trend analysis

  • test_performance_reporting

    • Test performance regression detection
    • Test performance trend analysis
    • Test performance benchmarking

Maintenance and Continuous Integration

Test Maintenance

  • test_test_reliability

    • Test flaky test detection
    • Test test stability analysis
    • Test test maintainability

  • test_test_documentation

    • Test test documentation quality
    • Test test case traceability
    • Test test requirement coverage

Continuous Integration

  • test_ci_pipeline_integration

    • Test CI pipeline configuration
    • Test test execution in CI
    • Test test result reporting

  • test_automated_testing

    • Test automated test execution
    • Test automated test reporting
    • Test automated test maintenance

This comprehensive test case document provides detailed testing scenarios for all TrustGraph microservices, ensuring thorough coverage of functionality, error handling, performance, security, and operational aspects. Each test case should be implemented following the patterns and best practices outlined in the TEST_STRATEGY.md document.