Skip to content

Flaky grounding test #54

@abrookins

Description

@abrookins

This test is flaking (TestThreadAwareContextualGrounding.test_multi_entity_conversation):

=================================== FAILURES ===================================
______ TestThreadAwareContextualGrounding.test_multi_entity_conversation _______

self = <tests.test_thread_aware_grounding.TestThreadAwareContextualGrounding object at 0x7f806c145970>

    @pytest.mark.requires_api_keys
    async def test_multi_entity_conversation(self):
        """Test contextual grounding with multiple entities in conversation."""
    
        session_id = f"test-multi-entity-{ulid.ULID()}"
    
        # Create conversation with multiple people
        messages = [
            MemoryMessage(
                id=str(ulid.ULID()),
                role="user",
                content="John and Sarah are working on the API redesign project.",
                timestamp=datetime.now(UTC).isoformat(),
                discrete_memory_extracted="f",
            ),
            MemoryMessage(
                id=str(ulid.ULID()),
                role="user",
                content="He's handling the backend while she focuses on the frontend integration.",
                timestamp=datetime.now(UTC).isoformat(),
                discrete_memory_extracted="f",
            ),
            MemoryMessage(
                id=str(ulid.ULID()),
                role="user",
                content="Their collaboration has been very effective. His Python skills complement her React expertise.",
                timestamp=datetime.now(UTC).isoformat(),
                discrete_memory_extracted="f",
            ),
        ]
    
        working_memory = WorkingMemory(
            session_id=session_id,
            user_id="test-user",
            namespace="test-namespace",
            messages=messages,
            memories=[],
        )
    
        await set_working_memory(working_memory)
    
        # Extract memories
        extracted_memories = await extract_memories_from_session_thread(
            session_id=session_id,
            namespace="test-namespace",
            user_id="test-user",
        )
    
        assert len(extracted_memories) > 0
    
        all_memory_text = " ".join([mem.text for mem in extracted_memories])
    
        print(f"\nMulti-entity extracted memories: {len(extracted_memories)}")
        for i, mem in enumerate(extracted_memories):
            print(f"{i + 1}. [{mem.memory_type}] {mem.text}")
    
        # Should mention both John and Sarah by name
        assert "john" in all_memory_text.lower(), "Should mention John by name"
>       assert "sarah" in all_memory_text.lower(), "Should mention Sarah by name"
E       AssertionError: Should mention Sarah by name
E       assert 'sarah' in 'john is handling the backend of the api redesign project.'
E        +  where 'john is handling the backend of the api redesign project.' = <built-in method lower of str object at 0x7f806114c5e0>()
E        +    where <built-in method lower of str object at 0x7f806114c5e0> = 'John is handling the backend of the API redesign project.'.lower

tests/test_thread_aware_grounding.py:207: AssertionError
----------------------------- Captured stdout call -----------------------------

Multi-entity extracted memories: 1
1. [MemoryTypeEnum.EPISODIC] John is handling the backend of the API redesign project.
------------------------------ Captured log call -------------------------------
INFO     agent_memory_server.working_memory:working_memory.py:206 Set working memory for session test-multi-entity-01K3P5M1H0YCCGZ31FW5FNR5NK with no TTL
INFO     agent_memory_server.long_term_memory:long_term_memory.py:192 Extracting memories from 3 messages in session test-multi-entity-01K3P5M1H0YCCGZ31FW5FNR5NK
INFO     openai._base_client:_base_client.py:1608 Retrying request to /chat/completions in 0.495191 seconds
INFO     agent_memory_server.long_term_memory:long_term_memory.py:247 Extracted 1 memories from session thread test-multi-entity-01K3P5M1H0YCCGZ31FW5FNR5NK
=============================== warnings summary ===============================
tests/test_extraction.py::TestTopicExtractionIntegration::test_bertopic_integration
  /home/runner/work/agent-memory-server/agent-memory-server/.venv/lib/python3.12/site-packages/hdbscan/plots.py:448: SyntaxWarning: invalid escape sequence '\l'
    axis.set_ylabel('$\lambda$ value')

tests/test_extraction.py::TestTopicExtractionIntegration::test_bertopic_integration
  /home/runner/work/agent-memory-server/agent-memory-server/.venv/lib/python3.12/site-packages/hdbscan/robust_single_linkage_.py:175: SyntaxWarning: invalid escape sequence '\{'
    $max \{ core_k(a), core_k(b), 1/\alpha d(a,b) \}$.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_thread_aware_grounding.py::TestThreadAwareContextualGrounding::test_multi_entity_conversation - AssertionError: Should mention Sarah by name
assert 'sarah' in 'john is handling the backend of the api redesign project.'
 +  where 'john is handling the backend of the api redesign project.' = <built-in method lower of str object at 0x7f806114c5e0>()
 +    where <built-in method lower of str object at 0x7f806114c5e0> = 'John is handling the backend of the API redesign project.'.lower
====== 1 failed, 375 passed, 26 skipped, 2 warnings in 151.50s (0:02:31) =======
Error: Process completed with exit code 1.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions