Agent Scenario - Change Point Detection - Hyperparameter Tuning

[Lucew]

Interesting benchmark!

Recently, you opened an issue in the repository for the changepoynt package.

I think that change point detection and especially hyperparameter tuning for the detection (algorithm, window size, lag, etc.) would make for a very interesting use case within the agentic evaluation benchmark. Given ground-truth change-point detections and an accuracy metric for those detections, agent performance would be easily quantifiable. From my experience, deciding on an algorithm and tuning the hyperparameters is the hardest part for engineers and practitioners, so having an agent loop that does that would really help.

For additional ideas and further details see my answer in the issue.

[DhavalRepo18]

@Lucew, I think this is a very valuable method, and also related to say how an engineer likes to define the monitoring

https://huggingface.co/datasets/ibm-research/AssetOpsBench/viewer/rule_logic?row=0

Please find anomalies for asset AHU HUR00118 between 2021-04-11T23:00:00.000Z and 2021-04-12T05:15:00.000Z according to: An anomaly is detected when the AHU is running, the cooling valve is open at 5% or more, and either the heating valve or preheat valve is also open at 5% or more. At the same time, if reported, neither the heating nor the cooling drained flags are active. These conditions must persist for at least 2 hours.

Ideally, Monitoring Changes is an interesting aspect, and especially if those tools are made available as an MCP-Change-Pointer Detection Server, it can be used along with the other tasks.

Maybe the above scenarios can be used to define the change point scenario for a Business user.

[Lucew]

My initial motivation to dive into change point detection was to detect signal relationships. We are using change point correlation (matching the change score instead of the raw signals) to find related signals, basically saying:

If two signals change together, they belong together.

What our business partners do with that:

• find related signals in very large collections of signals (e.g., signals that are from the same gas turbine within a power plant)
• For an event of interest (change), find all signals that changed simultaneously (root cause analysis)
• For signals that change simultaneously during normal behavior, find an occasion where they change independently (anomaly?)

These scenarios might be related to your linked scenario. Paired with an MCP server, an agent could answer these questions.

We have a working prototype that contains visualizations to answer the above questions, which I might publish in the next few weeks/months.

[DhavalRepo18]

@Lucew I agree with you, and I recall one of my papers (https://ieeexplore.ieee.org/abstract/document/8622596) in a similar direction.

Let us know how we can help you so you can bring a few more scenarios based on the change point. Also, you mention a business-partner-driven scenario generation.

[DhavalRepo18]

@Lucew let me know any interest in contrubuting few scenarios.