[Osquerybeat] Osquerybeat Uses Wrong Data Source for "removed" Differential Results

[tomsonpl]

Summary

There are two bugs in osquerybeat's differential results handling that cause:

1. "removed" events to contain the same data as "added" events (wrong values)
2. Duplicate events in the "removed" results

Both bugs are in x-pack/osquerybeat/beater/osquerybeat.go in the handleQueryResult function.

Bug Details

Bug 1: Wrong Data Source for "removed" (Line 436)

The code reads from DiffResults.Added instead of DiffResults.Removed:

// Line 435-436: BUG - uses Added instead of Removed
if len(res.DiffResults.Removed) > 0 {
    removed, err := cli.ResolveResult(ctx, qi.Query, res.DiffResults.Added)  // ✗ WRONG
    //                                               ^^^^^^^^^^^^^^^^^^^^
    //                                  Should be: res.DiffResults.Removed

Bug 2: Accumulating hits Slice Causes Duplicates (Lines 432 + 441)

The hits slice is not reset between processing "added" and "removed", so the "removed" publish includes the "added" results:

var (
    hits []map[string]interface{}
)

// ...

// Line 432: After this, hits = [added_items]
hits = append(hits, added...)
bt.pub.Publish(..., hits, ...)  // Publishes: [added_items] ✓

// Line 441: After this, hits = [added_items, removed_items]
hits = append(hits, removed...)
bt.pub.Publish(..., hits, ...)  // Publishes: [added_items, removed_items] ✗ WRONG
//                                            ^^^^^^^^^^^^
//                                            Should NOT be included

Steps to Reproduce

1. Create an osquery pack with differential mode

Via Kibana Osquery Manager or Fleet, create a pack with:

{
  "queries": {
    "services_differential": {
      "query": "SELECT name, display_name, path, start_type, status FROM services;",
      "interval": 1800,
      "platform": "windows",
      "snapshot": false,
      "removed": true
    }
  }
}

2. Deploy via Fleet/Elastic Agent to a Windows endpoint

3. Wait for a service to change state (e.g., BITS service naturally toggles)

4. Observe results in Elasticsearch

Expected Behavior

When a service changes from STOPPED to RUNNING:

// 1 removed event with OLD values
{ "osquery_meta": { "action": "removed" }, "osquery": { "name": "BITS", "status": "STOPPED" } }

// 1 added event with NEW values
{ "osquery_meta": { "action": "added" }, "osquery": { "name": "BITS", "status": "RUNNING" } }

Actual Behavior

// 2 removed events (duplicate!) with WRONG values (current instead of old)
{ "osquery_meta": { "action": "removed" }, "osquery": { "name": "BITS", "status": "RUNNING" } }
{ "osquery_meta": { "action": "removed" }, "osquery": { "name": "BITS", "status": "RUNNING" } }

// 1 added event with current values (correct)
{ "osquery_meta": { "action": "added" }, "osquery": { "name": "BITS", "status": "RUNNING" } }

Impact

• Cannot determine what changed: Both "added" and "removed" have identical values
• Duplicate events: Double the log volume and noise
• Security monitoring degraded: Differential mode is used for drift detection, compliance auditing, etc.

Environment

• Elastic Agent version: 9.3.0 (bug exists in current main branch)
• Operating System: Windows 11 Pro, Windows Server

[botelastic]

This issue doesn't have a Team:<team> label.