Implement flat-first archiving for Action reports to improve limiting and memory consumption by sgiehl · Pull Request #24191 · matomo-org/matomo

sgiehl · 2026-03-09T13:25:02Z

Summary

This PR introduces flat-first archiving for Action reports, controlled by a new config flag:

datatable_archiving_maximum_rows_actions_flat

Enable / Disable

Disabled (legacy behavior): flag is 0 or not set
- Only hierarchical Action tables are archived (same as before).
Enabled (new behavior): flag is > 0
- A flat Action archive is built first and limited to this configured row count.
- Hierarchical tables are then built from this already-limited flat data.

A useful initial value is 50000, which matches the current query/report row limit.

Archiving approach changes

With flat-first enabled:

Day Action archives build a flat limited table first.
The hierarchy is built from that flat table (instead of building hierarchy first and flattening later).
Higher periods (week/month/year) aggregate from flat archives.
For mixed historical data, if a period has no flat archive yet, existing hierarchical archives are flattened and
merged so re-archiving everything is not required.

Limits and `Others` behavior

The main limit is now applied at the flat table level using datatable_archiving_maximum_rows_actions_flat.
No additional hierarchical limiting is applied afterward (hierarchy is built from already-limited data).
Others is determined during flat limiting, so behavior is more consistent and avoids extra secondary Others created
by hierarchical re-limiting.

Impact on memory and archiving time

For high-cardinality Action data (many distinct URLs, changing month-to-month), this reduces resource pressure by
avoiding repeated deep hierarchical merges during period aggregation.

Memory: lower peak memory usage during aggregation.
Archiving time: typically faster/more stable for larger periods.
Tradeoff: additional flat archive data is stored, but bounded by the configured limit.

Testing

The changes contain a couple of tests to proof that flat-first archiving in general works. However, all tests were run with flat-archiving enabled, to ensure no other tests regress unexpectedly. Most of those failures were fixed by applying unrelated fixes (see linked PRs).
There is currently one mismatch in tests remaining. See https://github.com/matomo-org/matomo/actions/runs/23338988295

URL Metadata / Segment Mismatch Note

There is an existing inconsistency in how Matomo merges duplicate rows with the same label but different metadata (for
example URL rows where one variant has different segment metadata, empty segment metadata, or different include-depth
behavior like include2 vs include4).

With flat-first archiving, this can become more visible because row merging happens through a slightly different path
than pure legacy hierarchy aggregation. The underlying issue is not specific to flat-first itself: when multiple
candidate rows exist, metadata winner selection is not fully deterministic/explicit in all merge paths.

Practical impact:

Metric totals can still be correct.
Returned metadata fields (especially segment metadata for URL rows) may differ in edge cases depending on merge
order/source.
This is a pre-existing merge-policy gap and should be fixed separately by defining deterministic metadata precedence
during row aggregation.

Checklist

[✔] I have understood, reviewed, and tested all AI outputs before use
[✔] All AI instructions respect security, IP, and privacy rules

Review

…or aggregating higher periods with mixed tables (flat + hierachical)

nathangavin

It generally looks good to me. With regards to testing, would it be feasible to directly compare legacy hierarchy aggregation to flat aggregation to determine that we are getting the same output?

Other than that I don't see any issues.

sgiehl · 2026-03-24T08:57:48Z

@nathangavin you can check the test run linked in the PR description. That one was running with flat-first-archiving enabled for all tests. As I guess the mid-term plan is to enable the flat-first-archiving for everyone and remove the old approach, we don't need any further tests, but simply switch them to run with flat-first-archiving at some point.

tsteur

@sgiehl had a very quick initial look. Will spend more time eventually to properly understand logic. Feel free to ignore what doesn't make sense.

BTW do we need to update anything on https://developer.matomo.org/guides/writing-a-record-builder ?

tsteur · 2026-03-24T09:29:29Z

plugins/Actions/ArchivingHelper.php

+        $lastSegment = (string) $segments[$lastIndex];
+
+        if ($actionType === Action::TYPE_PAGE_URL) {
+            if (strpos($lastSegment, '/') === 0) {


would this need to use action_url_category_delimiter?

This is a leaf node prefix, rather then a delimiter. Basically the reverse handling of getActionExplodedNames:

matomo/plugins/Actions/ArchivingHelper.php

Lines 618 to 659 in 2a1891f

public static function getActionExplodedNames($name, $type, $urlPrefix = null)

{

// Site Search does not split Search keywords

if ($type == Action::TYPE_SITE_SEARCH) {

return array($name);

}

$name = str_replace("\n", "", $name);

if ($type == Action::TYPE_PAGE_TITLE && self::$actionTitleCategoryDelimiter === '') {

if ($name === '' || $name === false || $name === null || trim($name) === '') {

$name = self::getUnknownActionName($type);

}

return array(' ' . trim($name));

}

$name = self::parseNameFromPageUrl($name, $type, $urlPrefix);

// outlinks and downloads

if (is_array($name)) {

return $name;

}

$split = self::splitNameByDelimiter($name, $type);

if (empty($split)) {

$defaultName = self::getUnknownActionName($type);

return array(trim($defaultName));

}

$lastPageName = end($split);

// we are careful to prefix the page URL / name with some value

// so that if a page has the same name as a category

// we don't merge both entries

if ($type != Action::TYPE_PAGE_TITLE) {

$lastPageName = '/' . $lastPageName;

} else {

$lastPageName = ' ' . $lastPageName;

}

$split[count($split) - 1] = $lastPageName;

return array_values($split);

}

tsteur · 2026-03-24T09:29:45Z

plugins/Actions/ArchivingHelper.php

+            $segments[$lastIndex] = $lastSegment;
+            $delimiter = self::$actionUrlCategoryDelimiter;
+        } else {
+            if (strpos($lastSegment, ' ') === 0) {


would this need to use action_title_category_delimiter?

tsteur · 2026-03-24T09:31:55Z

plugins/Actions/RecordBuilders/ActionReports.php

+
+        if ($recordName === Archiver::PAGE_URLS_RECORD_NAME) {
+            $prefix = $archiveProcessor->getParams()->getSite()->getMainUrl();
+            $prefix = rtrim($prefix, '/') . '/';


Do we need to use action_url_category_delimiter here?

Not in this case. This one just provides the URL that will be used for links. You could e.g. define the delimiter to be ?, to only break at query strings. But for the final url you would still prefix http://example.com/ rather then http://example.com?

tsteur · 2026-03-24T09:33:48Z

core/ArchiveProcessor/Record.php

        return $this->multiplePeriodTransform;
    }
+
+    public function setBuiltFromFlatRecord(


for those new methods could we add some documentation to better understand when and how to use them?

added some documentation

config/global.ini.php

tsteur · 2026-03-24T09:36:28Z

core/ArchiveProcessor/RecordBuilder.php

+            }
+
+            if (!in_array($record->getName(), $requestedReports)) {
+                $requestedReports[] = $record->getName();


sometimes could be good to add some comments to less needing to figure out yourself why this is needed. I think I understood but takes some time to process.

added some comments here.

tsteur · 2026-03-24T09:40:42Z

core/ArchiveProcessor/RecordBuilder.php

+        return true;
+    }
+
+    protected function beforeInsertBuiltFromFlatHierarchyRecord(


I'm assuming this is becoming API for plugins to add extra action, be good to add some comment to the method here too.

tsteur · 2026-03-24T09:42:34Z

core/ArchiveProcessor/RecordBuilder.php

        }
    }

+    protected function aggregateBuiltFromFlatRecordForNonDay(


do you think for all these methods it's possible to add some tests? even if it's just a simple happy test? there's quite a lot of logic in these methods

generated some additional tests with code for those methods.

tsteur · 2026-03-24T09:47:06Z

tests/PHPUnit/System/BlobReportLimitingTest.php

        $generalConfig['datatable_archiving_maximum_rows_site_search'] = 5;
    }
+
+    protected static function setUpConfigOptionsFlatFirst()


Do we already have tests with a low row limit to see if "Other" row is still correctly being aggregated when it comes to flat archiving? I didn't look. Just thought to check.

When running the other existing action tests with flat archiving enabled, would we expect any changes in reporting data? I assume it would be good to run the same tests with flat archiving enabled and without?

I see your recent comment that we did a test run with the flag enabled and we'll remove the flag eventually so that may be a fine for now assuming we remove the flag soon 👍 Probably no change needed then.

tsteur · 2026-03-25T06:47:28Z

core/ArchiveProcessor/RecordBuilder.php

+                continue;
+            }
+
+            $tableToAddTo->addDataTable($blobTable);


this logic around here seems a bit similar with getAggregatedDataTableMapFromBlobs but I assume it is slightly differently and can't be reused?

I tried to refactor this into an external helper to reduce code duplication.

tsteur · 2026-03-25T06:49:24Z

core/ArchiveProcessor/RecordBuilder.php

+
+    protected function getSubtableIdFromBlobName(string $recordName): ?int
+    {
+        $parts = explode('_', $recordName);


I think the same logic we got in ArchiveProcessor::getSubtableIdFromBlobName(). Should we make these reusable? Public method in a new class or so and then have tests for it?

also refactored into a external helper

sgiehl added this to the 5.9.0 milestone Mar 9, 2026

sgiehl force-pushed the archivetest branch 8 times, most recently from f4c6778 to a5ef0f1 Compare March 12, 2026 16:07

sgiehl force-pushed the archivetest branch 5 times, most recently from 82902eb to f4fca88 Compare March 18, 2026 14:00

sgiehl mentioned this pull request Mar 18, 2026

Refresh stale archive ID cache when archive rows disappear #24246

Merged

10 tasks

sgiehl force-pushed the archivetest branch from f4fca88 to 2b54d22 Compare March 19, 2026 22:14

sgiehl added 10 commits March 20, 2026 11:32

implement flat archiving logic

f0f9497

move flat-first archiving behind config flag and implement handling f…

f04b2b2

…or aggregating higher periods with mixed tables (flat + hierachical)

code improvements & tests

8870320

Move build from flat logic to RecordBuilder

741b052

further improvements

893663b

move code to reusable helper methods

bb25240

more fixes and tests

a787634

simplify code

9715335

code simplification

8319084

more cleanup

c1c39ec

sgiehl force-pushed the archivetest branch 2 times, most recently from aafbd67 to 742c6b4 Compare March 20, 2026 12:24

sgiehl changed the title ~~Use flat-first archiving for Action reports to improve limitting and memory consumption~~ Implement flat-first archiving for Action reports to improve limitting and memory consumption Mar 20, 2026

sgiehl changed the title ~~Implement flat-first archiving for Action reports to improve limitting and memory consumption~~ Implement flat-first archiving for Action reports to improve limiting and memory consumption Mar 20, 2026

sgiehl requested review from a team, diosmosis and tsteur March 20, 2026 12:26

sgiehl marked this pull request as ready for review March 20, 2026 12:26

updates expected UI test file

570cd5e

sgiehl force-pushed the archivetest branch from 742c6b4 to 570cd5e Compare March 20, 2026 20:51

Ensure to correctly archive if only flat record is requested

0e4cd99

nathangavin reviewed Mar 24, 2026

View reviewed changes

tsteur reviewed Mar 24, 2026

View reviewed changes

tsteur reviewed Mar 25, 2026

View reviewed changes

sgiehl added 3 commits March 25, 2026 10:34

apply some review feedback

715511d

deduplicate/refactor code

aad763c

use constants for leaf markers

16caf11

	public static function getActionExplodedNames($name, $type, $urlPrefix = null)
	{
	// Site Search does not split Search keywords
	if ($type == Action::TYPE_SITE_SEARCH) {
	return array($name);
	}

	$name = str_replace("\n", "", $name);

	if ($type == Action::TYPE_PAGE_TITLE && self::$actionTitleCategoryDelimiter === '') {
	if ($name === '' \|\| $name === false \|\| $name === null \|\| trim($name) === '') {
	$name = self::getUnknownActionName($type);
	}
	return array(' ' . trim($name));
	}

	$name = self::parseNameFromPageUrl($name, $type, $urlPrefix);

	// outlinks and downloads
	if (is_array($name)) {
	return $name;
	}

	$split = self::splitNameByDelimiter($name, $type);

	if (empty($split)) {
	$defaultName = self::getUnknownActionName($type);
	return array(trim($defaultName));
	}

	$lastPageName = end($split);
	// we are careful to prefix the page URL / name with some value
	// so that if a page has the same name as a category
	// we don't merge both entries
	if ($type != Action::TYPE_PAGE_TITLE) {
	$lastPageName = '/' . $lastPageName;
	} else {
	$lastPageName = ' ' . $lastPageName;
	}
	$split[count($split) - 1] = $lastPageName;
	return array_values($split);
	}

Uh oh!

Conversation

sgiehl commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Enable / Disable

Archiving approach changes

Limits and Others behavior

Impact on memory and archiving time

Testing

URL Metadata / Segment Mismatch Note

Checklist

Review

Uh oh!

nathangavin left a comment

Choose a reason for hiding this comment

Uh oh!

sgiehl commented Mar 24, 2026

Uh oh!

tsteur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

sgiehl commented Mar 9, 2026 •

edited

Loading

Limits and `Others` behavior