system: Loosen definition of system.process.count #3864

Area(s)

area:system

What's missing?

The definition of system.process.count is a bit too restrictive. It is designed around the particular way we report the metric in host_metrics receiver today which is to readdir(2) /proc and count the results by each process's given state. There are some scenarios on Linux where we want to get a count not broken down by state, and on Windows there isn't really the same prevailing concept of state (which is why the metric is unsupported on Windows in the host_metrics receiver today).

I think we need to loosen the definition of this metric to support use cases where we don't want to break down the count by state, and just want a total for the system.

Describe the solution you'd like

I see two possible resolutions each with their own tradeoff.

Make process.state attribute opt-in

This is kind of a strange case where the "opt-in" is more based on what the instrumentation is capable of rather than the user asking for it specifically. I would try to make clear in the notes that the attribute is opt-in, but a user opting in may not always be possible.

The upside to this is that we retain the system.process.count name under any reporting scenario. This is a clean and obvious name that is relatively universal and easy to understand at a glance.

Introduce separate metrics for counts with and without process.state

Another alternative is to introduce two separate metrics, one for a total count and one for a count broken down by state. This might end up being a bit cleaner, since I worry there may be cases where system.process.count is reported ambiguously in a way that a backend would struggle to disambiguate. For the same host, a user might have instrumentation that reports system.process.count broken down by state, but then for the same host may also have separate instrumentation that reads the total count and reports it without any attribute. It would be difficult for backends to distinguish them and make it very difficult to use in queries and visualizations.

The downside is that the naming is a bit awkward. Both metrics are "system process counts", but they need to have distinct names that mean the same thing but somehow make clear at a glance what the difference is.
(It would be convenient if this could be separated by platform, like process.unix.count comes with a state and process.windows.count doesn't, but I don't think that works here. Even within the same platform, we may want a total count or a count broken down by state. So it has to be named with that as the distinction, not the platform.)

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}