Merge the ECP KPP metrics tool to develop branch

[rppawlo]

For the ECP KPP metric collection, there was a special profiling tool that exists on the branch "ecp-kpp3". This tool provides some very useful information. Could this be merged into the main develop branch so that it is maintained and available to the apps?

[dalg24]

Specifically which metrics reported by the tool are you interested in?

[rppawlo]

Just about everything in there is useful. If we could keep it all, that would be great. The breakdown of time in serial vs parallel algorithms and the total vs unique kernel numbers helps us to compare codes. It was eye opening to see how many kernels per second are called for different code bases. The total memory stats are also very helpful in reporting. It's great that it covers host, device, and host pinned all in the same run. Are there items you would prefer not to support?

[rppawlo]

Getting the aggregated list of the most expensive kernels is very helpful too. I was surprised to see how this list could change when running the same application on different problem types.

[dalg24]

Are there items you would prefer not to support?

Nothing in particular. Just trying to figure out how we should call that tool and what info belongs in there.

[vlkale]

Yes, the information output here is good. A problem I see with merging it in the develop branch is it does not look fully generalizable and is focused on ECP/DOE Kokkos applications.

I am also wondering about the name of such a Kokkos Tools connector library. What name captures its purpose for any arbitrary Kokkos application?

[rppawlo]

What parts are not generalizable? Is it the reporting of system hardware? I don't think that is needed, but is nice to have too. I would think something like ApplicationSummary, ApplicationStatistics, ApplicationAnalyzer, or AggregatedStatistics. I don't really care for any of these names, but it's giving a high level view of overall run including kernels and memory use.

[vlkale]

@rppawlo Yes, I was thinking about the profiling for specific kinds of system hardware but also that - as far as I see in the branch - it is geared towards ECP's Kokkos applications (i.e., Kokkos applications deployed onto DOE's exascale supercomputers). Would it work for, e.g., an AI application using Kokkos not using DOE's computing platforms? I agree these statistics were very useful for ECP for the success of Kokkos, and they certainly still are. I appreciate you bringing this up and I think it does need more attention.

Putting aside my generalizability argument, let's say we do agree to merge it into Kokkos Tools: I am trying to figure out exactly where it should be placed. Should it just be merged as it is, but renaming from 'kpp3-verifier' to something like 'verification-tooling' (again, need to think about a generalizable name)? Or, do we want to create a separate directory alongside the 'profiling' subdirectory called 'verification-tooling'? I would say the latter based on the purpose of this tool connector.