Help understanding output of MEM group with Marker API - reported memory bandwidth measurement is much higher than what likwid-bench suggests

[mladenivkovic]

Hi all!

I'm in the process of running some measurements on my software with likwid and I was hoping to ask for some help with interpreting the results likwid is showing me, as the way I understand them, they can't be right.

Here’s what I’ve been up to so far.

My Software

• Parallelised using pthreads (and “in-house” task scheduler based on ptheads), uses cudafor GPU acceleration.
• My code can pin the threads it runs on (using pthreads_setaffinity_np), so in the measurement runs with likwid-perfctr, I’ll be using the -c instead of the -C flag
• https://github.com/abouzied-nasar/SWIFT/tree/likwid-markers in the unexpected case it’s needed
• The cluster I’m using has likwid 5.4.1 installed

What I want to achieve

• I want to measure the memory bandwidth behaviour in specific regions of my code
  • I’m interested in CPU behaviour of a code that is GPU accelerated. The GPU bit is not interesting to me at this point, only what’s going on on the CPU side.
  • the MEM event group and likwid' s Marker API seems ideal to do this.

My Hardware

--------------------------------------------------------------------------------
CPU name:	Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
CPU type:	Intel Cascadelake SP processor
CPU stepping:	7
********************************************************************************
Hardware Thread Topology
********************************************************************************
Sockets:		2
CPU dies:		2
Cores per socket:	16
Threads per core:	2

What I’ve done so far

To begin with, I did some likwid-bench benchmarks first to assess the peak memory bandwidth of my system:

# Triad benchmark: A[i] = B[i] * C[i] + D[i]
# 32 threads
likwid-bench -t triad -w N:4GB:32
MByte/s:		78734.97

# single thread
MByte/s:		12036.72

# copy_mem benchmark: Copy with non-temporal store
# 32 threads
likwid-bench -t copy_mem -w N:4GB:32
MByte/s:		83644.83

likwid-bench -t copy_mem -w N:4GB:1
MByte/s:		8881.45

# load benchmark
# 32 threads
likwid-bench -t load -w N:4GB:32
MByte/s:		99647.42

# single thread
likwid-bench -t load -w N:4GB:1
MByte/s:		10869.33

# store benchmark
# 32 threads
likwid-bench -t store -w N:4GB:32
MByte/s:	47517.67

# single thread
likwid-bench -t store -w N:4GB:1 
MByte/s:	9423.61

I made sure I was the only one using the node while benchmarking. So far, so good, I guess.

Next, I ran a tiny (non-representative) example with my software to verify that everything works as it should.

First, a run on a single thread:

likwid-perfctr -f -c N:1 -g MEM ../swift_cuda --hydro --threads=1 --steps=2 --pin greshoGPU64.yml

gives:

--------------------------------------------------------------------------------
Group 1: MEM
+-----------------------+---------+------------+
|         Event         | Counter | HWThread 2 |
+-----------------------+---------+------------+
|   INSTR_RETIRED_ANY   |  FIXC0  |       6864 |
| CPU_CLK_UNHALTED_CORE |  FIXC1  |      30469 |
|  CPU_CLK_UNHALTED_REF |  FIXC2  |      25944 |
|      CAS_COUNT_RD     | MBOX0C0 |   14515348 |
|      CAS_COUNT_WR     | MBOX0C1 |    7941874 |
|      CAS_COUNT_RD     | MBOX1C0 |   14314478 |
|      CAS_COUNT_WR     | MBOX1C1 |    7980721 |
|      CAS_COUNT_RD     | MBOX2C0 |   14274722 |
|      CAS_COUNT_WR     | MBOX2C1 |    7958660 |
|      CAS_COUNT_RD     | MBOX3C0 |   14112250 |
|      CAS_COUNT_WR     | MBOX3C1 |    7972464 |
|      CAS_COUNT_RD     | MBOX4C0 |   14103420 |
|      CAS_COUNT_WR     | MBOX4C1 |    7950516 |
|      CAS_COUNT_RD     | MBOX5C0 |   14039041 |
|      CAS_COUNT_WR     | MBOX5C1 |    7934033 |
+-----------------------+---------+------------+

+-----------------------------------+--------------+
|               Metric              |  HWThread 2  |
+-----------------------------------+--------------+
|        Runtime (RDTSC) [s]        |      12.9292 |
|        Runtime unhalted [s]       | 1.327911e-05 |
|            Clock [MHz]            |    2694.7005 |
|                CPI                |       4.4390 |
|  Memory read bandwidth [MBytes/s] |     422.5309 |
|  Memory read data volume [GBytes] |       5.4630 |
| Memory write bandwidth [MBytes/s] |     236.3059 |
| Memory write data volume [GBytes] |       3.0552 |
|    Memory bandwidth [MBytes/s]    |     658.8368 |
|    Memory data volume [GBytes]    |       8.5182 |
+-----------------------------------+--------------+

which seems fine to me.

Next, using the full node (32 threads):

$ likwid-perfctr -f -c N:0-31 -g MEM ../swift_cuda --hydro --threads=32 --steps=2 --pin greshoGPU64.yml 

I get the following result:

--------------------------------------------------------------------------------
Group 1: MEM
+-----------------------+---------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|         Event         | Counter | HWThread 0 | HWThread 2 | HWThread 4 | HWThread 6 | HWThread 8 | HWThread 10 | HWThread 12 | HWThread 14 | HWThread 16 | HWThread 18 | HWThread 20 | HWThread 22 | HWThread 24 | HWThread 26 | HWThread 28 | HWThread 30 | HWThread 1 | HWThread 3 | HWThread 5 | HWThread 7 | HWThread 9 | HWThread 11 | HWThread 13 | HWThread 15 | HWThread 17 | HWThread 19 | HWThread 21 | HWThread 23 | HWThread 25 | HWThread 27 | HWThread 29 | HWThread 31 |
+-----------------------+---------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|   INSTR_RETIRED_ANY   |  FIXC0  | 3133822573 | 2813486090 | 3043728189 | 2930692486 | 2722090250 |  3126264246 |  2548880161 |  3018866070 |  3300903006 |  3360986726 |  2687726446 |  2669957110 |  2760906756 |  2552515195 |  3063501993 |  2917143245 | 3197656772 | 2801757639 | 2887380234 | 2864979720 | 2814551018 |  3008879024 |  2734287716 |  2509797720 |  3216438776 |  3118694950 |  3036079433 |  2657812274 |  3121342400 |  2937343518 |  3113016086 |  3179790190 |
| CPU_CLK_UNHALTED_CORE |  FIXC1  | 3563069606 | 3369129026 | 3415539366 | 3408741114 | 3249072324 |  3389731617 |  3231691812 |  3429669079 |  3557693499 |  3491897883 |  3291795213 |  3195020413 |  3237587084 |  3197102545 |  3458301541 |  3361217145 | 3368475391 | 3154197548 | 3214583426 | 3176180589 | 3182672830 |  3228687378 |  3127536670 |  3093572505 |  3307864643 |  3261535145 |  3235589514 |  3174752605 |  3243541576 |  3257180131 |  3255170846 |  3336653887 |
|  CPU_CLK_UNHALTED_REF |  FIXC2  | 2929354060 | 2768032612 | 2806316848 | 2800689944 | 2669804120 |  2784883240 |  2655251744 |  2818180616 |  2923486852 |  2868811252 |  2705133224 |  2625390292 |  2660995488 |  2627231856 |  2841621664 |  2762731940 | 2770029564 | 2591788304 | 2641191752 | 2609574388 | 2614745064 |  2652721284 |  2569715572 |  2541390060 |  2718158216 |  2679830280 |  2658276336 |  2608362012 |  2665197220 |  2676587556 |  2674313960 |  2742247864 |
|      CAS_COUNT_RD     | MBOX0C0 |   48962304 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   44916250 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX0C1 |   60148475 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   58904072 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX1C0 |   47741198 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   44173297 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX1C1 |   59112460 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   57916540 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX2C0 |   48105100 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   46236395 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX2C1 |   59278825 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   59745346 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX3C0 |   45992964 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   42878464 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX3C1 |   58695026 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   57316165 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX4C0 |   47163558 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   42345235 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX4C1 |   59299798 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   56955623 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX5C0 |   45281466 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   42076025 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX5C1 |   57684251 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |   56757595 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------+---------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------+---------+--------------+------------+------------+--------------+
|            Event           | Counter |      Sum     |     Min    |     Max    |      Avg     |
+----------------------------+---------+--------------+------------+------------+--------------+
|   INSTR_RETIRED_ANY STAT   |  FIXC0  |  93851278012 | 2509797720 | 3360986726 | 2.932852e+09 |
| CPU_CLK_UNHALTED_CORE STAT |  FIXC1  | 105465453951 | 3093572505 | 3563069606 | 3.295795e+09 |
|  CPU_CLK_UNHALTED_REF STAT |  FIXC2  |  86662045184 | 2541390060 | 2929354060 |   2708188912 |
|      CAS_COUNT_RD STAT     | MBOX0C0 |     93878554 |          0 |   48962304 | 2.933705e+06 |
|      CAS_COUNT_WR STAT     | MBOX0C1 |    119052547 |          0 |   60148475 | 3.720392e+06 |
|      CAS_COUNT_RD STAT     | MBOX1C0 |     91914495 |          0 |   47741198 | 2.872328e+06 |
|      CAS_COUNT_WR STAT     | MBOX1C1 |    117029000 |          0 |   59112460 | 3.657156e+06 |
|      CAS_COUNT_RD STAT     | MBOX2C0 |     94341495 |          0 |   48105100 | 2.948172e+06 |
|      CAS_COUNT_WR STAT     | MBOX2C1 |    119024171 |          0 |   59745346 | 3.719505e+06 |
|      CAS_COUNT_RD STAT     | MBOX3C0 |     88871428 |          0 |   45992964 | 2.777232e+06 |
|      CAS_COUNT_WR STAT     | MBOX3C1 |    116011191 |          0 |   58695026 | 3.625350e+06 |
|      CAS_COUNT_RD STAT     | MBOX4C0 |     89508793 |          0 |   47163558 | 2.797150e+06 |
|      CAS_COUNT_WR STAT     | MBOX4C1 |    116255421 |          0 |   59299798 | 3.632982e+06 |
|      CAS_COUNT_RD STAT     | MBOX5C0 |     87357491 |          0 |   45281466 | 2.729922e+06 |
|      CAS_COUNT_WR STAT     | MBOX5C1 |    114441846 |          0 |   57684251 | 3.576308e+06 |
+----------------------------+---------+--------------+------------+------------+--------------+

+-----------------------------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|               Metric              | HWThread 0 | HWThread 2 | HWThread 4 | HWThread 6 | HWThread 8 | HWThread 10 | HWThread 12 | HWThread 14 | HWThread 16 | HWThread 18 | HWThread 20 | HWThread 22 | HWThread 24 | HWThread 26 | HWThread 28 | HWThread 30 | HWThread 1 | HWThread 3 | HWThread 5 | HWThread 7 | HWThread 9 | HWThread 11 | HWThread 13 | HWThread 15 | HWThread 17 | HWThread 19 | HWThread 21 | HWThread 23 | HWThread 25 | HWThread 27 | HWThread 29 | HWThread 31 |
+-----------------------------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|        Runtime (RDTSC) [s]        |    10.2164 |    10.2164 |    10.2164 |    10.2164 |    10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |    10.2164 |    10.2164 |    10.2164 |    10.2164 |    10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |     10.2164 |
|        Runtime unhalted [s]       |     1.5529 |     1.4684 |     1.4886 |     1.4856 |     1.4160 |      1.4773 |      1.4085 |      1.4947 |      1.5505 |      1.5219 |      1.4347 |      1.3925 |      1.4110 |      1.3934 |      1.5072 |      1.4649 |     1.4681 |     1.3747 |     1.4010 |     1.3843 |     1.3871 |      1.4072 |      1.3631 |      1.3483 |      1.4417 |      1.4215 |      1.4102 |      1.3836 |      1.4136 |      1.4196 |      1.4187 |      1.4542 |
|            Clock [MHz]            |  2790.8562 |  2792.7461 |  2792.5928 |  2792.6339 |  2792.3188 |   2792.8225 |   2792.6033 |   2792.3407 |   2792.2378 |   2792.8305 |   2792.0884 |   2792.3174 |   2791.6588 |   2792.1785 |   2792.4257 |   2791.5335 |  2790.1914 |  2792.3791 |  2792.6068 |  2792.6758 |  2792.8503 |   2792.6685 |   2792.5591 |   2793.0196 |   2792.2739 |   2792.5425 |   2792.7902 |   2792.7177 |   2792.3840 |   2792.1924 |   2792.8423 |   2791.8332 |
|                CPI                |     1.1370 |     1.1975 |     1.1222 |     1.1631 |     1.1936 |      1.0843 |      1.2679 |      1.1361 |      1.0778 |      1.0390 |      1.2248 |      1.1967 |      1.1727 |      1.2525 |      1.1289 |      1.1522 |     1.0534 |     1.1258 |     1.1133 |     1.1086 |     1.1308 |      1.0731 |      1.1438 |      1.2326 |      1.0284 |      1.0458 |      1.0657 |      1.1945 |      1.0391 |      1.1089 |      1.0457 |      1.0493 |
|  Memory read bandwidth [MBytes/s] |  1774.3858 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |  1645.2070 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|  Memory read data volume [GBytes] |    18.1278 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |    16.8080 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write bandwidth [MBytes/s] |  2218.9884 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |  2177.4958 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write data volume [GBytes] |    22.6700 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |    22.2461 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory bandwidth [MBytes/s]    |  3993.3742 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |  3822.7027 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory data volume [GBytes]    |    40.7978 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |    39.0541 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------------------+------------+-----------+-----------+-----------+
|                 Metric                 |     Sum    |    Min    |    Max    |    Avg    |
+----------------------------------------+------------+-----------+-----------+-----------+
|        Runtime (RDTSC) [s] STAT        |   326.9248 |   10.2164 |   10.2164 |   10.2164 |
|        Runtime unhalted [s] STAT       |    45.9650 |    1.3483 |    1.5529 |    1.4364 |
|            Clock [MHz] STAT            | 89354.7117 | 2790.1914 | 2793.0196 | 2792.3347 |
|                CPI STAT                |    36.1051 |    1.0284 |    1.2679 |    1.1283 |
|  Memory read bandwidth [MBytes/s] STAT |  3419.5928 |         0 | 1774.3858 |  106.8623 |
|  Memory read data volume [GBytes] STAT |    34.9358 |         0 |   18.1278 |    1.0917 |
| Memory write bandwidth [MBytes/s] STAT |  4396.4842 |         0 | 2218.9884 |  137.3901 |
| Memory write data volume [GBytes] STAT |    44.9161 |         0 |   22.6700 |    1.4036 |
|    Memory bandwidth [MBytes/s] STAT    |  7816.0769 |         0 | 3993.3742 |  244.2524 |
|    Memory data volume [GBytes] STAT    |    79.8519 |         0 |   40.7978 |    2.4954 |
+----------------------------------------+------------+-----------+-----------+-----------+

Here, I’m not entirely sure how to read and interpret the final output table correctly.

I take it SUM is the sum of the individual measurements for each thread? And the memory bandwidth numbers are averaged per thread as total memory moved divided by the runtime? Is that correct?

Next, I added the Likwid Marker API macros to my code. (I made sure to call LIKWID_MARKER_INIT and LIKWID_MARKER_CLOSE in serial regions of the code, the LIKWID_MARKER_REGISTER() ones in the parallel region followed by a barrier before callingLIKWID_MARKER_START() and LIKWID_MARKER_END().

First a run on a single thread:

likwid-perfctr -m -f -c N:0 -g MEM ../swift_cuda_likwid_individual --hydro --threads=1 --steps=2 --pin greshoGPU64.yml

I’m measuring 6 different parts of my code with different markers, so I’ll only post one of them here:

Region pack_force_pair, Group 1: MEM
+-------------------+------------+
|    Region Info    | HWThread 0 |
+-------------------+------------+
| RDTSC Runtime [s] |   0.336539 |
|     call count    |      26624 |
+-------------------+------------+

+-----------------------+---------+------------+
|         Event         | Counter | HWThread 0 |
+-----------------------+---------+------------+
|   INSTR_RETIRED_ANY   |  FIXC0  | 1244383000 |
| CPU_CLK_UNHALTED_CORE |  FIXC1  | 1164683000 |
|  CPU_CLK_UNHALTED_REF |  FIXC2  |  956697300 |
|      CAS_COUNT_RD     | MBOX0C0 |    2357508 |
|      CAS_COUNT_WR     | MBOX0C1 |     914567 |
|      CAS_COUNT_RD     | MBOX1C0 |    2269415 |
|      CAS_COUNT_WR     | MBOX1C1 |     962587 |
|      CAS_COUNT_RD     | MBOX2C0 |    2369541 |
|      CAS_COUNT_WR     | MBOX2C1 |     935900 |
|      CAS_COUNT_RD     | MBOX3C0 |    2426628 |
|      CAS_COUNT_WR     | MBOX3C1 |     948586 |
|      CAS_COUNT_RD     | MBOX4C0 |    2362531 |
|      CAS_COUNT_WR     | MBOX4C1 |     955147 |
|      CAS_COUNT_RD     | MBOX5C0 |    2234895 |
|      CAS_COUNT_WR     | MBOX5C1 |     932785 |
+-----------------------+---------+------------+

+-----------------------------------+------------+
|               Metric              | HWThread 0 |
+-----------------------------------+------------+
|        Runtime (RDTSC) [s]        |     0.3365 |
|        Runtime unhalted [s]       |     0.5076 |
|            Clock [MHz]            |  2793.3552 |
|                CPI                |     0.9360 |
|  Memory read bandwidth [MBytes/s] |  2666.2937 |
|  Memory read data volume [GBytes] |     0.8973 |
| Memory write bandwidth [MBytes/s] |  1074.3839 |
| Memory write data volume [GBytes] |     0.3616 |
|    Memory bandwidth [MBytes/s]    |  3740.6776 |
|    Memory data volume [GBytes]    |     1.2589 |
+-----------------------------------+------------+

which still looks sensible so far.

Now the same on 32 threads:

$ likwid-perfctr -m -f -c N:0-31 -g MEM ../swift_cuda_likwid_individual --hydro --threads=32 --steps=2 --pin greshoGPU64.yml 

Gives:

Region pack_force_pair, Group 1: MEM
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|    Region Info    | HWThread 0 | HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
| RDTSC Runtime [s] |   0.010442 |   0.011560 |   0.016923 |   0.014710 |   0.015549 |   0.015303 |   0.015466 |   0.016514 |   0.016758 |   0.015745 |    0.011916 |    0.016441 |    0.017599 |    0.016243 |    0.015920 |    0.014819 |    0.017088 |    0.014930 |    0.015209 |    0.015394 |    0.016010 |    0.016345 |    0.016773 |    0.011244 |    0.015535 |    0.015468 |    0.014643 |    0.015012 |    0.014299 |    0.015546 |    0.013667 |    0.015316 |
|     call count    |        563 |        575 |        863 |        854 |        809 |        870 |        823 |        950 |        880 |        877 |         693 |         948 |         900 |         922 |         854 |         866 |         887 |         857 |         811 |         874 |         853 |         936 |         857 |         658 |         860 |         892 |         772 |         858 |         769 |         888 |         748 |         857 |
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|         Event         | Counter | HWThread 0 | HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|   INSTR_RETIRED_ANY   |  FIXC0  |   26391780 |   27555630 |   40398940 |   39661550 |   37776000 |   41807650 |   38587430 |   44776580 |   41759230 |   42211240 |    31988210 |    45117470 |    43678630 |    44675960 |    40679420 |    40229980 |    42016820 |    40494310 |    37820350 |    40984130 |    40460380 |    44748260 |    41132770 |    31308840 |    40851330 |    42629310 |    37221040 |    40382140 |    37025220 |    42376680 |    35505310 |    41337410 |
| CPU_CLK_UNHALTED_CORE |  FIXC1  |   35996690 |   38588760 |   59669280 |   51223730 |   53052250 |   52840090 |   53087420 |   57386460 |   56885390 |   54128200 |    41335000 |    57534850 |    59159950 |    55996910 |    54084730 |    51440720 |    58282140 |    51746050 |    51742580 |    53229480 |    54850200 |    55878340 |    56908520 |    39288500 |    53259380 |    53675150 |    50637850 |    52139160 |    49117240 |    53294570 |    47102640 |    53487450 |
|  CPU_CLK_UNHALTED_REF |  FIXC2  |   29570550 |   31699240 |   49016130 |   42078780 |   43576440 |   43390140 |   43601560 |   47144660 |   46738210 |   44462040 |    33950760 |    47259760 |    48595600 |    45992550 |    44420910 |    42252660 |    47869160 |    42506120 |    42502800 |    43715550 |    45054240 |    45880680 |    46745750 |    32274980 |    43752160 |    44094130 |    41599270 |    42818270 |    40341450 |    43771940 |    38686090 |    43940670 |
|      CAS_COUNT_RD     | MBOX0C0 |    3730206 |    7418379 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX0C1 |    3021638 |    7785870 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX1C0 |    2975139 |    7080849 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX1C1 |    2258303 |    7193349 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX2C0 |    2930728 |    7184927 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX2C1 |    2227178 |    7489519 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX3C0 |    3253827 |    7024592 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX3C1 |    2603471 |    7411129 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX4C0 |    2834863 |    7318420 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX4C1 |    2161552 |    7713646 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX5C0 |    2699138 |    7035477 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX5C1 |    2055074 |    7071022 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------+---------+------------+----------+----------+--------------+
|            Event           | Counter |     Sum    |    Min   |    Max   |      Avg     |
+----------------------------+---------+------------+----------+----------+--------------+
|   INSTR_RETIRED_ANY STAT   |  FIXC0  | 1263590000 | 26391780 | 45117470 | 3.948719e+07 |
| CPU_CLK_UNHALTED_CORE STAT |  FIXC1  | 1667049680 | 35996690 | 59669280 | 5.209530e+07 |
|  CPU_CLK_UNHALTED_REF STAT |  FIXC2  | 1369303250 | 29570550 | 49016130 | 4.279073e+07 |
|      CAS_COUNT_RD STAT     | MBOX0C0 |   11148585 |        0 |  7418379 |  348393.2812 |
|      CAS_COUNT_WR STAT     | MBOX0C1 |   10807508 |        0 |  7785870 |  337734.6250 |
|      CAS_COUNT_RD STAT     | MBOX1C0 |   10055988 |        0 |  7080849 |  314249.6250 |
|      CAS_COUNT_WR STAT     | MBOX1C1 |    9451652 |        0 |  7193349 |  295364.1250 |
|      CAS_COUNT_RD STAT     | MBOX2C0 |   10115655 |        0 |  7184927 |  316114.2188 |
|      CAS_COUNT_WR STAT     | MBOX2C1 |    9716697 |        0 |  7489519 |  303646.7812 |
|      CAS_COUNT_RD STAT     | MBOX3C0 |   10278419 |        0 |  7024592 |  321200.5938 |
|      CAS_COUNT_WR STAT     | MBOX3C1 |   10014600 |        0 |  7411129 |  312956.2500 |
|      CAS_COUNT_RD STAT     | MBOX4C0 |   10153283 |        0 |  7318420 |  317290.0938 |
|      CAS_COUNT_WR STAT     | MBOX4C1 |    9875198 |        0 |  7713646 |  308599.9375 |
|      CAS_COUNT_RD STAT     | MBOX5C0 |    9734615 |        0 |  7035477 |  304206.7188 |
|      CAS_COUNT_WR STAT     | MBOX5C1 |    9126096 |        0 |  7071022 |  285190.5000 |
+----------------------------+---------+------------+----------+----------+--------------+

+-----------------------------------+-------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|               Metric              |  HWThread 0 |  HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-----------------------------------+-------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|        Runtime (RDTSC) [s]        |      0.0104 |      0.0116 |     0.0169 |     0.0147 |     0.0155 |     0.0153 |     0.0155 |     0.0165 |     0.0168 |     0.0157 |      0.0119 |      0.0164 |      0.0176 |      0.0162 |      0.0159 |      0.0148 |      0.0171 |      0.0149 |      0.0152 |      0.0154 |      0.0160 |      0.0163 |      0.0168 |      0.0112 |      0.0155 |      0.0155 |      0.0146 |      0.0150 |      0.0143 |      0.0155 |      0.0137 |      0.0153 |
|        Runtime unhalted [s]       |      0.0157 |      0.0168 |     0.0260 |     0.0223 |     0.0231 |     0.0230 |     0.0231 |     0.0250 |     0.0248 |     0.0236 |      0.0180 |      0.0251 |      0.0258 |      0.0244 |      0.0236 |      0.0224 |      0.0254 |      0.0226 |      0.0226 |      0.0232 |      0.0239 |      0.0244 |      0.0248 |      0.0171 |      0.0232 |      0.0234 |      0.0221 |      0.0227 |      0.0214 |      0.0232 |      0.0205 |      0.0233 |
|            Clock [MHz]            |   2793.0634 |   2793.1200 |  2793.1188 |  2793.0949 |  2793.3779 |  2794.1526 |  2793.6193 |  2792.8947 |  2792.5843 |  2793.2630 |   2793.4832 |   2793.2969 |   2793.2414 |   2793.5355 |   2793.6040 |   2793.3838 |   2793.5555 |   2793.2086 |   2793.2394 |   2793.7909 |   2793.3167 |   2794.4174 |   2793.2692 |   2793.0396 |   2793.0212 |   2792.9947 |   2792.9758 |   2793.9108 |   2793.5733 |   2793.6037 |   2793.6245 |   2792.9480 |
|                CPI                |      1.3639 |      1.4004 |     1.4770 |     1.2915 |     1.4044 |     1.2639 |     1.3758 |     1.2816 |     1.3622 |     1.2823 |      1.2922 |      1.2752 |      1.3544 |      1.2534 |      1.3295 |      1.2787 |      1.3871 |      1.2779 |      1.3681 |      1.2988 |      1.3557 |      1.2487 |      1.3835 |      1.2549 |      1.3037 |      1.2591 |      1.3605 |      1.2911 |      1.3266 |      1.2576 |      1.3266 |      1.2939 |
|  Memory read bandwidth [MBytes/s] | 112925.0662 | 238401.0588 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|  Memory read data volume [GBytes] |      1.1791 |      2.7560 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write bandwidth [MBytes/s] |  87815.3772 | 247269.3603 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write data volume [GBytes] |      0.9169 |      2.8585 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory bandwidth [MBytes/s]    | 200740.4434 | 485670.4191 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory data volume [GBytes]    |      2.0961 |      5.6145 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------------------+-------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------------------+-------------+-----------+-------------+------------+
|                 Metric                 |     Sum     |    Min    |     Max     |     Avg    |
+----------------------------------------+-------------+-----------+-------------+------------+
|        Runtime (RDTSC) [s] STAT        |      0.4840 |    0.0104 |      0.0176 |     0.0151 |
|        Runtime unhalted [s] STAT       |      0.7265 |    0.0157 |      0.0260 |     0.0227 |
|            Clock [MHz] STAT            |  89387.3230 | 2792.5843 |   2794.4174 |  2793.3538 |
|                CPI STAT                |     42.2802 |    1.2487 |      1.4770 |     1.3213 |
|  Memory read bandwidth [MBytes/s] STAT | 351326.1250 |         0 | 238401.0588 | 10978.9414 |
|  Memory read data volume [GBytes] STAT |      3.9351 |         0 |      2.7560 |     0.1230 |
| Memory write bandwidth [MBytes/s] STAT | 335084.7375 |         0 | 247269.3603 | 10471.3980 |
| Memory write data volume [GBytes] STAT |      3.7754 |         0 |      2.8585 |     0.1180 |
|    Memory bandwidth [MBytes/s] STAT    | 686410.8625 |         0 | 485670.4191 | 21450.3395 |
|    Memory data volume [GBytes] STAT    |      7.7106 |         0 |      5.6145 |     0.2410 |
+----------------------------------------+-------------+-----------+-------------+------------+

And here is where I am getting lost. The Max (and Sum) memory bandwidths are much bigger than what my measurements with likwid-bench have shown (<~ 100GByte/s). What is going on here? How am I to understand these numbers?

[TomTheBear]

Please redo your likwid-bench measurements either:

• -W N:4GB:32:1:2: Initialization is done per thread and only on cores, not SMT threads. If your system is configured with SNC/CoD (#numa_domains > #sockets), then I recommend this mode.
• -w S0:2GB:16:1:2 -w S1:2GB:16:1:2: Place one chunk of the stream on socket 0, the other on socket 1. Only cores, not SMT threads

This will definitely increase the achieved bandwidth as only local accesses happen. The reported bandwidth by likwid-bench does not contain potential write-allocates/read-for-ownerships, thus it is beneficial to wrap it in with likwid-perfctr -m -g MEM to see the actual bandwidth at hardware level.

The reported bandwidth seems quite high in general but it might be caused by your short region times (563 in 0.0104s for HWthread 0). With these short region times, the overhead of accessing the hardware counters (more so for the distant Uncore counters) might skew the results. I have seen it also with likwid-perfctr in timeline mode with high-frequency reads. The counts can be way to high. For that, increase the runtime of each region traversal.

[mladenivkovic]

Hi Thomas

Thank you so much for the prompt reply. It’s indeed as you say, re-running the benchmarks correctly gives me a significantly higher peak memory bandwidth:

Running with -W N:4GB:32:1:2 :

# copy_mem benchmark
likwid-bench -t copy_mem -W N:4GB:32:1:2 
MByte/s:		170209.47

# load benchmark
likwid-bench -t load -W N:4GB:32:1:2 
MByte/s:		188044.17

# store benchmark
likwid-bench -t store -W N:4GB:32:1:2 
MByte/s:		93570.73

# Triad benchmark
likwid-bench -t triad -W N:4GB:32:1:2 
MByte/s:		158884.79

Running with

# copy_mem benchmark
likwid-bench -t copy_mem -w S0:2GB:16:1:2 -w S1:2GB:16:1:2
MByte/s:		170176.99

# load benchmark
likwid-bench -t load -w S0:2GB:16:1:2 -w S1:2GB:16:1:2
MByte/s:		188511.60

# store benchmark
likwid-bench -t store -w S0:2GB:16:1:2 -w S1:2GB:16:1:2
MByte/s:		93849.52

# Triad benchmark
likwid-bench -t triad -w S0:2GB:16:1:2 -w S1:2GB:16:1:2
MByte/s:		158061.00

Next, I also re-ran likwid-pefctr on my code with the markers, where I increased both my problem size and the time spent in each region. The output now appears more sensible:

Region pack_force_pair, Group 1: MEM
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|    Region Info    | HWThread 0 | HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
| RDTSC Runtime [s] |   0.908736 |   0.985275 |   1.403276 |   1.417688 |   1.413008 |   1.422084 |   1.419777 |   1.434309 |   1.413211 |   1.386697 |    1.413323 |    1.412368 |    1.412410 |    1.379927 |    1.391072 |    1.400529 |    1.425196 |    1.399566 |    1.401586 |    1.421314 |    1.369755 |    1.377147 |    1.423493 |    1.422127 |    1.368444 |    1.394394 |    1.400524 |    1.411471 |    1.412468 |    1.387591 |    1.401345 |    1.362512 |
|     call count    |       5375 |       5933 |       8320 |       8549 |       8457 |       8655 |       8542 |       8652 |       8454 |       8510 |        8386 |        8615 |        8532 |        8571 |        8463 |        8646 |        8494 |        8531 |        8336 |        8599 |        8347 |        8466 |        8519 |        8583 |        8345 |        8560 |        8346 |        8672 |        8433 |        8491 |        8393 |        8465 |
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|         Event         | Counter | HWThread 0 | HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|   INSTR_RETIRED_ANY   |  FIXC0  | 1629179000 | 1807012000 | 2565740000 | 2629167000 | 2557770000 | 2653278000 | 2612223000 | 2654072000 | 2600224000 | 2578248000 |  2581951000 |  2631825000 |  2610529000 |  2611076000 |  2591370000 |  2636583000 |  2601588000 |  2594985000 |  2565944000 |  2616371000 |  2536315000 |  2574103000 |  2610363000 |  2622145000 |  2537105000 |  2619296000 |  2555997000 |  2644250000 |  2581294000 |  2587128000 |  2561103000 |  2572648000 |
| CPU_CLK_UNHALTED_CORE |  FIXC1  | 2610643000 | 2821003000 | 4018885000 | 4056338000 | 4045512000 | 4063218000 | 4059642000 | 4101918000 | 4036863000 | 3967851000 |  4026625000 |  4035975000 |  4034532000 |  3945418000 |  3976067000 |  4007295000 |  4077728000 |  3999182000 |  4010968000 |  4060399000 |  3917862000 |  3942345000 |  4073243000 |  4060639000 |  3909697000 |  3986115000 |  4005392000 |  4036725000 |  4039089000 |  3971164000 |  4008585000 |  3894536000 |
|  CPU_CLK_UNHALTED_REF |  FIXC2  | 2144461000 | 2317267000 | 3301234000 | 3332033000 | 3323109000 | 3337626000 | 3334660000 | 3369425000 | 3316012000 | 3259335000 |  3307583000 |  3315293000 |  3314090000 |  3240917000 |  3266053000 |  3291689000 |  3349571000 |  3285065000 |  3294738000 |  3335357000 |  3218261000 |  3238360000 |  3345921000 |  3335526000 |  3211527000 |  3274349000 |  3290146000 |  3315912000 |  3317835000 |  3262036000 |  3292772000 |  3199111000 |
|      CAS_COUNT_RD     | MBOX0C0 |  140059800 |  114134700 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX0C1 |  116899800 |   89347920 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX1C0 |  135786700 |  111934400 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX1C1 |  112327900 |   87760580 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX2C0 |  138354600 |  111283300 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX2C1 |  114834100 |   87095650 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX3C0 |  139354800 |  111599100 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX3C1 |  116127900 |   87186360 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX4C0 |  135790300 |  113789400 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX4C1 |  112758900 |   89042180 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX5C0 |  134613000 |  114555300 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX5C1 |  111137000 |   89379330 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------+---------+--------------+------------+------------+--------------+
|            Event           | Counter |      Sum     |     Min    |     Max    |      Avg     |
+----------------------------+---------+--------------+------------+------------+--------------+
|   INSTR_RETIRED_ANY STAT   |  FIXC0  |  81330882000 | 1629179000 | 2654072000 | 2.541590e+09 |
| CPU_CLK_UNHALTED_CORE STAT |  FIXC1  | 125801454000 | 2610643000 | 4101918000 | 3.931295e+09 |
|  CPU_CLK_UNHALTED_REF STAT |  FIXC2  | 103337274000 | 2144461000 | 3369425000 | 3.229290e+09 |
|      CAS_COUNT_RD STAT     | MBOX0C0 |    254194500 |          0 |  140059800 | 7.943578e+06 |
|      CAS_COUNT_WR STAT     | MBOX0C1 |    206247720 |          0 |  116899800 | 6.445241e+06 |
|      CAS_COUNT_RD STAT     | MBOX1C0 |    247721100 |          0 |  135786700 | 7.741284e+06 |
|      CAS_COUNT_WR STAT     | MBOX1C1 |    200088480 |          0 |  112327900 |      6252765 |
|      CAS_COUNT_RD STAT     | MBOX2C0 |    249637900 |          0 |  138354600 | 7.801184e+06 |
|      CAS_COUNT_WR STAT     | MBOX2C1 |    201929750 |          0 |  114834100 | 6.310305e+06 |
|      CAS_COUNT_RD STAT     | MBOX3C0 |    250953900 |          0 |  139354800 | 7.842309e+06 |
|      CAS_COUNT_WR STAT     | MBOX3C1 |    203314260 |          0 |  116127900 | 6.353571e+06 |
|      CAS_COUNT_RD STAT     | MBOX4C0 |    249579700 |          0 |  135790300 | 7.799366e+06 |
|      CAS_COUNT_WR STAT     | MBOX4C1 |    201801080 |          0 |  112758900 | 6.306284e+06 |
|      CAS_COUNT_RD STAT     | MBOX5C0 |    249168300 |          0 |  134613000 | 7.786509e+06 |
|      CAS_COUNT_WR STAT     | MBOX5C1 |    200516330 |          0 |  111137000 | 6.266135e+06 |
+----------------------------+---------+--------------+------------+------------+--------------+

+-----------------------------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|               Metric              |  HWThread 0 | HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-----------------------------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|        Runtime (RDTSC) [s]        |      0.9087 |     0.9853 |     1.4033 |     1.4177 |     1.4130 |     1.4221 |     1.4198 |     1.4343 |     1.4132 |     1.3867 |      1.4133 |      1.4124 |      1.4124 |      1.3799 |      1.3911 |      1.4005 |      1.4252 |      1.3996 |      1.4016 |      1.4213 |      1.3698 |      1.3771 |      1.4235 |      1.4221 |      1.3684 |      1.3944 |      1.4005 |      1.4115 |      1.4125 |      1.3876 |      1.4013 |      1.3625 |
|        Runtime unhalted [s]       |      1.1378 |     1.2295 |     1.7515 |     1.7679 |     1.7632 |     1.7709 |     1.7693 |     1.7877 |     1.7594 |     1.7293 |      1.7549 |      1.7590 |      1.7584 |      1.7195 |      1.7329 |      1.7465 |      1.7772 |      1.7430 |      1.7481 |      1.7696 |      1.7075 |      1.7182 |      1.7752 |      1.7697 |      1.7040 |      1.7373 |      1.7457 |      1.7593 |      1.7604 |      1.7308 |      1.7471 |      1.6974 |
|            Clock [MHz]            |   2793.2682 |  2793.2562 |  2793.2678 |  2793.2393 |  2793.2655 |  2793.2883 |  2793.3122 |  2793.2801 |  2793.2591 |  2793.2490 |   2793.2753 |   2793.2503 |   2793.2652 |   2793.2411 |   2793.2755 |   2793.2887 |   2793.2665 |   2793.2545 |   2793.2616 |   2793.2493 |   2793.2588 |   2793.2693 |   2793.2380 |   2793.2728 |   2793.2823 |   2793.2394 |   2793.2716 |   2793.2478 |   2793.2637 |   2793.2665 |   2793.2689 |   2793.2494 |
|                CPI                |      1.6024 |     1.5611 |     1.5664 |     1.5428 |     1.5817 |     1.5314 |     1.5541 |     1.5455 |     1.5525 |     1.5390 |      1.5595 |      1.5335 |      1.5455 |      1.5110 |      1.5343 |      1.5199 |      1.5674 |      1.5411 |      1.5632 |      1.5519 |      1.5447 |      1.5315 |      1.5604 |      1.5486 |      1.5410 |      1.5218 |      1.5671 |      1.5266 |      1.5648 |      1.5350 |      1.5652 |      1.5138 |
|  Memory read bandwidth [MBytes/s] |  58029.3632 | 43994.7799 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|  Memory read data volume [GBytes] |     52.7334 |    43.3470 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write bandwidth [MBytes/s] |  48178.4192 | 34414.7261 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write data volume [GBytes] |     43.7815 |    33.9080 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory bandwidth [MBytes/s]    | 106207.7824 | 78409.5061 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory data volume [GBytes]    |     96.5149 |    77.2549 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------------------+-------------+-----------+-------------+-----------+
|                 Metric                 |     Sum     |    Min    |     Max     |    Avg    |
+----------------------------------------+-------------+-----------+-------------+-----------+
|        Runtime (RDTSC) [s] STAT        |     43.9926 |    0.9087 |      1.4343 |    1.3748 |
|        Runtime unhalted [s] STAT       |     54.8282 |    1.1378 |      1.7877 |    1.7134 |
|            Clock [MHz] STAT            |  89384.4422 | 2793.2380 |   2793.3122 | 2793.2638 |
|                CPI STAT                |     49.5247 |    1.5110 |      1.6024 |    1.5476 |
|  Memory read bandwidth [MBytes/s] STAT | 102024.1431 |         0 |  58029.3632 | 3188.2545 |
|  Memory read data volume [GBytes] STAT |     96.0804 |         0 |     52.7334 |    3.0025 |
| Memory write bandwidth [MBytes/s] STAT |  82593.1453 |         0 |  48178.4192 | 2581.0358 |
| Memory write data volume [GBytes] STAT |     77.6895 |         0 |     43.7815 |    2.4278 |
|    Memory bandwidth [MBytes/s] STAT    | 184617.2885 |         0 | 106207.7824 | 5769.2903 |
|    Memory data volume [GBytes] STAT    |    173.7698 |         0 |     96.5149 |    5.4303 |
+----------------------------------------+-------------+-----------+-------------+-----------+

as it’s definitely within the limits of the (updated) peak bandwidths reported.

However, at this point, a second question on my end arises. If I just increase my problem size without increasing the time spent in the marked regions, I once again obtain suspicious/impossible results:

Region pack_force_pair, Group 1: MEM
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|    Region Info    | HWThread 0 | HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
| RDTSC Runtime [s] |   1.010754 |   1.039080 |   1.511886 |   1.542096 |   1.495828 |   1.544144 |   1.502702 |   1.551239 |   1.515239 |   1.543327 |    1.491576 |    1.540171 |    1.516787 |    1.522129 |    1.499596 |    1.538549 |    1.519402 |    1.538117 |    1.519831 |    1.549201 |    1.518215 |    1.531220 |    1.515241 |    1.564619 |    1.504471 |    1.526878 |    1.508322 |    1.529661 |    1.514827 |    1.537452 |    1.497201 |    1.529903 |
|     call count    |      43190 |      44825 |      64762 |      69778 |      65815 |      69848 |      65767 |      69949 |      66603 |      69905 |       66448 |       70135 |       66860 |       70079 |       66517 |       70100 |       66428 |       69570 |       66122 |       69862 |       67001 |       69942 |       66243 |       69569 |       66433 |       70000 |       66107 |       69759 |       66259 |       69713 |       66532 |       69799 |
+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|         Event         | Counter | HWThread 0 | HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|   INSTR_RETIRED_ANY   |  FIXC0  | 2027699000 | 2113596000 | 3045839000 | 3300643000 | 3099609000 | 3301483000 | 3098428000 | 3311194000 | 3140171000 | 3313062000 |  3133363000 |  3323702000 |  3163698000 |  3325819000 |  3151482000 |  3333018000 |  3148373000 |  3305735000 |  3132653000 |  3320209000 |  3178326000 |  3326067000 |  3146609000 |  3326412000 |  3161975000 |  3337454000 |  3143100000 |  3331606000 |  3153746000 |  3335750000 |  3169622000 |  3332416000 |
| CPU_CLK_UNHALTED_CORE |  FIXC1  | 3409540000 | 3392907000 | 5070661000 | 5137968000 | 4947119000 | 5071921000 | 4958122000 | 5126604000 | 4996287000 | 5088765000 |  4908580000 |  5103605000 |  5002373000 |  5027081000 |  4949793000 |  5084187000 |  5017362000 |  5090442000 |  5038497000 |  5171595000 |  5030606000 |  5089577000 |  5010922000 |  5185277000 |  4951784000 |  5055587000 |  4976192000 |  5086847000 |  5020605000 |  5075197000 |  4984747000 |  5118814000 |
|  CPU_CLK_UNHALTED_REF |  FIXC2  | 2800566000 | 2787069000 | 4165115000 | 4220422000 | 4063645000 | 4166136000 | 4072763000 | 4211235000 | 4104162000 | 4179996000 |  4032097000 |  4192493000 |  4109127000 |  4129427000 |  4065934000 |  4176308000 |  4121333000 |  4181428000 |  4138808000 |  4248004000 |  4132278000 |  4180620000 |  4116267000 |  4259441000 |  4067514000 |  4152818000 |  4087549000 |  4178403000 |  4124019000 |  4168899000 |  4094716000 |  4204843000 |
|      CAS_COUNT_RD     | MBOX0C0 |  697830700 |  390894100 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX0C1 |  697798500 |  321538800 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX1C0 |  713717900 |  471843200 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX1C1 |  743714500 |  375468600 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX2C0 |  730750300 |  348288700 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX2C1 |  770608600 |  292482200 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX3C0 |  773041300 |  385226600 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX3C1 |  815561800 |  323501600 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX4C0 |  612564900 |  409212600 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX4C1 |  617320400 |  345468200 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_RD     | MBOX5C0 |  699120900 |  436717300 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|      CAS_COUNT_WR     | MBOX5C1 |  742137300 |  360689300 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------+---------+--------------+------------+------------+--------------+
|            Event           | Counter |      Sum     |     Min    |     Max    |      Avg     |
+----------------------------+---------+--------------+------------+------------+--------------+
|   INSTR_RETIRED_ANY STAT   |  FIXC0  | 101032859000 | 2027699000 | 3337454000 | 3.157277e+09 |
| CPU_CLK_UNHALTED_CORE STAT |  FIXC1  | 158179564000 | 3392907000 | 5185277000 |   4943111375 |
|  CPU_CLK_UNHALTED_REF STAT |  FIXC2  | 129933435000 | 2787069000 | 4259441000 | 4.060420e+09 |
|      CAS_COUNT_RD STAT     | MBOX0C0 |   1088724800 |          0 |  697830700 |     34022650 |
|      CAS_COUNT_WR STAT     | MBOX0C1 |   1019337300 |          0 |  697798500 | 3.185429e+07 |
|      CAS_COUNT_RD STAT     | MBOX1C0 |   1185561100 |          0 |  713717900 | 3.704878e+07 |
|      CAS_COUNT_WR STAT     | MBOX1C1 |   1119183100 |          0 |  743714500 | 3.497447e+07 |
|      CAS_COUNT_RD STAT     | MBOX2C0 |   1079039000 |          0 |  730750300 | 3.371997e+07 |
|      CAS_COUNT_WR STAT     | MBOX2C1 |   1063090800 |          0 |  770608600 | 3.322159e+07 |
|      CAS_COUNT_RD STAT     | MBOX3C0 |   1158267900 |          0 |  773041300 | 3.619587e+07 |
|      CAS_COUNT_WR STAT     | MBOX3C1 |   1139063400 |          0 |  815561800 | 3.559573e+07 |
|      CAS_COUNT_RD STAT     | MBOX4C0 |   1021777500 |          0 |  612564900 | 3.193055e+07 |
|      CAS_COUNT_WR STAT     | MBOX4C1 |    962788600 |          0 |  617320400 | 3.008714e+07 |
|      CAS_COUNT_RD STAT     | MBOX5C0 |   1135838200 |          0 |  699120900 | 3.549494e+07 |
|      CAS_COUNT_WR STAT     | MBOX5C1 |   1102826600 |          0 |  742137300 | 3.446333e+07 |
+----------------------------+---------+--------------+------------+------------+--------------+

+-----------------------------------+-------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|               Metric              |  HWThread 0 |  HWThread 1 | HWThread 2 | HWThread 3 | HWThread 4 | HWThread 5 | HWThread 6 | HWThread 7 | HWThread 8 | HWThread 9 | HWThread 10 | HWThread 11 | HWThread 12 | HWThread 13 | HWThread 14 | HWThread 15 | HWThread 16 | HWThread 17 | HWThread 18 | HWThread 19 | HWThread 20 | HWThread 21 | HWThread 22 | HWThread 23 | HWThread 24 | HWThread 25 | HWThread 26 | HWThread 27 | HWThread 28 | HWThread 29 | HWThread 30 | HWThread 31 |
+-----------------------------------+-------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|        Runtime (RDTSC) [s]        |      1.0108 |      1.0391 |     1.5119 |     1.5421 |     1.4958 |     1.5441 |     1.5027 |     1.5512 |     1.5152 |     1.5433 |      1.4916 |      1.5402 |      1.5168 |      1.5221 |      1.4996 |      1.5385 |      1.5194 |      1.5381 |      1.5198 |      1.5492 |      1.5182 |      1.5312 |      1.5152 |      1.5646 |      1.5045 |      1.5269 |      1.5083 |      1.5297 |      1.5148 |      1.5375 |      1.4972 |      1.5299 |
|        Runtime unhalted [s]       |      1.4859 |      1.4787 |     2.2099 |     2.2392 |     2.1560 |     2.2104 |     2.1608 |     2.2343 |     2.1775 |     2.2178 |      2.1393 |      2.2242 |      2.1801 |      2.1909 |      2.1572 |      2.2158 |      2.1867 |      2.2185 |      2.1959 |      2.2539 |      2.1924 |      2.2181 |      2.1839 |      2.2598 |      2.1581 |      2.2033 |      2.1687 |      2.2169 |      2.1881 |      2.2119 |      2.1724 |      2.2309 |
|            Clock [MHz]            |   2793.4668 |   2793.3012 |  2793.3871 |  2793.3738 |  2793.3807 |  2793.3964 |  2793.3259 |  2793.2759 |  2793.2925 |  2793.3803 |   2793.3055 |   2793.1756 |   2793.3158 |   2793.3131 |   2793.3172 |   2793.3318 |   2793.3880 |   2793.3439 |   2793.3108 |   2793.4000 |   2793.3433 |   2793.4090 |   2793.2361 |   2793.2698 |   2793.3553 |   2793.3299 |   2793.3650 |   2793.3920 |   2793.3730 |   2793.3481 |   2793.2697 |   2793.2712 |
|                CPI                |      1.6815 |      1.6053 |     1.6648 |     1.5567 |     1.5960 |     1.5363 |     1.6002 |     1.5483 |     1.5911 |     1.5360 |      1.5666 |      1.5355 |      1.5812 |      1.5115 |      1.5706 |      1.5254 |      1.5936 |      1.5399 |      1.6084 |      1.5576 |      1.5828 |      1.5302 |      1.5925 |      1.5588 |      1.5660 |      1.5148 |      1.5832 |      1.5268 |      1.5919 |      1.5215 |      1.5727 |      1.5361 |
|  Memory read bandwidth [MBytes/s] | 267651.3415 | 150421.2188 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|  Memory read data volume [GBytes] |    270.5297 |    156.2997 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write bandwidth [MBytes/s] | 277789.6802 | 124365.3201 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
| Memory write data volume [GBytes] |    280.7770 |    129.2255 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory bandwidth [MBytes/s]    | 545441.0217 | 274786.5389 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
|    Memory data volume [GBytes]    |    551.3067 |    285.5252 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |          0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |           0 |
+-----------------------------------+-------------+-------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

+----------------------------------------+-------------+-----------+-------------+------------+
|                 Metric                 |     Sum     |    Min    |     Max     |     Avg    |
+----------------------------------------+-------------+-----------+-------------+------------+
|        Runtime (RDTSC) [s] STAT        |     47.7695 |    1.0108 |      1.5646 |     1.4928 |
|        Runtime unhalted [s] STAT       |     68.9375 |    1.4787 |      2.2598 |     2.1543 |
|            Clock [MHz] STAT            |  89386.7447 | 2793.1756 |   2793.4668 |  2793.3358 |
|                CPI STAT                |     50.1838 |    1.5115 |      1.6815 |     1.5682 |
|  Memory read bandwidth [MBytes/s] STAT | 418072.5603 |         0 | 267651.3415 | 13064.7675 |
|  Memory read data volume [GBytes] STAT |    426.8294 |         0 |    270.5297 |    13.3384 |
| Memory write bandwidth [MBytes/s] STAT | 402155.0003 |         0 | 277789.6802 | 12567.3438 |
| Memory write data volume [GBytes] STAT |    410.0025 |         0 |    280.7770 |    12.8126 |
|    Memory bandwidth [MBytes/s] STAT    | 820227.5606 |         0 | 545441.0217 | 25632.1113 |
|    Memory data volume [GBytes] STAT    |    836.8319 |         0 |    551.3067 |    26.1510 |
+----------------------------------------+-------------+-----------+-------------+------------+

At this point, I’m wondering if there is a reliable way to diagnose that I’m in the regime where my access times are too short and the overheads become significant enough to skew the measurements. For example, the first, “working” output above has an average time per access of 1.9e-4s, while the second, “non-working” one, gives ~2.3e-5s. Is there a time scale threshold you could advise to be on the lookout for?

Thanks again, Mladen

[TomTheBear]

One way of identification is the ratio between Runtime (RDTSC) [s] STAT SUM and Runtime unhalted [s] STAT SUM. It is not a well research identification but it gives a hint. It should be around 1.0. In the "reasonable" case, it is 1.25 while in the "too short" one 1.44.

In the timeline mode documentation, I recommend not going below 100ms.

[TomTheBear]

Any more questions?

[mladenivkovic]

Hi Thomas

Thank you very much, this has been very helpful. I've got one remaining question.

I've been chatting with Tobias about my attempts to measure my code's behaviour with likwid and he mentioned that a while ago, he was told that the marker API isn't/wasn't intended to be used within a task-parallel region. (I'm essentially trying to measure the performance of specific tasks, while other threads are working on different stuff.) Could you confirm whether this is or isn't the case?

[TomTheBear]

Tobias (Weinzierl, Durham University) is correct, yes. Issue with task-based runtime systems is that the tasks are commonly rather small and you commonly instrument them so that each task execution is measured. This causes a high overhead from the markers (more specifically the accesses to the hardware performance counters). This overhead caused this issue ticket in the first place. Things are getting more complex if tasks can be yielded or preempted/moved to other hardware threads. The hardware thread shouldn't change in between start and stop, that's why application threads need to be pinned to hardware threads in MarkerAPI mode. (In general, it should be possible to re-pin outside of marked regions).

What you could try is to reduce the number of MarkerAPI calls but I cannot tell whether this really helps.

int task(taskdata* data)
{
  if (data->marker_counter % data->marker_freq == 0) LIKWID_MARKER_START("task_of_interest");
  <task code>
  if (data->marker_counter % data->marker_freq == 0) LIKWID_MARKER_STOP("task_of_interest");
}

For a task-based runtime system, I would probably try out ebpf based monitoring with uprobe and uretprobe. Since it is done at kernel-level, you can also access hardware performance counters with less overhead (no context switch). Maybe you can achieve something like that also from CLI with perf (perf probe -x /path/to/executable -a function) and then something like perf record -e probe_executable:function,instructions,cycles -aR /path/to/executable <args>. No clue whether that works because commonly perf probes print out the function arguments and do not restrict measurements to their execution, but it is worth investigating.