Should a test id have the index prefix?

[beraldoleal]

Depending on the size of a suite, the same test can have different ids because we prefix the test id with the index.

Ex: avocado run /bin/false /bin/true and avocado run /bin/true it will produce 2-/bin/true and 1-/bin/true, respectively.

Some might need to run a quick analysis over multiple job executions to find "test flakiness". However, since we are using this prefix as part of the test_id, the same test might be computed as different tests.

So,

1. Should we hide this prefix on the test_id?
2. Should we take into account other attributes (like parameters and args)?
3. Or the way it is today is enough?

What you think about this?

[clebergnu]

It's worth revisiting the definitions of Test ID and Test Name. I believe the issue here is really not what to do with the Test ID, but where to use the Test Name instead.

[beraldoleal]

@clebergnu Thanks for the quick reply.

Maybe we need to fix somethings then. plugins/jsonresult.py:45:

tests.append({'id': str(test.get('name', UNKNOWN)),

IIUC, today Avocado it is saving the prefix into the test.name and we are showing this as the test's id.

[clebergnu]

Absolutely... the internal attribute names seem off. Given that the JSON result file our custom format, it could include a name field too.

I wonder if, for other results that may not have the room for other fields (for instance xunit), if the test name shouldn't be used instead. If we intend to use data that is derived from Avocado's xunit files, that seems to be the way to go.

[beraldoleal]

Agree, if it is an external format that we don't have control, some kind of conversion is needed, even if confuses us.

[beraldoleal]

I created a new issue #5116 for tracking this.