kitchen-pester reports success when containers fail

[corbob]

🗣️ Foreword

Thank for taking the time to fill this bug report fully. Without it we may not be able to fix the bug, and the issue may be closed without resolution.

👻 Brief Description

Pester sets $LASTEXITCODE when it runs, but kitchen-pester overrides it with the number of failed tests. This results in non-failed test runs even when a container fails.

Version

1.2.0

Environment

[Details about the environment such as the Operating System, cookbook details, etc...]

Scenario

Pester goes through a number of phases as part of setting up tests. When one of these fails, it may be able to continue and complete other tests, but it will still report a failure in $LASTEXITCODE. If you're relying on the test results and not examining the output, you may miss that a test container failed.

Steps to Reproduce

Create a test file with the contents:

Describe 'testing' {
    throw 'oops'
    it 'succeeds'{
        $true | Should -BeTrue
    }
}

I have created a reproduction repository here: https://github.com/corbob/testkitchen-repros/. Logs from this are here: https://github.com/corbob/testkitchen-repros/tree/main/reproduction-logs/kitchen-pester-107

Expected Result

This should fail the tests.

Actual Result

This does not fail the tests.

➕ Additional context

This appears to be related to https://github.com/test-kitchen/kitchen-pester/blob/main/lib/kitchen/verifier/pester.rb#L314 where the exit code is being set to the number of failed tests, and overwriting Pester's set exit code.

I realize there is some details missing here. I intend to come back and fill this out further when I have more time and can generate a full Test Kitchen reproduction.