Interactive mode for reviewing the status of tests?

[mjambon]

Summary: An optional extension of Testo could provide an interactive, text-based interface for listing, inspecting, and approving tests quickly and safely.

An issue is that we often find ourselves with too much or too little output. This has been mitigated by first printing the output of all tests (in either run or status) and then recapitulating by printing only the tests that need attention. This is not good enough when test logs are long because it gets hard to scroll and see the other failing tests. We could add an option to control the verbosity like alcotest's --tail-errors but it may not be sufficient.

This is related to the general issue of navigating from one test to another. When working on testing a feature, often more than one test is involved. I use -s Suite_name to select the tests I'm working on and this part is satisfying. However, I often want to do the following and it's difficult:

• Viewing the captured output: requires copying the path that is printed for us and opening the file with less. After this, more scrolling up is need to get back to the status of the other tests. At this point, I may re-run ./test status which is relatively quick but not ideal. Printing the captured output by default would be too much. We're currently showing diffs but only if the test failed and an approved snapshot exists. If the test is new, reviewing the output is a little tedious.
• Hiding test logs to see the list of tests: we need a way to do this. It's not clear if printing all the failed tests in a one-line format at the end of the existing output is desirable. It would make the list of tests more obvious at the expense of not seeing the details of the last failed test. Maybe it's not too bad.
• Approving the output of the test I just reviewed. It requires grabbing its ID or category/name and running ./test approve -s ID which is more complicated than it could be and error-prone.

What if we could navigate tests interactively? We could offer an interface that shows a list of files. There would be options to filter the list of tests by status (e.g. press f to toggle between failed-only and all tests) or by search text (ID, category, or name). Hitting the right arrow key would take us to the view of a single test. From there, we could expand or collapse different output files of interest, notably the captured output and the uncaptured output (logs).

I have no experience in implementing text-based interactive interfaces. It seems difficult. It also seems like something we want here. Please chime in if you have ideas on the feasibility of this.

[mjambon]

Is https://github.com/mbacarella/curses the best library for the job? Does anyone have experience with it?

The Curses library won't work on Windows, see https://en.wikipedia.org/wiki/Curses_(programming_library)#Portability. For this reason and to keep dependencies minimal, an interactive extension of Testo should be optional and provided by a extra Opam package as we do for Lwt.

[mjambon]

A non-interactive way of addressing most of the problems would be to export the test data to HTML.

Pros:

• allows viewing the test data and zooming in and out to see more or fewer details
• no need to implement our own file browser

Cons:

• approving the output of a test would still require typing or pasting a command into a console
• selecting tests by tag or by keyword may not be easy without using JavaScript