Property-Based testing tool for futhark (solving #2163)#2405
Draft
Black-Speedy wants to merge 40 commits intodiku-dk:masterfrom
Draft
Property-Based testing tool for futhark (solving #2163)#2405Black-Speedy wants to merge 40 commits intodiku-dk:masterfrom
Black-Speedy wants to merge 40 commits intodiku-dk:masterfrom
Conversation
Very ugly but a start, also our prints dont work well with the current test printer.
…working) Currently it makes all test cases fail which is very bad
…ectory It now somewhat works, however it is a little ugly. To TestResult we added PropOut which needs to be explictly handled in runTestM, pureTestResults, and runTests. We should probably also move our logic of printing to stdout and using capture to get it as a string to just using a string instead or find another strategy
Black-Speedy
changed the title
We still need to find out how to add properties to runs so that we are counted. For now we have just added 2 (which is bad but works for our use since we always have a passing and failing while testing)
Now it is a requirement to in addition to the attribute also declare a test field meaning: -- == -- property: some_prop Additionally if you have multiple tests in the form of multiple attributes for a single property you additionally have to write a num field in the test block with the number of tests. If this is not done it will not complain but the result will not be entirely correct. Like the regular test if the entrypoint/propertyname is incorrect only a single of the test cases will count as a failure (could probably update this if needed/wanted)
Now -- == -- property: prop_simple_fail will match against something like: #[prop(gen(gen_simple), shrink(auto))] entry prop_simple_fail (x: i8) : bool = but -- == -- property: prop_simple_fai will not match and if no atttributed function of that name exists, it will count as a fail.
…unit tests and prop tests were in the same file Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
Member
|
I would like to see the tests related to testing property-based testing to a directory |
Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
…te + some cleanup Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
… prettier, moving them to messages instead of errors/fail Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
Still needs some cleanup but now extra info is provided if a property hasn't finished before timout is reached Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
Would probably benefit from some color, can maybe do that later. This error should be relatively rare anyway.
Have changed a large portion of fails to error messages but code still needs an overall look to finish and also make code easier to read.
Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
…ailures and user failures. Also moved tests. Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
Changed tactic to be a random number that can be used for whatever the user feels like. The return type no longer has a status, the shrinker logic will automatically stop after a property no longer fails a consecutive number of times (default is 5 at the moment) which can be set from cli with num-tries. Co-Authored-By: Simon August Mørk <100700440+simoerk@users.noreply.github.com>
… script. We should take a greater look at the array test, as they were harder to change since more is going on. It would probably be better to make the array tests simpler. Also look into making the shrinker library so that we can make fewer changes in case of protocol change later.
athas
reviewed
Apr 28, 2026
| -- Save to file helper functions | ||
| propToFile :: FilePath -> [TestResult] -> IO () | ||
| propToFile testFile results = do | ||
| let fileName = testFile <> ".propResult" |
Member
There was a problem hiding this comment.
Style nit: Futhark convention is for all files to use snake_case, except where other languages (such as Haskell) demands otherwise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is not the finished work but it's a start. We have made a basic integration of a separate property based testing tool into futhark test. The basic use inside of a futhark program is:
The attribute can take a size, seed, and pretty printer as well. So a fully custom attribute could be:
#[prop(gen(bar), shrink(baz), pprint(fooz), size(10), seed(12347)].The generator, property, and shrinker must be entry points or it will not work. Every attribute is assigned to a property and is specific to each property, they do not share anything. For the shrinker you can write auto to use our auto shrinking strategy or omit it completely which will not shrink. If you have created your own shrinker use its function name instead.
Note that currently it is not fully working perfectly together with the regular test. If the regular test fails on any test it will overwrite some of our printed outputs.