output quality depends on $PWD

[samuel-gelineau-at-well-dot]

in multi-package projects, it is common for each package to be placed in a separate sub-directory. the quality of the output depends on whether the tests are run from the root of the project or from the sub-directory of the package which contains the tests:

$ cd ~PROJECT_ROOT
$ cabal run toy-exe
━━━ Toy ━━━
  ✗ prop_failing failed at Main.hs:14:3
    after 1 test.
    shrink path: 1:

    forAll0 =
      0

    forAll1 =
      0

    This failure can be reproduced by running:
    > recheckAt (Seed 1302999818739436304 15345910533038617549) "1:" prop_failing

  ✗ 1 failed.

vs the much better looking:

$ cd $PROJECT_ROOT/my-package
$ cabal run toy-exe
━━━ Toy ━━━
  ✗ prop_failing failed at Main.hs:14:3
    after 1 test.
    shrink path: 1:

       ┏━━ Main.hs ━━━
     8 ┃ prop_failing :: Property
     9 ┃ prop_failing = property $ do
    10 ┃   x <- forAll $ Gen.int (Range.linear 0 10)
       ┃   │ 0
    11 ┃   y <- forAll $ Gen.int (Range.linear 0 10)
       ┃   │ 0
    12 ┃   assert (x == x)
    13 ┃   assert (y == y)
    14 ┃   assert (x < y)
       ┃   ^^^^^^^^^^^^^^
    15 ┃   assert (x <= x)
    16 ┃   assert (y <= y)

    This failure can be reproduced by running:
    > recheckAt (Seed 7577190907837601680 195968907661934709) "1:" prop_failing

  ✗ 1 failed.

For easy reproducibility, here is the file structure I have used to produce the output above. The only thing which matters is that my toy.cabal file is in $PROJECT_ROOT/my-package instead of $PROJECT_ROOT.

$ tree $PROJECT_ROOT
$PROJECT_ROOT
├──
├── my-package
│   ├── Main.hs
│   └── toy.cabal
└── cabal.project

$ cat $PROJECT_ROOT/cabal.project
packages: my-package

$ cat $PROJECT_ROOT/my-package/toy.cabal
cabal-version:      3.0
name:               toy
version:            0.1.0.0
build-type:         Simple

executable toy-exe
    main-is:          Main.hs
    build-depends:    base
                    , hedgehog == 1.4
    hs-source-dirs:   .
    default-language: Haskell2010

$ cat $PROJECT_ROOT/my-package/Main.hs
{-# LANGUAGE OverloadedStrings #-}
module Main where

import Hedgehog
import qualified Hedgehog.Gen as Gen
import qualified Hedgehog.Range as Range

prop_failing :: Property
prop_failing = property $ do
  x <- forAll $ Gen.int (Range.linear 0 10)
  y <- forAll $ Gen.int (Range.linear 0 10)
  assert (x == x)
  assert (y == y)
  assert (x < y)
  assert (x <= x)
  assert (y <= y)

main :: IO Bool
main
  = checkParallel
  $ Group "Toy"
    [ ("prop_failing", prop_failing)
    ]

[moodmosaic]

Thanks for the detailed report ❤️

I think that with Cabal, running tests from a sub-directory with its own .cabal file might affect how paths are resolved.

This could link error outputs directly to the source files in the sub-directory, potentially explaining the detailed error reporting when tests are run from $PROJECT_ROOT/my-package.

However, I couldn’t find specific documentation confirming this exact behavior, so it might very well be possible to fix this inside hedgehog itself.

[samuel-gelineau-at-well-dot]

it might very well be possible to fix this inside hedgehog itself.

I am guessing that hedgehog records the relative path of the source code at compile time, and thus whether this relative path resolves to a file or not depends on the $PWD. recording the absolute path instead of the relative path should fix the problem.

[moodmosaic]

Sounds reasonable and pull requests welcome 💯 👍

[sol]

recording the absolute path instead of the relative path should fix the problem.

I think this would require Template Haskell, I'm not aware of a way to achieve this with the HasCallStack-machinery.

Maybe just open a Cabal issue instead. When cabal test executes the test executable it could easily set the CWD to the package root. Better address it once at the root than requiring every test library to hack its way around it.

[moodmosaic]

Agreed with @sol, if it's possible to fix this upstream it'll be better so that the whole community may benefit for this.

[lrworth]

I occasionally place a property test generator function (like myProperty :: A -> Property) into a library, and call it from multiple application test suites. To see the source code context it is necessary to run the application tests from the library package root. To serve this kind of setup it would be nicer if the context source code were actually stored in the build object rather than loaded from the filesystem at runtime. This would also enable context when only a binary of the library is available, and would protect against odd situations where you've changed the source code between building the test and running it.

[moodmosaic]

Thank you, @lrworth. I think a safer move for now might be to detect when we can’t load context and emit a clear fallback message. Embedding source context at build time is probably the right long-term direction, though there are some caveats beyond just binary size—portability concerns and the possibility that users may not want source embedded come to mind.

[lrworth]

Fair enough. Another option is to allow individual package source paths to be supplied as command line arguments — that would solve OP's problem and mine at the same time. If this issue is still open in about 3 weeks I can look into doing that.

[sol]

that would solve OP's problem

As for the OP's issue, if you create a Cabal test-suite and use e.g. cabal test all then cabal runs each test suite in the root directory of the corresponding package and everything works as expected.

If you insist on using cabal run then it is your responsibility to change the working directory as needed.

From my perspective this works as intended.

No changes to hedgehog needed.

and mine at the same time

@lrworth As I mentioned above, I don't think OP's issue needs any changes to hedgehog. So I think any solution to your issue needs to stand on its own merits.

I'm still puzzled:

1. Do you really need to see source snippets from your library?
2. You might want to use withFrozenCallStack when defining library primitives, so that you get source snippets for the use site instead.

it would be nicer if the context source code were actually stored in the build object

I think this would require Template Haskell. Personally, I'm not eager to see this happening.

[lrworth]

1. Do you really need to see source snippets from your library?

It is useful when the library contains the properties being tested. For example we might have a library module defining monoid laws:

checkMonoidAssoc :: Monoid m => Gen m -> Property
checkMonoidAssoc gen = do
  x <- forAll gen
  y <- forAll gen
  z <- forAll gen
  (x <> y) <> z === x <> (y <> z)

To understand why such a test failed from the call-site in a different package, the developer needs to find the library code and determine how forAll0, forAll1, forAll2 in the output match up with the generated variables. If withFrozenCallStack would help with this, I don't understand how.

[sol]

Another option is to allow individual package source paths to be supplied as command line arguments

Just to be clear, as a third-party, I'm not opposed to this, provided the resulting code is reasonably clean.

(Not something I personally would lobby for or need, but if @moodmosaic is willing to accept it, not something I would lose sleep over neither.)

Another thing you could explore is implement a TH reversion of forAll that captures the call site at compile time.

I haven't looked at any code right now, but this will likely need some changes to hedgehog.

Hedgehog would need to provide something like:

forAllWithSourceSnippet :: (Monad m, Show a, HasCallStack) => SourceSnippet ->Gen a -> PropertyT m a

(and variations of this)

A third-party library could then use this to implement a TH version of forAll.

You could then use $forAll from that third-party library to bake in source snippets.

Not sure how this will turn out in terms of what changes are needed to hedgehog, but maybe worth a try.

[moodmosaic]

Thank you, @sol. I still lean towards the idea discussed in the past, but I haven't had the chance to explore it further. See: #517 (comment).