Skip to content

Add a new @PendingFeatureIf annotation / extension to allow features to be conditionally pending#988

Merged
leonard84 merged 4 commits intospockframework:masterfrom
ryangardner:add-pendingfeatureif
Feb 6, 2020
Merged

Add a new @PendingFeatureIf annotation / extension to allow features to be conditionally pending#988
leonard84 merged 4 commits intospockframework:masterfrom
ryangardner:add-pendingfeatureif

Conversation

@ryangardner
Copy link
Copy Markdown
Contributor

@ryangardner ryangardner commented Apr 9, 2019

In our specific case we have some geb-based tests that target different deployed environments and occasionally it is convenient to have tests marked as being conditionally Pending.

I could see other places this might be useful - (features that are implemented on one OS but not another, etc)


This change is Reviewable

- Allow features to be conditionally 'pending' similar to @IgnoreIf
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 9, 2019

Codecov Report

Merging #988 into master will not change coverage.
The diff coverage is 84.71%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #988   +/-   ##
=========================================
  Coverage     74.02%   74.02%           
  Complexity     3437     3437           
=========================================
  Files           383      383           
  Lines         10630    10630           
  Branches       1297     1297           
=========================================
  Hits           7869     7869           
  Misses         2299     2299           
  Partials        462      462
Impacted Files Coverage Δ Complexity Δ
...g/spockframework/junit4/AbstractRuleExtension.java 86.66% <ø> (ø) 10 <0> (ø) ⬇️
.../java/org/spockframework/runtime/SpockRuntime.java 71.11% <ø> (ø) 20 <0> (ø) ⬇️
...n/java/spock/util/concurrent/BlockingVariable.java 93.75% <ø> (ø) 6 <0> (ø) ⬇️
...kframework/spring/mock/SpockMockPostprocessor.java 85.03% <ø> (ø) 41 <0> (ø) ⬇️
...in/java/spock/util/concurrent/AsyncConditions.java 93.54% <ø> (ø) 9 <0> (ø) ⬇️
...pock-core/src/main/java/spock/mock/MockingApi.java 2.94% <ø> (ø) 2 <0> (ø) ⬇️
spock-core/src/main/java/spock/lang/Retry.java 100% <ø> (ø) 0 <0> (ø) ⬇️
.../spockframework/report/log/ReportLogExtension.java 100% <ø> (ø) 1 <0> (ø) ⬇️
...spockframework/runtime/SpockComparisonFailure.java 83.33% <ø> (ø) 3 <0> (ø) ⬇️
...spockframework/junit4/AbstractRuleInterceptor.java 85.71% <ø> (ø) 4 <0> (ø) ⬇️
... and 91 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0b8457...5acf243. Read the comment docs.

@leonard84
Copy link
Copy Markdown
Member

Hi, thanks for your contribution. We are currently working on spock-2.0, until that is merged to master we won't merge PRs.

@leonard84 leonard84 added this to the 2.0 milestone Apr 15, 2019
@leonard84
Copy link
Copy Markdown
Member

Hm, why don't you want to use @IgnoreIf/@Requires for this use-case?

@ryangardner
Copy link
Copy Markdown
Contributor Author

The behavior of PendingFeature - where it expects the test to fail (because the feature isn't implemented) and treats a failed test as passing is different from Ignore - where it doesn't run the test - we actually do want to run the test and be notified if it suddenly passes.

The scenario where we had conditional pending features is for a UI automation suite that hits multiple environments. (could be more than two, but for simplicity sake, lets assume a dev and prod environment)

A new feature is deployed but only active on the dev environment. We have some UI tests that actively run against that new feature - if we mark them as IgnoreIf(environment=prod) - the test suite will pass - with the tests running against our dev environment and not running in prod.

Some time passes - and the feature is released to production. The test suite still passes - with the tests running when we execute it against our dev environment but the tests are still ignored when running against the production environment. If some developer checks in code that breaks that feature - but only breaks it in prod - our regression suite wont detect it because the test is ignored.

In a perfect world, we'd remember to remove that @IgnoreIf annotation promptly after the feature was released. What's nice about the @PendingFeature annotation (and in this PR, the @PendingFeatureIf annotation) is that it gives you a reminder that this thing you were ignoring previously is now passing and you should update your tests to reflect that. It's a pretty harsh reminder (it fails your tests / build) - but it's an effective one.

The same thing that might motivate you to sometimes use @PendingFeature and to sometimes use @Ignore exist when you need to do those conditional activations - there is the @IgnoreIf which is great when you want to conditionally ignore a test, but the lack of a @PendingFeatureIf makes it hard if you want to conditionally mark a feature as pending.

If there's a way to combine @IgnoreIf and @Requires together and get the same behavior as conditionally applying a @PendingFeature annotation, we could use that instead

Comment thread docs/extensions.adoc Outdated

[source,groovy]
----
@PendingFeatureIf({ System.getProperty("targetEnvironment").startsWith("production") })
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at PreconditionContext we have a shorthand for that sys, and I would just use == 'prod'

And just merge those two blocks

import java.lang.annotation.Target;

/**
* If a passed in closure evaluates to true, this will be treated as a {@link PendingFeature}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use similar wording to IgnoreIf -> ... if the given condition holds.

*
* Otherwise it will be treated like a normal feature.
*
* The closure follows the same rules as those used by {@link IgnoreIf}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just copy the block with the information regarding the PreconditionContext from @Requires

@leonard84
Copy link
Copy Markdown
Member

@ryangardner do you have time to fix the comments soon, I'd like to use the annotation in #1075, but if you need more time then we'll merge that pr without it.

@szpak
Copy link
Copy Markdown
Member

szpak commented Feb 4, 2020

@ryangardner We want to merge #1075 this week. If you are unable to address the change requests made by Leonard by that time, we can understand that. Just let us know :).

@leonard84 leonard84 merged commit ca2963a into spockframework:master Feb 6, 2020
@szpak
Copy link
Copy Markdown
Member

szpak commented Feb 8, 2020

@ryangardner I made required changes and it is now merged. Thanks for your PR.

@ryangardner
Copy link
Copy Markdown
Contributor Author

oh, thanks! I just saw the most recent message but the notifications for this issue got swallowed up in my inbox for some reason

@ryangardner ryangardner deleted the add-pendingfeatureif branch February 9, 2020 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants