How to implement a test tag that is disabled by default?

[dweiss]

So this is a bit of an api puzzler for me - I'd like to have several custom annotation test groups, say @Weekly and @Nightly which are disabled by default. I can meta-annotate them with the corresponding @Tag but this then relies on the build system's configuration to exclude those tags. It's not exactly what I want - I'd like any launcher (IDEs, gradle, maven) to see those tests as disabled by default (unless their tags are explicitly included), without any additional configuration.

Is this even possible?

For now I've opted to implement the above using a layered ExecutionCondition that looks at a system property, so something like this:

@Tag("nightly")
@TagState(enabled = false, sysProperty = "tests.nightly")
public @interface Nightly {}

where TagState is:

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD})
@ExtendWith(TagState.Condition.class)
public @interface TagState {
  boolean enabled();
  String sysProperty();

  final class Condition implements ExecutionCondition {
    @Override
    public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext context) {
      var opt = context.getElement();
      if (opt.isPresent()) {
        var element = opt.get();
        var stateAnn_ = AnnotationSupport.findAnnotation(element, TagState.class);
        if (stateAnn_.isPresent()) {
          var stateAnn = stateAnn_.get();
          var tagAnns = AnnotationSupport.findRepeatableAnnotations(element, Tag.class);
          if (tagAnns.size() != 1) {
            throw new RuntimeException("Expected exactly one @Tag meta-annotation on: " + element);
          }

          String tag = tagAnns.getFirst().value();
          var enabled =
              context
                  .getConfigurationParameter(stateAnn.sysProperty(), Boolean::parseBoolean)
                  .orElse(stateAnn.enabled());
          return enabled
              ? ConditionEvaluationResult.enabled("@" + tag + " enabled")
              : ConditionEvaluationResult.disabled(
                  "@" + tag + " disabled; use -D" + stateAnn.sysProperty() + "=true to enable.");
        }
      }

      return ConditionEvaluationResult.enabled("no @TestGroupTag found");
    }
  }
}

I wonder if I'm missing any other implementation routes. Or maybe something obvious that just eluded me. Thanks!

[mpkorstanje]

One alternative you could look at would be registering a PostDiscoveryFilter as a service.

It has the same semantics as filtering by tags. The tests are removed from the test plan unlike the execution edition where tests are kept but then skipped.

Unfortunately filters don't have access to the original tag expressions so you'd still have to check an environment variable.