How might we automate CI against new releases of Rails?

[stevepolitodesign]

One of the most difficult things about maintaining Suspenders is ensuring it works with the latest version of Rails. Even though we have good test coverage on the last release, that did not matter because the dummy app was frozen in time.

One advantage we have with the latest release is that CI builds and tests a Rails application generated from Suspenders. The issue is that we only run CI anytime there's a PR.

[stevepolitodesign]

I think we'd need to explore something like suspenders new <app_name> --edge, since there's a rails new <app_name> --edge.

We'd only really need to support this for CI and development, so we could use a binstub + environment variable like we did in #1287

However, we'd still need to have this run on a schedule in CI.

[nickcharlton]

I've been working on two things to try and solve a similar version of this problem on Administrate: thoughtbot/administrate#2830 is the current PR which I've been gradually working on. It uses a gem I've been playing around with called sack_race.

The idea behind sack_race and that PR is to run shell commands as part of workflows (most of which are long running, which is why it's a complex problem and was worth building something for it) which are able to build projects from scratch. We can reasonably trust that the GitHub Actions environment is clean, so we can install Rails and build everything up command by command. The long-term goal is to run a set of common tests against this fresh environment (system tests, but more specific).

It's then easy enough to lean on matrix builds to test against lots of different versions (including current HEAD, for instance) and there's run_on triggers for time using the cron syntax.

It is very time consuming to develop this though, the feedback cycle can be painfully slow whilst you wait for a lot of commands to run!

[stevepolitodesign]

As an aside, I also wonder if using rails-diff (thank you @MatheusRich!) could help with local development as well as the issue at hand. We could just compare the "expected" diff with the "actual" diff.