Automate regression tests

[poslegm]

There is a very helpful ability to perform regression testing of formatting changes:

1. Clone https://github.com/olafurpg/scala-repos (a large bunch of real world Scala code)
2. Format it with scalafmt built from master (or last release)
3. Format it with scalafmt built from your branch
4. push scala-repos changes
5. Add link to commit difference to your PR

It would be good to have GitHub bot (or something like it) that can react on "regress, please" comment on pull request and run job with regression testing. And the add comment with difference link to PR.

Script draft:

#!/bin/bash

LAST_RELEASE=v2.4.2
REPOS=/tmp/regress-tests
BRANCH=alignment-fix
CONFIG=/tmp/.scalafmt.conf

build_scalafmt () {
  cd $REPOS/scalafmt
  git pull
  git checkout $1
  sbt cli/assembly
}

format_scala_repos () {
  cd $REPOS/scala-repos
  git checkout master
  git pull
  git branch $BRANCH
  git checkout $BRANCH 
  java -jar $REPOS/scalafmt/scalafmt-cli/target/scala-2.13/scalafmt.jar -c $CONFIG $REPOS/scala-repos/
  git commit -a -m "formatted with ${1}"
  git push -u origin HEAD
}

git clone git@github.com:poslegm/scalafmt.git $REPOS/scalafmt
git clone git@github.com:poslegm/scala-repos.git $REPOS/scala-repos

build_scalafmt $LAST_RELEASE
format_scala_repos $LAST_RELEASE

build_scalafmt $BRANCH
format_scala_repos $BRANCH

Related: #154

[kitbellew]

@poslegm excellent idea, would make life easier. i have a couple of points:

• suggestion: compare with the common ancestor of the proposed branch and the master, otherwise you will include formatting changes that weren't caused by the pull request and likely had been reviewed and approved previously
• challenge: what .scalafmt configuration to use for a given pull request? does it modify rewrite rules, or vertical multiline, or some other non-default or simply uncommon fork in the execution path? perhaps we can have a dozen of precanned configurations and use each.
• another challenge: can we also try to build the reformatted code? some of the recent bugs filed after 2.4.0 were about a rewrite breaking the code, but a newline in the wrong spot could do that, too.

[poslegm]

suggestion: compare with the common ancestor of the proposed branch and the master, otherwise you will include formatting changes that weren't caused by the pull request and likely had been reviewed and approved previously

Yes, you are right. It will be more correct way.

challenge: what .scalafmt configuration to use for a given pull request?

It will not be hard to specify .scalafmt.conf content in bot run command:

regress, please

maxColumn = 120

another challenge: can we also try to build the reformatted code?

In ideal world yes. But then we'll have to support scala-repos in buildable state. It's complicated task. And regress job duration will increase a lot with all projects build.