Keep sdist in sync with git (include all files in source build, including docs, tests and examples)

[chanicpanic]

This PR adopts setuptools-scm's file finder functionality to automatically include files tracked by git in the sdist. The version-extraction functionality is not enabled.

Resolves #1559.

Result of check-sdist without nearley submodule before:

SDist does not match git

SDist only:


Git only:
  .gitignore
  .gitmodules
  CHANGELOG.md
  docs/_static/comparison_memory.png
  docs/_static/comparison_runtime.png
  docs/_static/lark_cheatsheet.pdf
  docs/_static/sppf/sppf.html
  docs/_static/sppf/sppf_111.svg
  docs/_static/sppf/sppf_abcd.svg
  docs/_static/sppf/sppf_abcd_noint.svg
  docs/_static/sppf/sppf_cycle.svg
  docs/ide/app.html
  docs/ide/app.js
  docs/ide/app/app.py
  docs/ide/app/core.py
  docs/ide/app/examples.py
  docs/ide/app/ext.py
  docs/ide/app/files.json
  docs/ide/app/html5.py
  docs/ide/app/ignite.py
  docs/ide/app/utils.py
  docs/ide/is-loading.gif
  docs/ide/lark-logo.png
  examples/README.rst
  examples/advanced/README.rst
  examples/advanced/_json_parser.py
  examples/advanced/conf_earley.py
  examples/advanced/conf_lalr.py
  examples/advanced/create_ast.py
  examples/advanced/custom_lexer.py
  examples/advanced/dynamic_complete.py
  examples/advanced/error_handling.py
  examples/advanced/error_reporting_earley.py
  examples/advanced/error_reporting_lalr.py
  examples/advanced/prioritizer.py
  examples/advanced/py3to2.py
  examples/advanced/python2.lark
  examples/advanced/python_parser.py
  examples/advanced/qscintilla_json.py
  examples/advanced/reconstruct_json.py
  examples/advanced/reconstruct_python.py
  examples/advanced/template_lark.lark
  examples/advanced/templates.py
  examples/advanced/tree_forest_transformer.py
  examples/composition/README.rst
  examples/composition/combined_csv_and_json.txt
  examples/composition/csv.lark
  examples/composition/eval_csv.py
  examples/composition/eval_json.py
  examples/composition/json.lark
  examples/composition/main.py
  examples/composition/storage.lark
  examples/grammars/README.rst
  examples/grammars/verilog.lark
  examples/relative-imports/multiple2.lark
  examples/relative-imports/multiple3.lark
  examples/relative-imports/multiples.lark
  examples/relative-imports/multiples.py
  examples/standalone/README.rst
  examples/standalone/create_standalone.sh
  examples/standalone/json.lark
  examples/standalone/json_parser_main.py
  examples/tests/negative_priority.lark
  examples/tests/no_newline_at_end.lark
  readthedocs.yml
  test-requirements.txt
  tests/test_nearley/nearley
  tox.ini

After:

SDist does not match git

SDist only:


Git only:
  tests/test_nearley/nearley

[erezsh]

With this change, the source dist will also include the following files:

.pre-commit-config.yaml
.gitmodules
.gitignore
docs/_static

I'm not sure that is the desired behavior.

It also starts including docs/ide, which tbh probably should have been removed by now?

[chanicpanic]

Right, git-related configuration files do not need to be in the sdist. I agree it makes sense to exclude the ide especially if it's not really supported.

docs/_static/*.png are referenced by README.md, and docs/_static/lark_cheetsheet.pdf is referenced by README.md and docs/index.rst. I left those files in for now, but if you think all of docs/_static should be excluded that can be done.

[erezsh]

In that vain, why exclude docs/_static/sppf ?

[erezsh]

Either way, looks good to me, and seems like generally the Python community agrees with the "include everything" approach.

@MegaIng Do you have an opinion?

[MegaIng]

How much larger is the sdist with this change? Also, I assume that none of the new files end up in the resulting wheel?

Generally I weakly agree that making the sdist more complete is a good idea.

[erezsh]

Good question. Seems like it grows the zip from 260 KB to 365 KB. Unzipped, from 0.96 MB to 1.3 MB.

The wheel is 113 KB, so I assume it's not affected.

(though for some reason, I can't reproduce it. Now if I try to build the existing master from scratch, it collects all the files, as if I was in the new branch! very odd)

edit: confirmed with a fresh build. this PR brings the zip from 256 KB to 364 KB. The wheel remains the same.

[chanicpanic]

In that vain, why exclude docs/_static/sppf ?

It wasn't referenced by any other files, but since we have roughly settled on completeness, I have included it again. As such, someone with the sdist should be able to produce the full set of documentation that is on Read the Docs.

My measurements of the sdist with the latest changes are:

Zipped: 388 KB
Unzipped: 1.7 MB

The wheel is not affected by this PR.

[erezsh]

Thanks @chanicpanic !