WIP: Use pep517 to extract metadata.

[tomprince]

I'm leaving this branch with my work-in-progress for using pep517 (both the package and the PEP) to extract package metadata, instead of a patch setuptools. In particular, this should work with more packages.

An incomplete list of things that still need to be done:

• Properly parse requires-dist (since this includes the equivalent of the old install_requires and extra_requires). This should perhaps be done in the python, before writing out the metadata.
  • This might not be necessary, as

    mach-nix/mach_nix/data/providers.py

    Line 473 in c914064

    requirements[t] = list(filter_reqs_by_eval_marker(reqs, self.context))

    process markers in install_requires.
  • On the other hand this code (and the copy of that code in mach_nix/nix/extract-metadata.nix) doesn't handle markers (in particular, extra markers) which is a bug (related to markers not interpreted on automatically extracted requirements #190).
• Provide code paths for use by pypi-deps-db.
• Provide code paths for use by flake's extract-reqs.
• Remove obsolete code.
  • Most of the code in lib/extractor/default.nix is unused after this PR.
• Decide on whether to drop python2 support. If so, document that change; if not, provide fallback code to handle that case.
  • I think it might be possible to still support python2 by using python3 for the base-env environment in mach_nix/nix/extract-metadata.nix but python2 for the metadata-env and build-env.
• ....

Note: I currently have no plans to continue this work in the future.

[DavHau]

Thanks for sharing your work

[chaoflow]

@tomprince @DavHau Thank you very much for your work! I'd like to help to get this into master. Could you maybe sketch what needs to be done to finish this?

[DavHau]

@chaoflow as you probably know, mach-nix sources information about python dependencies from pypi-deps-db. One problem is that some information in pypi-deps-db is incorrect or incomplete because of missing pep517 support.
The crawler which regularly updates the pypi-deps-db utilizes logic from mach-nix' lib/extractor/to extract dependency information from package source tarballs. This logic needs to be updated in order to support pep-517.

So one way to test what works and what not could be to clone pypi-deps-db (shallowly ;)), then update the mach-nix flake input to this PR and execute the flake app update-sdist.

It's probably better to first test extraction on a single package. For this you can use mach-nix' extract-deps flake app.

[tomprince]

There is also a draft PR on pypi-deps-db which uses the code here: DavHau/pypi-deps-db#6