Mirgecom breakage: upstream pkg pytools+pyvisfile

[MTCam]

Currently, fresh installs of mirgecom are failing:
(https://github.com/illinois-ceesd/mirgecom/actions/runs/9883394155/job/27298055024)

To get mirgecom installs working, we need to revert to these upstream packages@versions:
github.com/inducer/pyvisfile@09c01f2
github.com/inducer/pytools@444c4b6

CC: @tulioricci, @inducer, @matthiasdiener

[inducer]

Updating loopy should work. This change specifically.

[MTCam]

Updating loopy should work. This change specifically.

Thanks! Using this change in loopy obviates reverting pytools, but a fresh install still needs:
github.com/inducer/pyvisfile@09c01f2

Else, this error occurs:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/site-packages/mpi4py/run.py", line 214, in main
    run_command_line(args)
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/site-packages/mpi4py/run.py", line 46, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "wave.py", line 290, in <module>
    main_func(actx_class, use_logmgr=args.log, casename=casename, mpi=args.mpi)
  File "/Users/mtcampbe/CEESD/devel/pristine3/mirgecom/mirgecom/mpi.py", line 138, in wrapped_func
    func(*args, **kwargs)
  File "wave.py", line 227, in main
    vis.write_parallel_vtk_file(
  File "/Users/mtcampbe/CEESD/devel/pristine3/meshmode/meshmode/discretization/visualization.py", line 688, in write_parallel_vtk_file
    self.write_vtk_file(
  File "/Users/mtcampbe/CEESD/devel/pristine3/meshmode/meshmode/discretization/visualization.py", line 745, in write_vtk_file
    from pyvisfile.vtk import (
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1138, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1078, in _find_spec
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/site-packages/_pyvisfile_editable_loader.py", line 310, in find_spec
    tree = self._rebuild()
           ^^^^^^^^^^^^^^^
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/site-packages/_pyvisfile_editable_loader.py", line 343, in _rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=subprocess.DEVNULL)
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/mtcampbe/CEESD/devel/pristine3/miniforge3/envs/pristine3/lib/python3.11/subprocess.py", line 1953, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/5h/cdhm_hv9657bbbxc1s78pd4m0000gn/T/pip-build-env-lpvxekij/overlay/bin/ninja'

[inducer]

inducer/pyvisfile@09c01f2 seems completely unrelated to the error message you show though.

What might help is to avoid installing pyvisfile editable. I think what it's trying to do is build the silo dependency upon import, without ever installing all the pieces (ninja etc.) needed for the build.

cc @alexfikl

[MTCam]

inducer/pyvisfile@09c01f2 seems completely unrelated to the error message you show though.

What might help is to avoid installing pyvisfile editable. I think what it's trying to do is build the silo dependency upon import, without ever installing all the pieces (ninja etc.) needed for the build.

cc @alexfikl

Right, thanks! Just doing pip install . with pyvisfile@main worked fine. We'll need to update mirgecom/requirements.txt and update our loopy to fix our installs.

[MTCam]

inducer/pyvisfile@09c01f2 seems completely unrelated to the error message you show though.
What might help is to avoid installing pyvisfile editable. I think what it's trying to do is build the silo dependency upon import, without ever installing all the pieces (ninja etc.) needed for the build.
cc @alexfikl

Right, thanks! Just doing pip install . with pyvisfile@main worked fine. We'll need to update mirgecom/requirements.txt and update our loopy to fix our installs.

Just to keep this issue updated, fresh installs using non-editable install of pyvisfile@main seem to work fine when I install it using emirge locally. It still fails to make a working installation in CI, however:

(https://github.com/illinois-ceesd/mirgecom/actions/runs/9899463425/job/27348408256?pr=926)

Still trying to track down how this happens.

[alexfikl]

What might help is to avoid installing pyvisfile editable. I think what it's trying to do is build the silo dependency upon import, without ever installing all the pieces (ninja etc.) needed for the build.

cc @alexfikl

Just to clarify this a bit: pyvisfile uses meson-python now even if it doesn't try to build the silo wrapper (which is actually disabled by default as before).

I'm not sure why installing it non-editable would work, since it should still go through meson and ninja? Does pip automatically install the dependencies in build-system.requires then? If yes, you should probably be able to keep the editable install by just adding meson-python and ninja to your requirements.txt.

[MTCam]

What might help is to avoid installing pyvisfile editable. I think what it's trying to do is build the silo dependency upon import, without ever installing all the pieces (ninja etc.) needed for the build.
cc @alexfikl

Just to clarify this a bit: pyvisfile uses meson-python now even if it doesn't try to build the silo wrapper (which is actually disabled by default as before).

I'm not sure why installing it non-editable would work, since it should still go through meson and ninja? Does pip automatically install the dependencies in build-system.requires then? If yes, you should probably be able to keep the editable install by just adding meson-python and ninja to your requirements.txt.

Installing those packages didn't seem to help. I still get the Popen error I posted above when pyvisfile is installed as an --editable package. Re-installing as non-editable resolves/works-around that problem.

[alexfikl]

Installing those packages didn't seem to help. I still get the Popen error I posted above when pyvisfile is installed as an --editable package. Re-installing as non-editable resolves/works-around that problem.

Oops, yeah, might also need to add a --no-build-isolation --editable .... There's some explanation of how these things work together in the docs
https://meson-python.readthedocs.io/en/latest/how-to-guides/editable-installs.html#editable-installs

[MTCam]

Installing those packages didn't seem to help. I still get the Popen error I posted above when pyvisfile is installed as an --editable package. Re-installing as non-editable resolves/works-around that problem.

Oops, yeah, might also need to add a --no-build-isolation --editable .... There's some explanation of how these things work together in the docs meson-python.readthedocs.io/en/latest/how-to-guides/editable-installs.html

I reckon the pip we are installing is behind the curve. We don't have the --no-build-isolation option available. Is the pip version something we can easily update @matthiasdiener ?

[matthiasdiener]

Installing those packages didn't seem to help. I still get the Popen error I posted above when pyvisfile is installed as an --editable package. Re-installing as non-editable resolves/works-around that problem.

Oops, yeah, might also need to add a --no-build-isolation --editable .... There's some explanation of how these things work together in the docs meson-python.readthedocs.io/en/latest/how-to-guides/editable-installs.html

I reckon the pip we are installing is behind the curve. We don't have the --no-build-isolation option available. Is the pip version something we can easily update @matthiasdiener ?

--no-build-isolation should work, but it is not supported within a requirements.txt file:
https://pip.pypa.io/en/stable/reference/requirements-file-format/

[alexfikl]

--no-build-isolation should work, but it is not supported within a requirements.txt file: pip.pypa.io/en/stable/reference/requirements-file-format

Ah, I didn't know that was inside of a requirements.txt file. Then yeah, installing it non-editable should be the best choice. Sorry for the noise!