Add skylab-1.0.0-public template to avoid license issues, complete documentation on chaining spack environments (#233)

* Add instructions for building openmpi@4.1.3 with gcc@10.1.0 on Discover
* Add configs/templates/skylab-1.0.0-public/spack.yaml
* Complete documentation for spack-stack-1.0.0 release
* [skip ci] Update version numbers of solo, r2d2, ewok
* [skip-ci] Update .gitmodules and submodule pointer for spack

Author: climbfuji

.gitmodules

@@ -3,7 +3,7 @@
   #url = https://github.com/spack/spack
   #branch = develop
   url = https://github.com/NOAA-EMC/spack
-  branch = release/jcsda_emc_release_v1
+  branch = spack-stack-1.0.0
 [submodule "doc/CMakeModules"]
   path = doc/CMakeModules
   url = https://github.com/noaa-emc/cmakemodules

configs/templates/skylab-1.0.0-public/spack.yaml

@@ -0,0 +1,84 @@
+spack:
+  concretizer:
+    unify: when_possible
+
+  view: false
+
+  #packages:
+  #  fiat:
+  #    version: [1.0.0]
+  #  ectrans:
+  #    version: [1.0.0]
+
+  include: []
+
+  specs:
+
+    # Virtual environment packages
+    - base-env@1.0.0
+    - jedi-base-env@1.0.0 ~fftw
+    - jedi-ewok-env@1.0.0
+    - jedi-fv3-env@1.0.0
+    - jedi-mpas-env@1.0.0
+    - jedi-ufs-env@1.0.0
+    #- jedi-um-env@1.0.0
+    #- soca-env@1.0.0
+    - ufs-weather-model-env@1.0.0
+
+    # Individual packages
+    - bacio@2.4.1
+    - bison@3.8.2
+    - bufr@11.7.0
+    - crtm@2.3.0
+    - ecbuild@3.6.5
+    - eccodes@2.25.0
+    - ecflow@5
+    - eckit@1.19.0
+    - ecmwf-atlas@0.29.0 ~trans ~fftw
+    # DH* fake version number
+    #- ectrans@1.0.0
+    - eigen@3.4.0
+    - esmf@8.3.0b09
+    # DH* fake version number
+    - ewok@0.0.1
+    - fckit@0.9.5
+    # DH* fake version number
+    #- fiat@1.0.0
+    - flex@2.6.4
+    - fms@2022.01
+    # DH* fake version number
+    - fms@release-jcsda
+    - g2@3.4.5
+    - g2tmpl@1.10.0
+    #- gdal@3.4.3
+    #- geos@3.9.1
+    - gftl-shared@1.5.0
+    - hdf5@1.12.1
+    - hdf@4.2.15
+    - ip@3.3.3
+    - jasper@2.0.32
+    - jedi-cmake@1.3.0
+    - libpng@1.6.37
+    - mapl@2.12.3
+    - nccmp@1.9.0.1
+    - netcdf-c@4.8.1
+    - netcdf-cxx4@4.3.1
+    - netcdf-fortran@4.5.4
+    - nlohmann-json-schema-validator@2.1.0
+    - nlohmann-json@3.10.5
+    - parallel-netcdf@1.12.2
+    - parallelio@2.5.4
+    - py-numpy@1.22.3
+    - py-pandas@1.4.0
+    - py-scipy@1.8.0
+    - py-shapely@1.8.0
+    # DH* fake version number
+    - r2d2@0.0.1
+    # DH* fake version number
+    #- shumlib@macos_clang_linux_intel_port
+    - solo@1.0.0
+    - sp@2.3.3
+    - udunits@2.2.28
+    - w3nco@2.4.1
+    - yafyaml@0.5.1
+    - zlib@1.2.12

configs/templates/skylab-1.0.0/spack.yaml

@@ -40,7 +40,7 @@ spack:
     - eigen@3.4.0
     - esmf@8.3.0b09
     # DH* fake version number
-    - ewok@1.0.0
+    - ewok@0.0.1
     - fckit@0.9.5
     # DH* fake version number
     - fiat@1.0.0
@@ -73,10 +73,9 @@ spack:
     - py-scipy@1.8.0
     - py-shapely@1.8.0
     # DH* fake version number
-    - r2d2@1.0.0
+    - r2d2@0.0.1
     # DH* fake version number
     - shumlib@macos_clang_linux_intel_port
-    # DH* fake version number
     - solo@1.0.0
     - sp@2.3.3
     - udunits@2.2.28

doc/source/Environments.rst

@@ -11,7 +11,7 @@ Environments can be constructed in two ways in spack-stack:
 
     - Configure the environment as shown in :numref:`Sections %s <Quickstart>` and :numref:`%s <Platforms>`.
 
-    - Add spack packages (also referred to as ``specs``) to the environment using ``spack add``. These packages can be virtual environments described in :numref:`Section %s <EnvironmentsVirtualEnvironments>` below, or individual packages, e.g. ``esmf`` or ``atlas``. Examples:
+    - Add spack packages (also referred to as ``specs``) to the environment using ``spack add``. These packages can be virtual environments described in :numref:`Section %s <EnvironmentsVirtualPackages>` below, or individual packages, e.g. ``esmf`` or ``atlas``. Examples:
 
         .. code-block:: console
 

doc/source/MaintainersSection.rst

@@ -109,7 +109,7 @@ On Gaea, ``qt`` needs to be installed as a one-off before spack can be used.
 
 qt (qt@5)
    The default ``qt@5`` in ``/usr`` is incomplete and thus insufficient for building ``ecflow``. After loading/unloading the modules as shown below, refer to 
-   :numref:`Section %s <Prerequisites_Qt5>` to install ``qt@5.15.3`` in ``/lustre/f2/pdata/esrl/gsd/spack-stack/qt-5.15.3``.
+   :numref:`Section %s <Prerequisites_Qt5>` to install ``qt@5.15.2`` in ``/lustre/f2/pdata/esrl/gsd/spack-stack/qt-5.15.2``.
 
 .. code-block:: console
 
@@ -127,7 +127,7 @@ miniconda
 
 qt (qt@5)
    The default ``qt@5`` in ``/usr`` is incomplete and thus insufficient for building ``ecflow``. After loading/unloading the modules as shown below, refer to 
-   :numref:`Section %s <Prerequisites_Qt5>` to install ``qt@5.15.3`` in ``/scratch1/NCEPDEV/jcsda/jedipara/spack-stack/qt-5.15.3``.
+   :numref:`Section %s <Prerequisites_Qt5>` to install ``qt@5.15.2`` in ``/scratch1/NCEPDEV/jcsda/jedipara/spack-stack/qt-5.15.2``.
 
 .. code-block:: console
 
@@ -153,17 +153,6 @@ TACC Stampede2
 
 Several packages need to be installed as a one-off before spack can be used.
 
-Intel oneAPI compilers
-   The latest version of the Intel compiler on Stampede2 is 19.1.1, and the default modulefile created by the system administrators ties it to `gcc-9.1.0`. The way the module file has been written is incompatible with spack. We therefore recommend installing the latest Intel oneAPI compiler suite (Intel oneAPI Base and HPC Toolkits). The following instructions install Intel oneAPI 2022.2 in ``/work2/06146/tg854455/stampede2/spack-stack``.
-
-.. code-block:: console
-
-   wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18679/l_HPCKit_p_2022.2.0.191.sh
-   wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18673/l_BaseKit_p_2022.2.0.262.sh
-   # Customize the installations to install in /work2/06146/tg854455/stampede2/spack-stack/intel-oneapi-2022.2
-   sh l_BaseKit_p_2022.2.0.262.sh
-   sh l_HPCKit_p_2022.2.0.191.sh
-
 miniconda
    Follow the instructions in :numref:`Section %s <Prerequisites_Miniconda>` to create a basic ``miniconda`` installation and associated modulefile for working with spack. Don't forget to log off and back on to forget about the conda environment.
 
@@ -180,7 +169,7 @@ git-lfs
    rpm2cpio git-lfs-1.2.1-1.el7.x86_64.rpm | cpio -idmv
    mv usr/* ../
 
-   Create modulefile ``/work2/06146/tg854455/stampede2/spack-stack/modulefiles/git-lfs/1.2.1`` from template ``doc/modulefile_templates/git-lfs`` and update ``GITLFS_PATH`` in this file.
+Create modulefile ``/work2/06146/tg854455/stampede2/spack-stack/modulefiles/git-lfs/1.2.1`` from template ``doc/modulefile_templates/git-lfs`` and update ``GITLFS_PATH`` in this file.
 
 .. _MaintainersSection_S4:
 
@@ -193,7 +182,7 @@ miniconda
 
 qt (qt@5)
    The default ``qt@5`` in ``/usr`` is incomplete and thus insufficient for building ``ecflow``. After loading/unloading the modules as shown below, refer to 
-   :numref:`Section %s <Prerequisites_Qt5>` to install ``qt@5.15.3`` in ``/data/prod/jedi/spack-stack/qt-5.15.3``.
+   :numref:`Section %s <Prerequisites_Qt5>` to install ``qt@5.15.2`` in ``/data/prod/jedi/spack-stack/qt-5.15.2``.
 
 .. code-block:: console
 
@@ -217,9 +206,60 @@ Using spack to test/add packages
 
 The simplest case of adding new packages that are available in spack-stack is described in :numref:`Section %s <QuickstartExtendingEnvironments>`. As mentioned there, it is advised to take a backup of the spack environment (and install directories if outside the spack environment directory tree). It is also possible to chain spack installations, which means creating a test environment that uses installed packages and modulefiles from another (e.g. authoritative) spack environment and build the packages to be tested in isolation.
 
-**WORK IN PROGRESS**
+Chaining spack-stack installations
+----------------------------------
+
+Chaining spack-stack installations is a powerful way to test adding new packages without affecting the existing packages. The idea is to define one or more upstream spack installations that the environment can use as dependencies. One possible way to do this is:
+
+1. Mirror the environment config of the upstream repository, i.e. copy the entire directory without the ``install`` and ``.spack_env`` directories and without `spack.lock`. For example:
+
+.. code-block:: console
+
+   rsync -av --exclude='install' --exclude='.spack-env' --exclude='spack.lock' \
+       envs/jedi-ufs/ \
+       envs/jedi-ufs-chain-test/
 
-More details and a few words of caution can be found in the  `Spack documentation <https://spack.readthedocs.io/en/latest/chain.html?highlight=chaining%20spack%20installations>`_
+2. Edit `envs/jedi-ufs-chain-test/spack.yaml`` and add an upstream configuration entry directly under the ``spack:`` config so that the contents looks like:
+
+.. code-block:: console
+
+   spack:
+     upstreams:
+       spack-instance-1:
+         install_tree: /path/to/spack-stack-1.0.0/envs/jedi-ufs/install
+     concretizer:
+       unify: when_possible
+     ...
+
+3. Activate the environment
+
+4. Install the new packages, for example:
+
+.. code-block:: console
+
+    spack install -v --reuse esmf@8.3.0b09+debug
+
+5. Create modulefiles
+
+.. code-block:: console
+
+    spack module [lmod|tcl] refresh
+
+6. When using ``tcl`` module files, run the ``spack stack setup-meta-modules`` script. This is not needed when using ``lmod`` modulefiles, because the meta modules in ``/path/to/spack-stack-1.0.0/envs/jedi-ufs-chain-test/install/modulefiles/Core`` will be ignored entirely.
+
+To use the chained spack environment, first load the usual modules from the upstream spack environment. Then add the full path to the newly created modules manually, ignoring the meta modules (``.../Core``), for example:
+
+.. code-block:: console
+
+    module use /path/to/spack-stack-1.0.0/envs/jedi-ufs-chain-test/install/modulefiles/openmpi/4.1.3/apple-clang/13.1.6
+
+7. Load the newly created modules. When using `tcl` module files, make sure that conflicting modules are unloaded (`lmod` takes care of this).
+
+.. note::
+   After activating the chained environment, ``spack find`` doesn't show the packages installed in upstream, unfortunately.
+
+.. note::
+   More details and a few words of caution can be found in the  `Spack documentation <https://spack.readthedocs.io/en/latest/chain.html?highlight=chaining%20spack%20installations>`_. Those words of caution need to be taken seriously, especially those referring to not deleting modulefiles and dependencies in the upstream spack environment (if having permissions to do so)!
 
 ----------------------------------------
 Testing/adding packages outside of spack
@@ -233,6 +273,7 @@ Sometimes, users may want to build new versions of packages frequently without u
 Users can build multiple packages outside of spack and install them in a separate install tree, for example ``MY_INSTALL_TREE``. In order to find these packages, users must extend their environment as required for the system/the packages to be installed:
 
 .. code-block:: console
+
    export PATH="$MY_INSTALL_TREE/bin:$PATH"
    export CPATH="$MY_INSTALL_TREE/include:$PATH"
    export LD_LIBRARY_PATH="$MY_INSTALL_TREE/lib64:$MY_INSTALL_TREE/lib:$LD_LIBRARY_PATH"
@@ -241,5 +282,17 @@ Users can build multiple packages outside of spack and install them in a separat
    # Python packages, use correct lib/lib64 and correct python version
    export PYTHONPATH="$MY_INSTALL_TREE/lib/pythonX.Y/site-packages:$PYTHONPATH"
 
-Python packages can be added using ``python setup.py install --prefix=...`` or ``python3 -m pip install --no-deps --prefix=...``. The ``--no-deps`` options is very important, because ``pip`` may otherwise attempt to install dependencies that already exist in spack-stack. These dependencies are not only duplicates, they may also be different versions and/or compiled with different compilers/libraries (because they are wheels).
+Python packages can be added in various ways:
+
+1. Using ``python setup.py install --prefix=$MY_INSTALL_TREE ...`` or ``python3 -m pip install --no-deps --prefix=$MY_INSTALL_TREE ...``. The ``--no-deps`` options is very important, because ``pip`` may otherwise attempt to install dependencies that already exist in spack-stack. These dependencies are not only duplicates, they may also be different versions and/or compiled with different compilers/libraries (because they are wheels). This approach requires adding the appropriate subdirectories of ``$MY_INSTALL_TREE`` to the different search paths, as shown above.
+
+2. Using Python virtual environments. Two important flags need to be passed to the command that creates the environment ``--system-site-packages`` and ``--without-pip``. After activating the environment, packages can be installed using `python3 -m pip` without having to specify ``--no-deps`` or ``--prefix``, and without having to manually modify ``PATH``, ``PYTHONPATH``, etc.
 
+.. code-block:: console
+
+   python3 -m venv --system-site-packages --without-pip $MY_INSTALL_TREE
+   source $MY_INSTALL_TREE/bin/activate
+   python3 -m pip install ...
+
+.. note::
+   Users are equally strongly advised to not use ``conda`` or ``miniconda`` in combination with Python modules provided by spack-stack, as well as not installing packages other than ``poetry`` in the basic ``miniconda`` installation for spack-stack (if using such a setup).

doc/source/Overview.rst

@@ -7,7 +7,7 @@ spack-stack is a collaborative effort between the NOAA Environmental Modeling Ce
 
  `Spack <https://github.com/spack/spack>`_ is a community-supported, multi-platform, Python-based package manager originally developed by the Lawrence Livermore National Laboratory (LLNL; https://computing.llnl.gov/projects/spack-hpc-package-manager). It is provided as a submodule so that a stable version can be referenced. See the `Spack Documentation <https://spack.readthedocs.io/en/latest>`_ for more information.
 
-spack-stack is mainly a collection of Spack configuration files, but provides a Spack extension to simplify the installation process (see :numref:`Section %s <SpackStackExtension` for details):
+spack-stack is mainly a collection of Spack configuration files, but provides a Spack extension to simplify the installation process (see :numref:`Section %s <SpackStackExtension>` for details):
 
 - ``spack stack create`` is provided to copy common, site-specific, and application-specific configuration files into a coherent Spack environment and to create container recipes