Skip to content

Update Silo and HDF5#20752

Merged
cyrush merged 114 commits intodevelopfrom
feat-mcm86-04dec25-upd-hdf5-2.0-and-friends
Mar 11, 2026
Merged

Update Silo and HDF5#20752
cyrush merged 114 commits intodevelopfrom
feat-mcm86-04dec25-upd-hdf5-2.0-and-friends

Conversation

@markcmiller86
Copy link
Member

@markcmiller86 markcmiller86 commented Dec 8, 2025

Description

  • Updates Silo to 4.12.0 and CMake
  • Updates HDF5 to 2.0 and CMake
  • Patches build of Xdmf 2.1.1 for HF5-2.0
  • Fixes a few HDF5 plugins for HDF5-2.0
  • Patches and or adjusts bv_xxx.sh for some HDF5 dependent TPLs

Type of change

  • Bug fix~~
  • New feature~~
  • Documentation update~~
  • Other~~

How Has This Been Tested?

Reminders:

  • Please follow the style guidelines of this project.
  • Please perform a self-review of your code before submitting a PR and asking others to review it.
  • Please assign reviewers (see VisIt's PR procedures for more information).

Checklist:

  • I have commented my code where applicable.~~
  • I have updated the release notes.~~
  • I have made corresponding changes to the documentation.~~
  • I have added debugging support to my changes.~~
  • I have added tests that prove my fix is effective or that my feature works.~~
  • I have confirmed new and existing unit tests pass locally with my changes.~~
  • I have added new baselines for any new tests to the repo.~~
  • I have NOT made any changes to protocol or public interfaces in an RC branch.~~

@cyrush
Copy link
Member

cyrush commented Dec 8, 2025

So far looks great! I recommend we create a ticket that lists plugins that still need to use old HDF5 APIs via defines.

@markcmiller86
Copy link
Member Author

markcmiller86 commented Dec 8, 2025 via email

@markcmiller86
Copy link
Member Author

I was trying to build Xdmf against the one, parallel HDF5-2.0 I have installed and that fails, even after my patches for HDF5-2.0. Why? It fails because including hdf5.h in Xdmf sources causes a #include <mpi.h> and bv_xdfm.py ordinarily doesn't deal with building Xdmf against a parallel HDF5. Xdmf's CMake logic is very, very old. It is not using find_package() for HDF5 and so doesn't prime itself properly when building against a parallel-enabled HDF5.

There are a few options for situations like this...

  • Simply adjust bv_hdf5.sh to install both hdf5 and hdf5_mpi as before (which was the plan last week and my intention if I keep hitting issues like this).
  • There is a swizzle on the above I've at least tested with Xdmf and that is to arrange for H5_HAVE_PARALLEL to be #undef'd in HDF5 headers when building a serial lib like Xdmf against HDF5. This seems to work!
  • When building for parallel, inject -I/path/to/mpi/headers into Xdmf CMAKE_CXX_FLAGS
    • Note: Xdmf 2.1.1 does have some MPI functionality in it but it is related to their network-distributed global memory (NDGM) and unrelated to our needs for simple Xdmf I/O.

@markcmiller86
Copy link
Member Author

To get Xdmf library to add -I/path/to/mpi/includes when HDF5 library it is using is parallel enabled, I added some logic to use CMake to probe the HDF5 installation. That seems to work.

@markcmiller86
Copy link
Member Author

Ok, netcdf requires more love because it a) assumes hid_t is an int and in HDF5-2.0 that type was changed to int64_t and it has the same H5Oget_info...() issues seen in other libs.

I suspect HDF5-2.0 has a bug in the API mapping macros for H5Oget_info... methods. I've submitted an inquiry to The HDF Group about it. Its popping up in a lot of places due to a missing arg when compiling against 2.0.

@biagas
Copy link
Contributor

biagas commented Dec 11, 2025

You might try Netcdf 4.5.0. That's what I've been using on Windows (for ease of building). I had no issues with the VisIt using this version, nor with building it against HDF5 2.0 in my test builds this week.

@markcmiller86
Copy link
Member Author

You might try Netcdf 4.5.0. That's what I've been using on Windows (for ease of building). I had no issues with the VisIt using this version, nor with building it against HDF5 2.0 in my test builds this week.

So, the set of patches needed for netcdf-4.1.1 (now ~15 years old) with HDF5-2.0 is getting larger and more complex. I am close to having it working but I kinda like @biagas suggestion. @cyrush do you have strong opinions?

@cyrush
Copy link
Member

cyrush commented Dec 12, 2025

@markcmiller86 I think we should try the new version, would be good to remove ancient patches and have same code used on windows as linux if possible.

@markcmiller86
Copy link
Member Author

So, I was 4 patches in and apparently 2 patches away from a successful netcdf-4.1.1 build with HDF5-2.0. So, I've currently got that in this PR. I will revisit the issue before taking out of draft mode. But, for time being, I think its working now.

@markcmiller86
Copy link
Member Author

I added logic to silo plugin to provide a little more info about the file (and plugin) software versions...

Screenshot from 2025-12-15 10-54-09

@cyrush
Copy link
Member

cyrush commented Jan 9, 2026

documenting a snag on macOS:

[harrison37@zeliak hdf5-2.0.0-build (feat-mcm86-04dec25-upd-hdf5-2.0-and-friends)]$ cat bv_run_cmake.sh 
"/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/bin/cmake" -DBUILD_TESTING:BOOL=OFF -DHDF5_BUILD_CPP_LIB:BOOL=OFF -DHDF5_BUILD_JAVA:BOOL=OFF -DHDF5_BUILD_EXAMPLES:BOOL=OFF -DHDF5_BUILD_TOOLS:BOOL=ON -DHDF5_ENABLE_SUBFILING_VFD:BOOL=OFF -DCMAKE_C_COMPILER:STRING=clang -DCMAKE_CXX_COMPILER:STRING=clang++ -DCMAKE_C_FLAGS:STRING="-Wno-error=implicit-function-declaration -O2 " -DCMAKE_CXX_FLAGS:STRING="-O2 " -DCMAKE_INSTALL_PREFIX:PATH=/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/hdf5/2.0.0/darwin-arm64 -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_STATIC_LIBS:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=ON -DONLY_SHARED_LIBS:BOOL=ON -DHDF5_BUILD_HL_LIB:BOOL=ON -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON -DZLIB_INCLUDE_DIR:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/zlib/1.3.1/darwin-arm64/include" -DZLIB_LIBRARY_RELEASE:FILEPATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/zlib/1.3.1/darwin-arm64/lib/libz.dylib" -DHDF5_BUILD_FORTRAN:BOOL=OFF -DHDF5_ENABLE_PARALLEL:BOOL=ON -DMPI_C_COMPILER:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/mpich/3.3.1/darwin-arm64/bin/mpicc" -DMPI_CXX_COMPILER:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/mpich/3.3.1/darwin-arm64/bin/mpic++" -DMPI_C_COMPILER_INCLUDE_DIRS:STRING="" -DMPI_C_HEADER_DIR:PATH="" ../hdf5-2.0.0
-- Configuring incomplete, errors occurred!
[harrison37@zeliak hdf5-2.0.0-build (feat-mcm86-04dec25-upd-hdf5-2.0-and-friends)]$ ./bv_run_cmake.sh 
-- Final: share
-- Checking if complex number support is available
-- Using C99 complex number types
-- Checking if _Float16 support is available
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
-- ....All Warnings are enabled
-- Could NOT find MPI_C (missing: MPI_C_HEADER_DIR) (found version "3.1")
CMake Error at /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find MPI (missing: MPI_C_FOUND) (found version "3.1")
Call Stack (most recent call first):
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindMPI.cmake:1842 (find_package_handle_standard_args)
  CMakeLists.txt:784 (find_package)

I think this logic is not quite correct

    if [[ "$PAR_INCLUDE" != "" ]] ; then
         cmk_opts="${cmk_opts} \
             -DMPI_C_COMPILER_INCLUDE_DIRS:STRING=\"${PAR_INCLUDE_PATH}\" \
             -DMPI_C_HEADER_DIR:PATH=\"${PAR_INCLUDE_PATH}\""
     fi
    
     if [[ "$PAR_LIBS" != "" ]] ; then
         cmk_opts="${cmk_opts} -DMPI_C_LINK_FLAGS:STRING=\"${PAR_LINKER_FLAGS}\""
     fi

because we end up with:

-DMPI_C_COMPILER_INCLUDE_DIRS:STRING="" -DMPI_C_HEADER_DIR:PATH="" 

which short circuits the mpi detection logic and creates the failure.

@markcmiller86
Copy link
Member Author

Ok, so that is the build of HDF5-2.0.0 then via bv_hdf5.sh. I thought I copied that logic verbatim from bv_conduit.sh. What was your build_visit command-line and env?

@cyrush
Copy link
Member

cyrush commented Jan 9, 2026

I am confused as to why those flags are getting set since they seem to be empty.

I also hit issues with ADIOS2 and CGNS, here are the CGNS issues:

Configuring CGNS . . .
Invoking command to configure CGNS
CMake Error at /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find MPI (missing: MPI_C_FOUND)
Call Stack (most recent call first):
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindMPI.cmake:1842 (find_package_handle_standard_args)
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/hdf5/2.0.0/darwin-arm64/cmake/hdf5-config.cmake:120 (find_package)
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindHDF5.cmake:513 (find_package)
  CMakeLists.txt:3 (find_package)


HDF5 MPI include flag detected as ""

CGNS actually has a cmake build system, but it seems we are calling cmake via make and config.
We may want to directly use cmake.

I am going to try to get further w/o solving ADIOS2 or CGNS now.

@cyrush
Copy link
Member

cyrush commented Jan 9, 2026

Silo failed as well.

I think anything downstream of HDF5 that is cmake and that is not setting up MPI - or perhaps is setting it up after it looks for HDF5) is failing.

I don't understand the logic for multiple versoins of hdf5 (one with mpi and one without), but could things be crossed? I

CMake'ing Silo
"/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/bin/cmake" -DCMAKE_INSTALL_PREFIX:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/silo/4.12.0/darwin-arm64" -DBUILD_TESTING:BOOL=OFF -DSILO_ENABLE_INSTALL_LITE_HEADERS:BOOL=ON -DCMAKE_BUILD_TYPE:STRING=Release -DSILO_ENABLE_FORTRAN:BOOL=OFF -DSILO_ENABLE_PYTHON_MODULE:BOOL=ON -DSILO_PYTHON_DIR:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/python/3.13.9/darwin-arm64" -DSILO_ENABLE_SHARED:BOOL=ON -DSILO_ENABLE_HDF5:BOOL=ON -DSILO_HDF5_DIR:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/hdf5/2.0.0/darwin-arm64" -DSILO_HDF5_SZIP_DIR:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/szip/2.1/darwin-arm64" -DSILO_BUILD_FOR_BSD_LICENSE:BOOL=OFF -DSILO_ENABLE_HZIP:BOOL=ON -DSILO_ENABLE_FPZIP:BOOL=ON -DSILO_ZLIB_DIR:PATH="/Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/zlib/1.3.1/darwin-arm64" ../Silo-4.12.0
bash bv_run_cmake.sh
-- The CXX compiler identification is AppleClang 17.0.0.17000404
-- The C compiler identification is AppleClang 17.0.0.17000404
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND)
Call Stack (most recent call first):
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindMPI.cmake:1842 (find_package_handle_standard_args)
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/hdf5/2.0.0/darwin-arm64/cmake/hdf5-config.cmake:120 (find_package)
  /Users/harrison37/Work/github/visit-dav/visit/build-mb-develop-darwin-22-arm64/thirdparty_shared/third_party/cmake/3.31.8/darwin-arm64/share/cmake-3.31/Modules/FindHDF5.cmake:513 (find_package)
  CMake/SiloFindHDF5.cmake:65 (find_package)
  CMakeLists.txt:229 (include)

@cyrush
Copy link
Member

cyrush commented Mar 5, 2026

resolves #20727

Comment on lines +14 to +16
# Temporary install of these libs...
# qt szip vtk zlib
SET(HDF52O_BUILD_TMP /g/g11/miller86/visit/visit/build-Xdmf/my_install)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a public location?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed only for dane? Why is that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct...not a public location. It was for when I was first building just a portion of the TPLs involving some kind of HDF5 dependence. I have a public install of all TPLs now in 3.5.0-hdf5200 and will update the daneX.cmake files for that shortly.

Comment on lines -34 to -41
### NOTE: Modified from build_visit output
#-------
##
## Turn on DDT support.
##
VISIT_OPTION_DEFAULT(VISIT_DDT ON TYPE BOOL)
#-------

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why turn DDT off now? Was this an oversight when copying the new config site in?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch Justin. Will put this back. Thanks!


float** data = NULL;
int dims[3] = {1, 1, 1};
//int dims[3] = {1, 1, 1};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just delete this line instead of commenting it out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to because I wanted it to be consistent with other portions of the file where this same coding was used including a commented out identical line.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha

bounds[0] = (x < bounds[0] ? x : bounds[0]);
bounds[1] = (x > bounds[1] ? x : bounds[1]);
float y = current_tmp[0];
float y = (rank > 1 ? current_tmp[1] : 0.0f);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why make this change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I will revert. This was part of an AI assisted change to fix hid_t/int confusions and open/close pairings.

Comment on lines +639 to +650
if (rank < 3)
{
bounds[4] = -(bounds[1]-bounds[0])*0.5;
bounds[5] = +(bounds[1]-bounds[0])*0.5;
}
else
{
// HACK! The VTK point locator doesn't work very well with 2D objects,
// so make this appear to be 3D by using the X-coordinate.
bounds[4] = bounds[0];
bounds[5] = bounds[1];
}
else
{
bounds[4] = -(bounds[1]-bounds[0])*0.5;
bounds[5] = +(bounds[1]-bounds[0])*0.5;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why swap the if condition logic?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is supposed to handle the case where there are less than 3 dimensions (rank < 3). Looking, neither the original nor my (AI assisted) swap makes sense. The original set the Z min to minus 1/2 the X range and the Z max to plus 1/2 the X range if there rank < 3 and set Z min to X min and Z max to X max when rank >= 3. Neither of those make sense to me but neither does reversing it.

Comment on lines -316 to +322
H5Dclose(coordMetaData[i].varId);
/*H5Dclose(coordMetaData[i].varId);*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to leave this comment or can we delete?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, unfort. we have no test data for this plugin. I think this and a few other HDF5 plugins should be wholly removed. I don't think this is being used.

That said, the actual close for coordMetaData[i].varId I think occurs elsewhere (FreeUpResources or maybe the destructor). So, there should be a note there explaining why we don't close it here and I left it in commented to suggest that. But, lemme make that clearer

err = H5Oget_info_by_name(groupId,
objectName,
&objectInfo,
H5O_INFO_ALL,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatting?

…com:visit-dav/visit into feat-mcm86-04dec25-upd-hdf5-2.0-and-friends
@cyrush
Copy link
Member

cyrush commented Mar 9, 2026

Re DDT support, I am not sure it matters if we keep it on for rzwhippet.
We should discuss.

markcmiller86 and others added 8 commits March 9, 2026 13:02
Co-authored-by: Justin Privitera <35237779+JustinPrivitera@users.noreply.github.com>
Co-authored-by: Justin Privitera <35237779+JustinPrivitera@users.noreply.github.com>
Co-authored-by: Justin Privitera <35237779+JustinPrivitera@users.noreply.github.com>
Ensure all hdf5-dependent libraries find/use the same zlib.
Modified our Find zlib to use find_package(ZLIB).
Replaced use of ZLIB_LIB in DataManualExamples to ZLIB_LIBRARY to match rest of VisIt.
Add zlib patch for silo.
Tweak xdmf settings.
…com:visit-dav/visit into feat-mcm86-04dec25-upd-hdf5-2.0-and-friends
@cyrush
Copy link
Member

cyrush commented Mar 11, 2026

Success!

@cyrush cyrush merged commit a6c0c30 into develop Mar 11, 2026
7 checks passed
@cyrush cyrush deleted the feat-mcm86-04dec25-upd-hdf5-2.0-and-friends branch March 11, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants