Linux + Windows dependency builds

[ericmehl]

With pieces of Gaffer for Windows slowly getting merged into Cortex and soon Gaffer, I suppose it's time to face the terrifying idea of how we are going to get the Windows dependencies organized and maintainable.

Especially for the "maintainable" part I'm guessing we all agree that it's best to have as few differences between the Linux and Windows configs, correct? Alternatively we could go with parallel configurations and maintain them separately. That's pretty much how I have them building now, but I think it's worth an effort to get them combined as much as possible.

A while back we were converting Linux syntax to Windows for things like file copying but it eventually grew into a mess of special case handling that was getting less and less reliable. Now I'm thinking it might be better to extend the configuration file scheme with more "semantic" commands that take the place of the system commands. Something like {"copy": ("source", "destination")} to replace cp source destination. Then the config.py script can do the right thing based on the platform.

Scanning through the configs, it doesn't look there would be too many commandlets to make: cp, mv, rm, mkdir, cd, and a couple for the build systems look like it would take care of almost all of it?

The other big issue is getting the different build commands in sync. I've resorted to a "just get it to work" approach for now, where I create totally separate Windows configurations. Not ideal, but as an intermediate step it has the advantage of easily being able to see what parameters are common to both platforms.

For example, cmake commands seem to often need to include additional parameters on Windows. I'm not 100% sure those are really always needed so some can come out. For the others I'm thinking it might be useful to have a cumulative set of parameters passed to cmake that build.py compiles. All the parameters in common can go in the main configuration, then each platform can have additional parameters that config.py adds on to the common parameters. This would probably benefit also from the semantic style as well such as {"cmake": ["all", "parameters", "here"]}. Then it would be easy for config.py to combine the lists.

That would make the differences between platforms more explicit and easier to figure out where something is going wrong when upgrading packages. Hopefully?

Another more out-there idea I had was converting the config files to yaml, which has some built-in functionality for things like combining lists. I personally find yaml to be kind of esoteric, but maybe I'd warm to it if I worked with it more often. It would also add a dependency to the dependency builder (as far as I know yaml parsing isn't built in to Python) which makes me a little sad.

I'm certainly open to other ideas...

[ericmehl]

The latest, and rather unholy, Windows dependency build is here: https://github.com/hypothetical-inc/gafferDependencies/tree/msvc2017_v9

[johnhaddon]

Especially for the "maintainable" part I'm guessing we all agree that it's best to have as few differences between the Linux and Windows configs, correct?

Agreed. I fairly regularly have to tweak a version here or there, and I'd love to have those tweaks automatically reflected in the Windows builds too. What terrifies me though is that if such a tweak breaks the Windows build, I will be floundering. But at least we'll know there's a problem earlier. The dependencies builds take an absolute age, but I think the timeout on GitHub actions might actually be generous enough to plod through them, and we don't make PRs all that often. So I think a part of this might be to set up CI, and make the dependencies release builds via that too.

Now I'm thinking it might be better to extend the configuration file scheme with more "semantic" commands that take the place of the system commands.

That sounds good to me.

cmake commands seem to often need to include additional parameters on Windows...All the parameters in common can go in the main configuration, then each platform can have additional parameters that config.py adds on to the common parameters.

That sounds reasonable. We already do something similar in places using variables to split out platform-specific part of the commands.

Another more out-there idea I had was converting the config files to yaml, which has some built-in functionality for things like combining lists. I personally find yaml to be kind of esoteric, but maybe I'd warm to it if I worked with it more often.

I find YAML a bit esoteric too - if we want some programming functionality, I'd be inclined to just use Python. We all know Python already, and it is quite good at building dictionaries and lists. This does raise the question though : if we're trending towards needing programming, then should we abandon the "purely declarative config" idea entirely?

I suppose actually it has never been purely declarative. All the metadata like downloads and url and license etc are simply declarative, and that has merit because we can easily validate them and automate the handling for all projects. But the commands section has always been programming, albeit by shell scripts. It seems this is where the need for more complex logic comes in, so perhaps we should just allow arbitrary python code in there, and provide utilities like doCMakeBuild( [ package, specific, args ] ) that encapsulate the common platform-specific logic?

[johnhaddon]

perhaps we should just allow arbitrary python code in there

This would also mean we could use shutil.copyfile() to replace cp etc, giving us a ready-made source of "semantic commands".

[ericmehl]

perhaps we should just allow arbitrary python code in there, and provide utilities like doCMakeBuild( [ package, specific, args ] ) that encapsulate the common platform-specific logic?

That sounds good to me. I suppose we are kind of dancing around that obvious need by forcing things to be declarative.

Or, if we really want to be meta, we could create a Gaffer graph to build all the dependencies...

[johnhaddon]

Or, if we really want to be meta, we could create a Gaffer graph to build all the dependencies...

😄 I actually did start playing around with this at one point, using the TaskNode system to build Gaffer itself. Contexts and Wedges were actually pretty cool tools in that regard, and so would be Spreadsheets now that we have them. What was missing was the dispatcher having an idea of tasks that were unchanged from a previous dispatch, so it could do partial rebuilds...

Now that we've considered that option, adding a few little Python functions seem quite harmless in comparison...

[boberfly]

Scanning through the configs, it doesn't look there would be too many commandlets to make: cp, mv, rm, mkdir, cd, and a couple for the build systems look like it would take care of almost all of it?

What I've found to be beneficial is to use cmake's platform-independent commands cmake -E for these ones which might omit the need for making declarative commands? eg. https://github.com/boberfly/GafferCycles/blob/cycles_2.93/dependencies/NanoVDB/config.py#L15

It also has a build system independent way of running make/make install eg. https://github.com/boberfly/GafferCycles/blob/cycles_2.93/dependencies/Embree/config.py#L31

Or, if we really want to be meta, we could create a Gaffer graph to build all the dependencies...

Totally! For something a bit more practical, Gaffer would be an amazing build tool for something like games. Often you have the game engine code that declares the asset schemas and need to build with the engine. Or if you target multiple platforms with varying contexts eg. compress textures for this game console's esoteric format, or use the lower LOD models for the Nintendo Switch but put the high-res ones on PS5, or the shaders need to build a certain way per-platform, that kind of stuff.