airflow 1.10.6 conflict jsonschema version with boundary-layer 1.7.24

[freephys]

airflow 1.10.6 Requirement.parse('jsonschema>=3.0.1<4'), {'flask-appbuilder'})

boundary-layer 1.7.24 has requirement jsonschema<3.0,>=2.6.0, but you'll have jsonschema 3.2.0 which is incompatible.

So any suggestions to by-pass this by relaxing version or what is the version of airflow to work with boundary-layer?

[dossett]

Hi @freephys, we target 1.10.3 with boundary layer currently. Python dependencies are challenging, we'd love to hear any ideas for making them more flexible across different versions of Airflow.

[mchalek]

Oh this is an interesting issue. Oddly enough, I don't think we even install boundary-layer on the same hosts that run airflow (Aaron could correct me on this, but in principle at least it's not necessary). We always run boundary-layer on our CI hosts, which push the rendered DAGs to the Airflow hosts. So, aside from probably an initial attempt to match up any relevant versions years ago, we do not make an active attempt to ensure that Airflow and boundary-layer have compatible library dependencies.

But, maybe we should! I have seen some other libraries out there that approach the dag-rendering problem differently from us, and one of the things I have seen that could potentially be interesting is an approach wherein the yaml files are loaded directly within the Airflow DAG and rendered at runtime. I'm not sure I view this as a better approach for production deployments like those at Etsy, but I can see how it might be a convenient way to get started for people who don't want to set up a full CI workflow for testing and rendering the DAGs before pushing them to Airflow.

More to the point, though, @freephys the jsonschema library is one I can't imagine being too difficult to upgrade. I don't have the time to do this myself anytime soon, but if you wanted to try incrementing the version dependency in boundary-layer to match Airflow, I imagine the unit tests would be able to detect any problems, and I think this seems like a reasonable improvement for us to incorporate.

[dossett]

@mchalek That's right, we don't install boundary-layer on Airflow hosts. I agree that making it easy to do that, at least for the version we target (again, currently 1.10.3), is a nice idea. I checked and 1.10.3 has no dependency on jsonschema so we'd have a lot of latitude with that!

@freephys Could you open a PR that aligns our jsonschema dependency with what 1.10.6 expects? Our use of that library is very minimal so hopefully an upgrade would be smooth.