Use conda-lock files for 2i2c maintained images

[jmunroe]

Context

2i2c encourages communities to follow best-practices established in our hub image template when communities need to customise their hub images. This repository is now quite old, and needs to be updated. Furthermore, repo2docker, the underlying tool we use to build OCI images from the Git repo, does not support lock-files: jupyterhub/repo2docker#1157. This means that every package addition leads to a resolve, which might not be what we want from a security and stability perspective.

What we need to do

• Update the hub image template to establish CI workflows and a Dockerfile that integrates with a locking solver like conda-lock.
• Look to work with existing communities like nasa-cryo to update their existing workflows

Definition of Done

• The feature/service is technically complete
• The feature/service been tested with one or more users (if applicable)
• The feature/service been deployed to a production cluster
• The feature/service has been shown to be replicably deployable
• The feature/service follows our definition of minimally documented
• The feature/service has been added to our product menu (if applicable).
• Lessons learned have been documented
• The feature's availability has been communicated to Sales
• The feature has been communicated to our communities

[Gman0909]

@jmunroe could you please refine this initiative with some context following the standard Platform Initiative template?

[jmunroe]

I can't take credit for this idea -- I think this was a suggestion put forward by @agoose77 (maybe from a PS retro?)

Do you still want me to try to refine it?

[Gman0909]

Without context and a statement of value, it’s never going to be picked up, so I think someone should :)

[agoose77]

On it!

[weiji14]

Just wanted to chime in that we've been using conda-lock files on the CryoCloud hub, so happy to point people to that as a reference if needed. Some things we've done recently:

More automation 🤖:

• Enabled dependabot updates to conda environment.yml files, and have a bot update the associated conda-lock.yml on dependabot PRs.
  • Implementation: Enable dependabot updates to conda environment.yml file CryoInTheCloud/hub-image#152 + patches 1 & 2
  • Example dependabot PR where /condalock is being run: Bump jupyter-book from 2.0.0b3 to 2.1.0 CryoInTheCloud/hub-image#159 (comment)

Pressing issues ☹️:

• The conda-lock solver might take a long time if you have too many pinned dependencies - Condalock action failed after ~2 hours CryoInTheCloud/hub-image#157

I'll admit that it's been a lot of maintenance work trying to ensure reproducibility with these conda/pip lockfiles, and we've discussed in a thread on faster alternatives to conda-lock here, with an eye on conda/ceps#107. But I'm hopeful that we'll have a reliable semi- to fullly- automated dependency update workflow in the next year or so, and would love it if it can be rolled out across more 2i2c hubs (either via 2i2c, or from the ground-up by the community hubs themselves).

[yuvipanda]

@weiji14 thank you for your amazing work there! I'd love to hear more when you have it more fully automated