[Jobs] joblib integration

[lhoestq]

Joblib integration

Programmatically run many python functions in parallel and get the results back

e.g.

from joblib import Parallel, delayed

from huggingface_hub.joblib import register_hf_jobs


register_hf_jobs()
parallel = Parallel(backend="hf-jobs")  # pass flavor, secrets etc. here

def train(params):
    # your training code here

all_params = [...]  # all the params to try

# Define your tasks
tasks = [delayed(train)(params) for params in all_params]

# Run the tasks in parallel on HF Jobs
results = parallel(tasks)

Happy to mark this as experimental if you think it's needed, or to make a separate package if you think it should live in huggingface_hub

---

Details

Under the hood, it submits a uv script from this template

import cloudpickle

print("Running {func_name}")
func = cloudpickle.loads({cloudpickled_func})
try:
    result = func()
except Exception as err:
    print("{error_sentinel}")
    print(cloudpickle.dumps(err))
    exit(1)
else:
    print("{result_sentinel}")
    print(cloudpickle.dumps(result))

and the result or error is retrieved from the logs thanks to the sentinel that separate regular logs from the result or error.

TODO:

• tests
• docs

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.