Feature request: Ability to dynamically build job/template description

[lithorus]

Use Case

When writing the job/template description it currently requires you to know all information before creating the object. There is no way to add eg. new Steps to a job, so one have to create an internal model first and then convert it to OJD instead of natively building the job directly.

Proposed Solution

Add CRUD functions to the model.

[ddneilson]

Hey Jimmy. Sorry for the delay, I thought that I had already responded but I see now that I had not.

You're spot on that this would be a very useful feature. The challenge that I imagine that you're hitting is that the model classes' constructor always runs the validation logic, and things like a JobTemplate with no Steps is not a valid job template.

It's not pretty, but as a workaround for now you could leverage the fact that all of the model implementations are Pydantic models. It is possible to construct models without validation (see: https://docs.pydantic.dev/1.10/usage/models/#creating-models-without-validation). That should allow for creating instances of the model classes that don't pass validation, and then you could, say, list.append() to the list of Steps in a JobTemplate to add to it.

I'd suggest still running the final model through the validation logic (JobTemplate(**template_model)) after construction to make sure that all is good, though.

If you have the time, and inclination then we do accept pull requests. ;-)

[lithorus]

Hey Jimmy. Sorry for the delay, I thought that I had already responded but I see now that I had not.

You're spot on that this would be a very useful feature. The challenge that I imagine that you're hitting is that the model classes' constructor always runs the validation logic, and things like a JobTemplate with no Steps is not a valid job template.

It's not pretty, but as a workaround for now you could leverage the fact that all of the model implementations are Pydantic models. It is possible to construct models without validation (see: https://docs.pydantic.dev/1.10/usage/models/#creating-models-without-validation). That should allow for creating instances of the model classes that don't pass validation, and then you could, say, list.append() to the list of Steps in a JobTemplate to add to it.

I'd suggest still running the final model through the validation logic (JobTemplate(**template_model)) after construction to make sure that all is good, though.

If you have the time, and inclination then we do accept pull requests. ;-)

Ok, that makes sense. So if I would be adding functions for eg. adding Steps, is there a desired naming structure for the functions?

Edit:
Hmm.. reading this, makes me wonder if it's a proper solution.

construct() does not do any validation, meaning it can create models which are invalid. You should only ever use the construct() method with data which has already been validated, or you trust.

Would there be a way to have multiple "levels" of validations? Eg. one for constructing and another level for when exporting/sending it? (Not (yet) familar with Pydantic)

[ddneilson]

So if I would be adding functions for eg. adding Steps, is there a desired naming structure for the functions?

Just follow the patterns of other methods. The main things to know are in the DEVELOPMENT.md:

1. We're intentional about the public interface: https://github.com/OpenJobDescription/openjd-model-for-python/blob/mainline/DEVELOPMENT.md#the-packages-public-interface
2. We use keyword args in the public functions/methods: https://github.com/OpenJobDescription/openjd-model-for-python/blob/mainline/DEVELOPMENT.md#use-of-keyword-only-arguments

Would there be a way to have multiple "levels" of validations? Eg. one for constructing and another level for when exporting/sending it?

Not in a way that doesn't involve duplicating the data model in its entirety that I can immediately see. Pydantic works by introspecting the model definitions and creating the validation logic from the annotations on the member declarations. It doesn't have a way to say "conditionally do this validation" that I'm aware of. The trick is that there are two use-cases for these models:

1. Input validation. As in a tool/system is given a template and it wants to make sure that it's correctly formed.
2. Template construction. Your use-case.

The implementation is heavily biased to (1) right now, and that makes case (2) challenging.

[lithorus]

This is what I was afraid of, that to eg. implement OJD in eg. OpenCue with it's more dynamic nature, one would first have to use the native outline format and then convert everything into OJD in a single step.
Although doing some quick googling, it looks like it might still be possible with a custom Validator?

[lithorus]

Hmm.. just checked my test script and it's indeed possible to create a job without any steps..

Edit:
Script I used (steps=[] also works) :

#!/usr/bin/python3
from openjd.model import model_to_object, JobParameterDefinition, decode_job_template
from openjd.model.v2023_09 import *
import yaml

command = CommandString('echo test 12434')
action = Action(command=command)

layer1 = Step(
    name="layer1",
    script=StepScript(
        actions=StepActions(
            onRun=action
        )
    ),
    parameterSpace=StepParameterSpace(
        taskParameterDefinitions={
            "tags": RangeListTaskParameterDefinition(
                type=TaskParameterType.STRING,
                range=["blender"]
            ),
            "services": RangeListTaskParameterDefinition(
                type=TaskParameterType.STRING,
                range=["blender"]
            ),
            "frames": RangeListTaskParameterDefinition(
                type=TaskParameterType.INT,
                range=["1-1"]
            ),
            "outputs": RangeListTaskParameterDefinition(
                type=TaskParameterType.STRING,
                range=["/tmp/blender/blender.#####.exr"]
            )
        },
        combination="(tags, services, frames)"
    )
)

job = Job(
    name='testJob',
    steps=[layer1],
    parameters={"show": JobParameter(value="testing", type=JobParameterType.STRING),
                "shot": JobParameter(value="sh0010", type=JobParameterType.STRING),
                "user": JobParameter(value="jimmy", type=JobParameterType.STRING),
                "email": JobParameter(value="jimmy@example.com", type=JobParameterType.STRING),
                "uid": JobParameter(value="1000", type=JobParameterType.INT),
                "facility": JobParameter(value="local", type=JobParameterType.STRING),
                "paused": JobParameter(value="0", type=JobParameterType.INT),
                "priority": JobParameter(value="1", type=JobParameterType.INT),
                "maxretries": JobParameter(value="2", type=JobParameterType.INT),
                "autoeat": JobParameter(value="0", type=JobParameterType.INT)
                }
)
obj = model_to_object(model=job)
print(yaml.dump(obj))

[lithorus]

Also, if I were to add eg. add_step(), would it require a new version of the model?

[ddneilson]

Ah, that worked because you used the Job model class instead of the JobTemplate model class. There's a nuance in the model... the Job is what you get after applying the job parameters to a JobTemplate. The JobTemplate is what you submit to a render farm.
You'll notice that the main difference is that the Job doesn't have a property for adding the job parameter definitions whereas the JobTemplate model does.

Also, if I were to add eg. add_step(), would it require a new version of the model?

Nope. That's an additive change that would be just fine. The model version is specific to the data model, not the functional model that we layer on top of that. If that makes sense...

[lithorus]

Ah, that worked because you used the Job model class instead of the JobTemplate model class. There's a nuance in the model... the Job is what you get after applying the job parameters to a JobTemplate. The JobTemplate is what you submit to a render farm. You'll notice that the main difference is that the Job doesn't have a property for adding the job parameter definitions whereas the JobTemplate model does.

Personally I still feel that the Job model is more usefull for something like OpenCue and other render managers, since the template model, seems too limiting and feels wrong to create custom templates per job. Job templates are definately nice to have for simple jobs, but the jobs I see at work are much more dynamic in nature and the concept of job templates doesn't fit very well there.

Edit:
If the template system really is the backbone of the system, it should be renamed to open-job-template-description 😄

[mwiebe]

After Felix joined the ASWF OpenCue discussion, I've taken a closer look at this issue and how Open Job Description in OpenCue could look. My thought was to take your idea here, of writing a pyoutline-like API for Open Job Description, and instead try writing an OpenJD backend for pyoutline directly. I used Kiro/Claude Opus to do this, and I think the structure is in good shape.

One thing I found is that without the EXPR extension I've been prototyping and just put up the RFC proposal for, swapping in the various tokens isn't very clean. I decided to just go ahead and use EXPR in this prototype, because I have a fully functional prototype of the EXPR extension working too. See OpenJobDescription/openjd-specifications#112, and I'm looking for feedback on that too!

Here's the prototype to add OpenJD output support for the pyoutline library: https://github.com/mwiebe/OpenCue/tree/openjd-prototype