Scheduling and MPI Interfaces

[FrankD412]

So this issue documents the proposed breakdown for the design of a SchedulingInterface and MpiInterface. From prior experience it is horrible to couple the MPI generation with the scheduling interface, and it is also horrible to rely on an actual instance of the class because it requires generating a bunch of instances when a class method API would do just fine. There would be commiserate enumerations to standardize returns.

I propose the following:

class SchedulingInterface(ABC):
    @abstractclassmethod
    def submit_script(cls, script_path, resource_spec):
        """
        Submit a script to a scheduler.
        """

    @abstractclassmethod
    def submit_command(cls, script_body, resource_spec):
        """
        Submit a string of commands to a scheduler.
        """

    @abstractclassmethod
    def resource_header(cls, resources, cluster_spec=None):
        """
        Construct the resource header for a scheduler.
        """

    @abstractclassmethod
    def cancel_job(cls, identifier):
        """
        Cancel the specified job through the scheduler API.
        """

   @abstractclassmethod
   def cancel_jobs(cls, identifier_list):
        """
        Cancel a list of jobs through the scheduler API.
        """

A generalized version of the above SchedulingInterface can be created that would take a serialized specification that could be indexed into that would only make it so that specialized cases needed a custom adapter (think something like Flux's Python APIs).

I'll write up a second comment about the MPI interface in a few.

[FrankD412]

I meant to come around to this sooner, but got sidetracked with other things. This functionality in the past is something I've coupled together with writing out scripts. By leaving this as a class API, we can now let a tool handle what it wants to do with the header, submission command, etc. If a tool wants to write out a script, it can rely on the SchedulingInterface can provide the header, and then we can rely on an MpiInterface to provide the command line contents. Something like:

script_contents = f"{SchedulingInterface.resource_header(resources)}\n{MpiInterface.substitute_mpi(resources)\n"
with open(path_to_script, "w") as script:
     script.write(script_contents)

SchedulingInterface.submit_script(path_to_script)

Obviously the above is simplified, but I want to get to something that's a bit more simple like above. Maestro couples the MPI with the scheduling and has hidden methods that subclasses have to override. It's not the most seamless thing in the world, and introduces complexities/required hardcoding when not using the scheduler's flavor of MPI.

Relatedly, I had an idea about adding methods to this for getting scheduler specific items from the environment (or for performing smart queries if a scheduler supports it). Something that would allow someone to write a Python worker that could know how much time was left, or to query the total resources in an allocation, etc. This functionality would also make it much easier to write resource aware workers that could be allocated and then used to manage pools of resources.