Is it possible to run jobs configured with the API via CLI?

[jpavlav]

Assume the minimal example provided in the documentation is in a file called job_test.py:

import sys

from avocado.core.job import Job

with Job.from_config({'run.references': ['/bin/true']}) as job:
    sys.exit(job.run())

Is there some way to execute this test via avocado run or perhaps one of the other entry points?

[clebergnu]

Hi @jpavlav ,

Yes, it should be as easy as adding:

#!/usr/bin/env python

import sys

from avocado.core.job import Job

with Job.from_config({'run.references': ['/bin/true']}) as job:
    sys.exit(job.run())

And setting the executable bit on the file, say:

$ chmod +x job_test.py
$ ./job_test.py 
JOB ID     : f3469067e8a9727630ff152efee92b7e0f16a6e3
JOB LOG    : /home/cleber/avocado/job-results/job-2021-04-30T17.07-f346906/job.log
 (1-1/1) /bin/true: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/cleber/avocado/job-results/job-2021-04-30T17.07-f346906/results.html
JOB TIME   : 0.36 

[clebergnu]

@jpavlav I now realize your question was a little bit different than what I answered. We do not have a tool / entry point that would let you do something like:

avocado job-run [PATH_TO_JOB_SCRIPT]

Simply because Python does the job or running the API pretty well. But, we do anticipate the usefulness of a tool or entrypoint that can read from a config and run a job. Such as:

avocado run-job-from-config [PATH_TO_CONFIG]

And config may be a JSON file with the "same" contents you'd give to Job.from_config().

[clebergnu]

IIUC, the first point is "be able to generate a TestSuite where the list of run.references were calls to run_kselftest.sh -c {collection} for each collection", and that's easy to do.

A TestSuite's tests, when using the nrunner architecture, contains Runnables.

A Runnable that will execute a command as a test (via the ExecTestRunner), can be created with:

runnable = Runnable(kind='exec-test', uri='run_kselftest.sh', args=('-c', '{collection'}))

So a simple loop over the list of "collections" should do the trick.

tests = []
for collection in collections:
   tests.append(Runnable(kind='exec-test', uri='run_kselftest.sh', args=('-c', collection))

suite = TestSuite("kselftests", tests=tests)

The second point is indeed missing, but somewhat on purpose because AFAICT we haven't seen the need for that. But we can always reconsider (and take contributions :).

[jpavlav]

Yeah I had the first bit figured out'ish 😄. Was planning on doing something like this:

run_references = [f'run_kselftest.sh -c {collection}' for collection in collections]
config = {'run.test_runner': 'nrunner', 'run.references': run_references}

suite = TestSuite.from_config(name='kselftests', config=config)
with Job(config, [suite]) as j:
    sys.exit(j.run())

It was that second bit that I'd been digging through the code base/playing with all the different entrypoints to see if it existed. If I were to make a contribution to build out something like run-job-from-config, would you see that as a subcommand to avocado? It would be a plugin, but where do you think it would fit structurally @clebergnu ?

[clebergnu]

Cool. But FIY, references that contain a command and then arguments are not currently supported as "test references". This comes from a quite a long time ago when it was very hard (if not impossible) to unequivocally distinguish between arguments and another reference in the command line parsing.

Now, with the Job API, that limitation is gone, but I have the feeling some tweaks would be needed.

About the command, I'd go with a sub-command to avocado indeed. FIY, the avocado replay command would be very similar to what we're describing here.

[jpavlav]

Hey there! Been super swamped and so I haven't followed up, but I wanted to raise a bit of strange behavior and find out if I'm just missing something here. I've tried setting up something similar to what you described in an earlier comment as a test:

import sys
from pathlib import Path

from avocado.core.job import Job
from avocado.core.suite import TestSuite
from avocado.core.nrunner import Runnable


collections = ['one', 'two']

runnable_config = {
    'runner.exectest.exitcodes.skip': [],
    'runner.output.color': 'auto',
    'runner.output.colored': True,
    'runner.output.utf8': True,
    'runner.timeout.after_interrupted': 60,
    'runner.timeout.process_alive': 60,
    'runner.timeout.process_died': 10
}

tests = [
    Runnable(
        kind='exec-test',
        uri=f'{Path.cwd()}/test.sh',
        args=('-u', collection),
        config=runnable_config
    )
    for collection
    in collections
]

config = {'run.test_runner': 'runner'}

suite = TestSuite('funtests', tests=tests)

with Job(config, test_suites=[suite]) as j:
    sys.exit(j.run())

And the test script that it is calling is super simple, just this:

#!/bin/bash

while getopts "u:" opt; do
    case $opt in
        u)
            echo "${OPTARG}"
            exit 0
            ;;
    esac
done

Whenever I run this, the tests are SKIPped for some reason:

python experiment.py
JOB ID     : 658c1728fafd2ce23c9c93188af2f13d1caa5cd7
JOB LOG    : /Users/jpalmer/avocado/job-results/job-2021-06-02T10.10-658c172/job.log
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 1.90 s

I've been banging my head against this for a bit. Setting breakpoints at various places (statemachine.py, runner_nrunner.py etc) and just haven't been able to come up with a reason why these are being skipped. I can provide more information if need be, even file a bug if that's what it comes to. I figured I'd reach out first to someone who has more context than I do 😄 .

[jpavlav]

Oh forgot to @clebergnu ☝️ whenever you get a chance 😄 . Thanks!