Skip to content

jenkins: use Python venv#2722

Merged
lunarpapillo merged 3 commits intoLunarG:devfrom
lunarpapillo:bob-gfxrecon-venv
Mar 3, 2026
Merged

jenkins: use Python venv#2722
lunarpapillo merged 3 commits intoLunarG:devfrom
lunarpapillo:bob-gfxrecon-venv

Conversation

@lunarpapillo
Copy link
Contributor

@lunarpapillo lunarpapillo commented Feb 24, 2026

As per:

Use venv and do install for all jobs that use VulkanTests
https://github.com/LunarG/Projects/issues/1018

this updates the Jenkins integration for GFXReconstruct pipelines to use the common boilerplate now used for all Jenkins multi-configuration jobs, including the creation and use of a Python virtual environment, as per:

https://github.com/LunarG/VulkanTests/blob/master/doc/administrator/jenkins-jobs.md

Other than the Python virtual environment, the pattern for checking out repositories into a subdirectory has been modified as per current best practices, to use the git "-C " switch to manipulate the repository in the subdirectory instead of explict "cd" commands; and artifact archival has been expanded to include the "python-venv.txt" file created during Python virtual environment initialization, so it can be examined in case of unexpected failure.

ci/runJob.bat:

  • Windows implementation

ci/runJob.sh:

  • Mac and Linux implementation

ci/runJobAndroid.sh:

  • Android on Linux host implementation

ci/runJob.groovy:

  • update all artifact archival to include python-venv.txt

@lunarpapillo lunarpapillo requested a review from a team as a code owner February 24, 2026 02:27
@lunarpapillo lunarpapillo requested review from a team, beau-lunarg and johnzupin and removed request for a team February 24, 2026 02:27
antonio-lunarg
antonio-lunarg reviewed Feb 24, 2026
View reviewed changes
antonio-lunarg
antonio-lunarg approved these changes Feb 24, 2026
View reviewed changes
Copy link
Contributor

@antonio-lunarg antonio-lunarg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once addressing my comment, LGTM

@lunarpapillo lunarpapillo added the approved-to-run-ci Can run CI check on internal LunarG machines label Feb 24, 2026
mikes-lunarg
mikes-lunarg approved these changes Mar 2, 2026
View reviewed changes
Copy link
Contributor

@mikes-lunarg mikes-lunarg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one question

ci/runJobAndroid.sh
python3 VulkanTests/gfxrecontest.py --os AndroidTestOS --build-mode $BUILD_MODE $ANDROID_TEST_ARGS --suite "ci-gfxr-suites/$GFXRECON_TRACE_SUBDIR/$TEST_SUITE" --trace-dir "$GFXRECON_TRACE_DIR" --result-dir "$RESULTS_DIR" No newline at end of file

python3 -m venv $WORKSPACE/python-venv
$WORKSPACE/python-venv/bin/python3 -m pip install --no-cache-dir -r VulkanTests/requirements.txt > $WORKSPACE/python-venv.txt 2>&1
Copy link
Contributor

@mikes-lunarg mikes-lunarg Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious if there is a specific reason to use the full venv python executable path instead of using python-venv/bin/activate

Copy link
Contributor Author

@lunarpapillo lunarpapillo Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, to be as consistent as possible with other CI.

In "mainline" CI, the test script typically starts with the current directory in the repository code, but sometimes the current directory is in the workspace root.

Using the absolute path to $WORKSPACE works in all cases.

In a related issue... we could just create the venv wherever we are (whether it's the workspace root or the repository root) - but it turns out creating the venv in the VulkanTests or ci-servers repository root breaks their tests, because their tests search for all Python files within the repository root and runs pylint on them. Creating the venv inside the repository root taints the repository with hundreds of additional Python files that don't pass pylint and causes tests to fail.

So we always create the venv explicitly in the workspace root, and we always refer to it directly in the workspace root for consistency both with its creation and with its use across all Jenkins jobs.

@lunarpapillo
Copy link
Contributor Author

lunarpapillo commented Mar 2, 2026

Squashing and rebasing to bring branch up-to-date

@lunarpapillo
jenkins: use Python venv
f34f1af 
As per:

    Use venv and do install for all jobs that use VulkanTests
    LunarG/Projects#1018

this updates the Jenkins integration for GFXReconstruct pipelines to use the
common boilerplate now used for all Jenkins multi-configuration jobs, including the
creation and use of a Python virtual environment, as per:

    https://github.com/LunarG/VulkanTests/blob/master/doc/administrator/jenkins-jobs.md

Other than the Python virtual environment, the pattern for checking out repositories
into a subdirectory has been modified as per current best practices, to use the
git "-C <subdirectory>" switch to manipulate the repository in the subdirectory instead
of explict "cd" commands; and artifact archival has been expanded to include the
"python-venv.txt" file created during Python virtual environment initialization, so
it can be examined in case of unexpected failure.

ci/runJob.bat:
-   Windows implementation

ci/runJob.sh:
-   Mac and Linux implementation

ci/runJobAndroid.sh:
-   Android on Linux host implementation

ci/runJob.groovy:
-   update all artifact archival to include python-venv.txt
@lunarpapillo lunarpapillo merged commit b34c591 into LunarG:dev Mar 3, 2026
8 checks passed
@lunarpapillo lunarpapillo deleted the bob-gfxrecon-venv branch March 3, 2026 23:48
lunarpapillo added a commit to lunarpapillo/gfxreconstruct that referenced this pull request Mar 4, 2026
@lunarpapillo
jenkins: second effort to get pipelines using venv
cfbfc54 
The first attempt here:

    jenkins: use Python venv
    LunarG#2722

was completely wrong; it modified files "runJob.bat", "runJob.sh", and "runJobAndroid.sh"
that aren't actually used in CI.

This effort modifies the "runTest.bat", "runTest.sh", and "runTestAndroid.sh" files
instead, which seems to be the correct way to affect CI behavior.

This PR temporarily includes a change to "test.ref" so that its testing will also evoke
the latest VulkanTests changes (which require a new Python module, and will clearly fail
if run without a Python virtual environment).  The test.ref changes will be kept only
for testing, and will be removed before being merged.
zongdu-arm pushed a commit to zongdu-arm/gfxreconstruct that referenced this pull request Mar 6, 2026
@lunarpapillo @zongdu-arm
jenkins: use Python venv (LunarG#2722)
8c9078e 
As per:

    Use venv and do install for all jobs that use VulkanTests
    https://github.com/LunarG/Projects/issues/1018

this updates the Jenkins integration for GFXReconstruct pipelines to use the
common boilerplate now used for all Jenkins multi-configuration jobs, including the
creation and use of a Python virtual environment, as per:

    https://github.com/LunarG/VulkanTests/blob/master/doc/administrator/jenkins-jobs.md

Other than the Python virtual environment, the pattern for checking out repositories
into a subdirectory has been modified as per current best practices, to use the
git "-C <subdirectory>" switch to manipulate the repository in the subdirectory instead
of explict "cd" commands; and artifact archival has been expanded to include the
"python-venv.txt" file created during Python virtual environment initialization, so
it can be examined in case of unexpected failure.

ci/runJob.bat:
-   Windows implementation

ci/runJob.sh:
-   Mac and Linux implementation

ci/runJobAndroid.sh:
-   Android on Linux host implementation

ci/runJob.groovy:
-   update all artifact archival to include python-venv.txt
lunarpapillo added a commit to lunarpapillo/gfxreconstruct that referenced this pull request Mar 6, 2026
@lunarpapillo
jenkins: second effort to get pipelines using venv
ade8560 
The first attempt here:

    jenkins: use Python venv
    LunarG#2722

was completely wrong; it modified files "runJob.bat", "runJob.sh", and "runJobAndroid.sh"
that aren't actually used in CI.

This effort modifies the "runTest.bat", "runTest.sh", and "runTestAndroid.sh" files
instead, which seems to be the correct way to affect CI behavior.

This PR temporarily includes a change to "test.ref" so that its testing will also evoke
the latest VulkanTests changes (which require a new Python module, and will clearly fail
if run without a Python virtual environment).  The test.ref changes will be kept only
for testing, and will be removed before being merged.

During testing I noticed an unexpected Windows warning:

    C:\j\workspace\gfxr-pipelines_gfxr_PR-2753\gfxreconstruct\ci-gfxr-suites>ls .
    'ls' is not recognized as an internal or external command,
    operable program or batch file.

So I also fixed "cloneSuites.bat", which had a Linux "ls .", to use a Windows "dir" instead.
lunarpapillo added a commit that referenced this pull request Mar 7, 2026
@lunarpapillo
jenkins: second effort to get pipelines using venv (#2753)
1426423 
The first attempt here:

    jenkins: use Python venv
    #2722

was completely wrong; it modified files "runJob.bat", "runJob.sh", and "runJobAndroid.sh"
that aren't actually used in CI.

This effort modifies the "runTest.bat", "runTest.sh", and "runTestAndroid.sh" files
instead, which seems to be the correct way to affect CI behavior.

This PR temporarily includes a change to "test.ref" so that its testing will also evoke
the latest VulkanTests changes (which require a new Python module, and will clearly fail
if run without a Python virtual environment).  The test.ref changes will be kept only
for testing, and will be removed before being merged.

During testing I noticed an unexpected Windows warning:

    C:\j\workspace\gfxr-pipelines_gfxr_PR-2753\gfxreconstruct\ci-gfxr-suites>ls .
    'ls' is not recognized as an internal or external command,
    operable program or batch file.

So I also fixed "cloneSuites.bat", which had a Linux "ls .", to use a Windows "dir" instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mikes-lunarg mikes-lunarg mikes-lunarg approved these changes

@beau-lunarg beau-lunarg beau-lunarg approved these changes

@antonio-lunarg antonio-lunarg antonio-lunarg approved these changes

@johnzupin johnzupin Awaiting requested review from johnzupin

+1 more reviewer

@beauknowssoftware beauknowssoftware beauknowssoftware approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved-to-run-ci Can run CI check on internal LunarG machines

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@lunarpapillo @mikes-lunarg @beauknowssoftware @beau-lunarg @antonio-lunarg