Skip to content

feat: add RPM-only build mode for presubmit CI#6961

Draft
agullon wants to merge 38 commits into
openshift:mainfrom
agullon:shiftweek/rpm-presubmit-ci
Draft

feat: add RPM-only build mode for presubmit CI#6961
agullon wants to merge 38 commits into
openshift:mainfrom
agullon:shiftweek/rpm-presubmit-ci

Conversation

@agullon

@agullon agullon commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add -rpm_only flag to ci_phase_iso_build.sh that skips composer config and image builds, only building RPMs
  • Skip manage_composer_config.sh create when running in RPM-only mode
  • Move rpm-install scenarios from scenarios-bootc/ to scenarios-rpm/

Context

Shift Week project: RPM-based presubmit CI reduces total CI time from ~55 min to ~30-35 min (~35-40% faster) by eliminating the image build phase for scenarios that don't need ostree/bootc deployment.

Test plan

  • Run ci_phase_iso_build.sh -rpm_only and verify it exits after build_rpms.sh
  • Verify existing (non-rpm_only) builds are unaffected
  • End-to-end RPM presubmit run with converted scenarios

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added an -rpm_only command-line option to run an RPM-focused build and stop after RPM package creation.
  • Bug Fixes
    • Improved RPM-only mode to bypass cache-related logic and skip later image/bootc build steps.
    • For RPM presubmit scenarios, mirror registry setup is now conditionally skipped to avoid unnecessary work.
  • Chores
    • Reduced GNU Parallel citation output noise during RPM builds by using a will-cite marker instead of invoking citation output.

Skip composer config and image builds when running with -rpm_only flag.
Move rpm-install scenarios from scenarios-bootc/ to scenarios-rpm/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 29, 2026
@openshift-ci

openshift-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci

openshift-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agullon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds RPM-only build handling in ci_phase_iso_build.sh, skips mirror-registry startup for matching presubmits, and changes build_rpms.sh to prepare GNU Parallel citation state with a file.

Changes

RPM-only CI flow

Layer / File(s) Summary
RPM-only build selection
test/bin/ci_phase_iso_build.sh
Recognizes -rpm_only, disables composer CLI builds, and runs ./bin/build_rpms.sh instead of the image and bootc build flow.
Mirror registry gating
test/bin/ci_phase_boot_and_test.sh, test/bin/scenario.sh
Sets SKIP_MIRROR_REGISTRY=true for RPM-only presubmits and checks that flag before starting mirror_registry.sh.
Parallel citation file setup
test/bin/build_rpms.sh
Creates ~/.parallel/will-cite before running parallel build jobs.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~5 minutes

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding an RPM-only build mode for presubmit CI.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Touched files are shell/scenario scripts only; no It/Describe/Context/When test titles were added or changed.
Test Structure And Quality ✅ Passed No Ginkgo test code was modified; the PR only changes shell scripts and scenario scripts, with no *_test.go files or Describe/It blocks touched.
Microshift Test Compatibility ✅ Passed The PR only changes shell/scenario scripts; no new Ginkgo It/Describe tests or MicroShift-unsupported APIs were added.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR only changes shell scripts; no Go/Ginkgo test declarations or SNO-unsafe test logic were added.
Topology-Aware Scheduling Compatibility ✅ Passed Only CI/test shell scripts changed; no manifests, operators, controllers, or scheduling constraints were introduced.
Ote Binary Stdout Contract ✅ Passed PR only changes CI shell scripts; no OTE binary main/init/test setup code writes to stdout were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The PR only changes shell orchestration and scenario routing; it adds no new Ginkgo tests or IPv4-sensitive test code.
No-Weak-Crypto ✅ Passed Touched scripts only add rpm-only branching and mirror-registry env handling; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges ✅ Passed PASS: The PR only changes shell scripts/scenario placement; no container/K8s manifests were added or modified with privileged settings.
No-Sensitive-Data-In-Logs ✅ Passed Touched lines only add RPM-only branching and parallel citation suppression; no new logging of secrets, tokens, PII, or customer data.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 29, 2026
agullon added 4 commits June 29, 2026 13:27
osbuild-composer is not installed in rpm_only mode, so create-workers
fails trying to start osbuild-worker systemd units that don't exist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
parallel --citation returns non-zero on newer versions (RHEL 10),
which kills the script under set -euo pipefail. Creating the file
directly is simpler and version-agnostic.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
configure-composer.sh installs parallel, mock, nginx etc. needed by
build_rpms.sh. Skipping it in rpm_only mode caused parallel not found.
The create-workers step is still skipped via COMPOSER_CLI_BUILDS=false.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
RPM-install scenarios pull container images directly from upstream
registries and don't need the local Quay mirror. The mirror requires
container-images-list which is only generated during image builds.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/bin/ci_phase_boot_and_test.sh`:
- Around line 56-59: Reset SKIP_MIRROR_REGISTRY before the scenario branching in
ci_phase_boot_and_test.sh so inherited values do not leak into non-RPM jobs.
Update the pre-submit logic around the SCENARIO_TYPE and SCENARIO_SOURCES checks
to initialize SKIP_MIRROR_REGISTRY to false (or explicitly clear it) before
setting it to true for the rpm-presubmits path, and ensure the non-RPM branches
leave it disabled so scenario.sh sees the correct value.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 5ef256d2-d333-4097-a80d-9ac3d8a2059d

📥 Commits

Reviewing files that changed from the base of the PR and between 69eca43 and d7b5067.

📒 Files selected for processing (2)
  • test/bin/ci_phase_boot_and_test.sh
  • test/bin/scenario.sh

Comment on lines +56 to +59
# RPM-only presubmits skip the mirror registry — scenarios pull images directly.
if [[ "${SCENARIO_TYPE:-}" =~ ^rpm-presubmits-.*$ ]]; then
export SKIP_MIRROR_REGISTRY=true
elif [[ "${SCENARIO_SOURCES:-}" =~ .*releases.* ]]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Reset SKIP_MIRROR_REGISTRY before branching.

This only exports true on the RPM-only path. Since scenario.sh now reads the inherited environment, a caller that already has SKIP_MIRROR_REGISTRY=true will also skip mirror setup for non-RPM jobs. Initialize it to false first, or clear it in the non-RPM branches.

Suggested fix
+# Default for all scenarios; override only for RPM-only presubmits.
+export SKIP_MIRROR_REGISTRY=false
+
 # RPM-only presubmits skip the mirror registry — scenarios pull images directly.
 if [[ "${SCENARIO_TYPE:-}" =~ ^rpm-presubmits-.*$ ]]; then
     export SKIP_MIRROR_REGISTRY=true
 elif [[ "${SCENARIO_SOURCES:-}" =~ .*releases.* ]]; then
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# RPM-only presubmits skip the mirror registry — scenarios pull images directly.
if [[ "${SCENARIO_TYPE:-}" =~ ^rpm-presubmits-.*$ ]]; then
export SKIP_MIRROR_REGISTRY=true
elif [[ "${SCENARIO_SOURCES:-}" =~ .*releases.* ]]; then
# Default for all scenarios; override only for RPM-only presubmits.
export SKIP_MIRROR_REGISTRY=false
# RPM-only presubmits skip the mirror registry — scenarios pull images directly.
if [[ "${SCENARIO_TYPE:-}" =~ ^rpm-presubmits-.*$ ]]; then
export SKIP_MIRROR_REGISTRY=true
elif [[ "${SCENARIO_SOURCES:-}" =~ .*releases.* ]]; then
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/bin/ci_phase_boot_and_test.sh` around lines 56 - 59, Reset
SKIP_MIRROR_REGISTRY before the scenario branching in ci_phase_boot_and_test.sh
so inherited values do not leak into non-RPM jobs. Update the pre-submit logic
around the SCENARIO_TYPE and SCENARIO_SOURCES checks to initialize
SKIP_MIRROR_REGISTRY to false (or explicitly clear it) before setting it to true
for the rpm-presubmits path, and ensure the non-RPM branches leave it disabled
so scenario.sh sees the correct value.

agullon added 4 commits June 29, 2026 16:12
SCENARIO_TYPE is not propagated to EC2 via SSH — use SCENARIO_SOURCES
which is already passed. Also fix el10 directory typo (presumits ->
presubmits).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Without this, rpm scenarios get type 'unknown', causing wrong output
directory names and broken VM domain lookups.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
In the normal flow, build_images.sh creates vm-storage/ as ec2-user.
When virsh pool-build runs later, it finds the dir already exists and
keeps ec2-user ownership. In rpm_only mode, build_images.sh is skipped,
so pool-build creates it as root, causing permission denied when
scenario.sh tries to mkdir inside it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
The rpm-install scenarios need installer ISOs to boot VMs via
kickstart. In normal flow, build_images.sh creates them. In rpm_only
mode, download them from the build cache instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
@agullon agullon force-pushed the shiftweek/rpm-presubmit-ci branch from 286676e to f940d1c Compare June 30, 2026 07:06
agullon added 12 commits June 30, 2026 09:44
Remove remove.robot and upgrade-successful.robot temporarily to
validate the CI pipeline end-to-end with just the install test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
- Extract common functions to scenarios-rpm/common-scenarios-rpm.sh
- Merge el9/el10 scenario directories into scenarios-rpm/presubmits/
- Rename scenarios to el98@standard1, el102@standard1, el98@upgrade
- Add install_microshift() bash function replacing install.robot
- Run suites/standard1/ for functional validation after RPM install

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
- Move functional tests (standard, storage, router, configuration,
  network, low-latency) to scenarios-rpm/presubmits for both el98/el102
- Move upgrade/backup/ostree scenarios from presubmits to periodics
- Remove duplicated scenarios between presubmits and periodics
- Remove empty scenarios/presubmits directory

22 RPM presubmit scenarios now provide fast (~34 min) functional
feedback. Ostree-specific tests (upgrades, greenboot, backups) remain
in bootc presubmits and periodics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
- el102: use configure_cdn_repo for fast-datapath (subscription-manager
  doesn't have it on RHEL 10)
- storage: wait for LVMS operator pod before running tests (greenboot
  normally handles this)
- multi-nic: start greenboot-healthcheck explicitly (RF Setup waits
  for it but it never ran as a boot-time oneshot)
- low-latency: keep local repo after install so additional RPMs like
  microshift-low-latency can be installed
- dual-stack/ipv6: add NM connectivity check after RPM install to
  avoid race with NetworkManager restart from %post scriptlet

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Remove el98/el102 duplication — one set of 11 scenario files that
derive RHEL version from SCENARIO_TYPE (rpm-presubmits-el9 or
rpm-presubmits-el10). common-scenarios-rpm.sh reads SCENARIO_TYPE
from _output/scenario_type (written by iso-build step) and sets
RPM_RHEL_VERSION, RPM_INSTALLER_IMAGE, and configure_rpm_repos()
accordingly.

22 scenario files -> 11 (one per suite).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Add download_isos action to manage_build_cache.sh that downloads only
installer ISOs from the cache. Replace inline S3 sync in
ci_phase_iso_build.sh with the new method.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
- CRITICAL: remove || echo suppression in download_isos, let failures
  propagate via run_aws_cli
- HIGH: fail explicitly when no cache access in rpm_only mode (ISOs
  are mandatory)
- HIGH: validate cache_last is non-empty before using it
- IMPORTANT: expand standard1.sh to run full suites/standard1/ not
  just version.robot
- IMPORTANT: add explicit el9 case and warning for unknown
  SCENARIO_TYPE values
- IMPORTANT: log nmcli connectivity result and warn if not 'full'

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Split the scenario lifecycle into create -> setup -> run:
- scenario_create_vms: only prepare_kickstart + launch_vm
- scenario_setup_vms: configure firewall, RHSM, repos, install MicroShift
- scenario_run_tests: only run_tests

Default scenario_setup_vms in common-scenarios-rpm.sh handles the
common case. Scenarios with extra setup (storage, multi-nic,
low-latency, upgrade) override it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Every scenario now defines scenario_setup_vms explicitly, calling
rpm_setup_and_install_microshift (or rpm_configure_vm for upgrade).
No default scenario_setup_vms in common — each scenario owns its
setup lifecycle.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
rpm_configure_vm and rpm_install_microshift are now called explicitly
in each scenario_setup_vms, making the lifecycle clearer.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
agullon added 17 commits June 30, 2026 16:57
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Single function handles repo setup, dnf install, NM connectivity
check, service start, and endpoint readiness wait.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
- upgrade: restore Y-1 rhocp repos in configure_rpm_repos for
  previous MicroShift version install
- low-latency: reboot after installing microshift-low-latency
  (tuned requires reboot, not just service restart)
- multi-nic: reboot instead of manual greenboot-healthcheck start
  (service refuses manual start, only runs at boot)
- dual-stack/ipv6: reboot after install to ensure clean network
  state (NM restart during RPM install disrupts IPv6)
- storage: wait for vg-manager and topolvm-node daemonsets in
  addition to lvms-operator deployment
- Add rpm_reboot_and_wait helper for scenarios needing clean boot

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
- storage: wait for topolvm-node daemonset to be created by the
  lvms-operator before waiting for it to be ready (it's created
  at runtime, not a static resource)
- low-latency: disable — tuned profile needs kernel-rt and specific
  configuration that RPM mode doesn't provide yet

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
topolvm-node is a legacy daemonset that's deleted during migration,
not created. The current LVMS uses vg-manager instead. Waiting for
lvms-operator + vg-manager is sufficient.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Pure k8s functional tests with no ostree/greenboot dependencies.
Also remove low-latency (needs kernel-rt + tuned config), remove
storage (reboot tests need greenboot), and add storage to periodics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
…-latency disabled

- Remove el96-prel and el96-yminus2 upgrade-ok from bootc presubmits
  (already exist in bootc periodics)
- Add low-latency as disabled to scenarios/periodics (needs kernel-rt)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Aligns with scenarios-bootc/ and scenarios-rpm/ naming convention.
Updates get_scenario_type_from_path and default SCENARIO_SOURCES.

Also adds storage (partial), optional (partial), and
gitops-telemetry-clusterid RPM presubmit scenarios.
Moves all bootc el9/el10 presubmits to periodics.
Restores deleted ostree presubmits into ostree periodics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Split scenarios-bootc/el9/periodics, scenarios-bootc/el10/periodics,
and scenarios-ostree/periodics into periodics-1 and periodics-2
directories (~15 scenarios each) to reduce per-job runtime.

Also move scenario_rpm.sh to test/bin/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
microshift-gitops is not in the local build repo — it's installed
from the CDN gitops repo which needs entitlement certs configured
differently. Rename scenario to telemetry-clusterid (the tests that
pass).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
periodics-1: upgrades, rollbacks, backup/recovery, greenboot
periodics-2: functional tests (config, storage, networking, optional,
ai-model-serving, fips, osconfig, standard suites)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant