Skip to content

Commit e0fb407

Browse files
dhalperidhalperi
authored
Add unified lab validation interface for Docker and PR testing (#81)
## Summary This PR adds a unified interface to lab-validation that supports multiple Batfish/Pybatfish sources for integration with both Docker release processes and PR testing workflows. ## Changes **Core Interface:** - Modified `ci.yml` to accept workflow inputs for different Batfish/Pybatfish sources: - Git refs (existing behavior + PR testing) - Docker images (for release artifact testing) - PyPI versions with test/prod repository distinction - Maintains full backwards compatibility with existing workflows **New Workflows:** - `reusable-lab-validation.yml` - External repositories can call this workflow - `pr-comment-lab-validation.yml` - Template for batfish repo PR comment triggers (`/test-labs` or `/test-labs eos_*`) - `docker-release-integration-example.yml` - Example integration for Docker release process **Features:** - Lab filtering support for selective testing (e.g., `eos_*` pattern) - Conditional Batfish JAR building vs Docker container usage - Proper Pybatfish version handling (requirements.txt + override logic) - Enhanced error logging for both JAR and Docker modes ## Usage Examples **Current behavior (unchanged):** ```yaml uses: batfish/lab-validation/.github/workflows/reusable-lab-validation.yml@main # Uses master branches by default ``` **Docker release validation:** ```yaml uses: batfish/lab-validation/.github/workflows/reusable-lab-validation.yml@main with: batfish_docker: "batfish/batfish:2024.12.20.1234" pybatfish_version: "1.2.3" pybatfish_pypi_repo: "test" ``` **PR testing:** ```yaml uses: batfish/lab-validation/.github/workflows/reusable-lab-validation.yml@main with: batfish_ref: "refs/pull/123/head" lab_filter: "eos_*" ``` ## Integration Plan 1. **This PR**: Adds unified interface to lab-validation (no breaking changes) 2. **Next**: Integrate into Docker release workflow using the reusable workflow 3. **Next**: Add PR comment trigger to batfish repo using the template ## Test Plan - [x] Pre-commit hooks pass - [x] Existing CI continues to work (backwards compatibility) - [ ] New reusable workflow can be called externally - [ ] Docker mode works with test containers
1 parent 1e6934b commit e0fb407

3 files changed

Lines changed: 310 additions & 6 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 71 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,40 @@
11
name: CI
22

3+
# NOTE: The lab validation job in this workflow is kept in sync with
4+
# reusable-lab-validation.yml. Changes to lab validation logic should be
5+
# applied to both files to maintain consistency.
6+
37
on:
48
push:
59
branches: [ main, develop ]
610
pull_request:
711
branches: [ main, develop ]
812
workflow_dispatch:
913
workflow_call:
14+
inputs:
15+
batfish_ref:
16+
description: "Git ref for Batfish version to test"
17+
required: false
18+
default: "master"
19+
type: string
20+
batfish_docker:
21+
description: "Docker image for Batfish (e.g., batfish/batfish:2024.12.20.1234)"
22+
required: false
23+
type: string
24+
pybatfish_ref:
25+
description: "Git ref for Pybatfish version to test"
26+
required: false
27+
default: "master"
28+
type: string
29+
pybatfish_version:
30+
description: "PyPI version for Pybatfish (e.g., 2025.7.7.2423)"
31+
required: false
32+
type: string
33+
pybatfish_pypi_repo:
34+
description: "PyPI repository (test or prod)"
35+
required: false
36+
default: "prod"
37+
type: string
1038

1139
jobs:
1240
lint:
@@ -83,16 +111,17 @@ jobs:
83111
echo "labs=$labs" >> $GITHUB_OUTPUT
84112
echo "Discovered labs: $labs"
85113
86-
# Build Batfish JAR once and cache it
114+
# Build Batfish JAR once and cache it (only if not using Docker)
87115
build-batfish:
88116
runs-on: ubuntu-latest
117+
if: inputs.batfish_docker == ''
89118
steps:
90119
- name: Checkout Batfish
91120
uses: actions/checkout@v4
92121
with:
93122
repository: batfish/batfish
94123
path: batfish
95-
ref: master
124+
ref: ${{ inputs.batfish_ref || 'master' }}
96125

97126
- name: Set up Java 17
98127
uses: actions/setup-java@v4
@@ -128,8 +157,9 @@ jobs:
128157
# Matrix job to run each lab in parallel
129158
lab-integration-test:
130159
runs-on: ubuntu-latest
131-
timeout-minutes: 20 # Reduced since we don't build JAR
160+
timeout-minutes: 20
132161
needs: [discover-labs, build-batfish]
162+
if: always() # Run even if build-batfish was skipped (for Docker mode)
133163
strategy:
134164
matrix:
135165
lab: ${{ fromJson(needs.discover-labs.outputs.labs) }}
@@ -155,6 +185,7 @@ jobs:
155185
java-version: '17'
156186

157187
- name: Download Batfish artifacts
188+
if: inputs.batfish_docker == ''
158189
uses: actions/download-artifact@v4
159190
with:
160191
name: batfish-artifacts
@@ -164,13 +195,41 @@ jobs:
164195
working-directory: lab-validation
165196
run: |
166197
python -m pip install --upgrade pip
198+
199+
# Install base dependencies from requirements.txt
167200
pip install -r requirements.txt
201+
202+
# Install Pybatfish based on input parameters (may override version from requirements.txt)
203+
if [ "${{ inputs.pybatfish_version }}" != "" ]; then
204+
if [ "${{ inputs.pybatfish_pypi_repo }}" = "test" ]; then
205+
pip install -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple pybatfish==${{ inputs.pybatfish_version }}
206+
else
207+
pip install pybatfish==${{ inputs.pybatfish_version }}
208+
fi
209+
else
210+
# Default: install from git ref (override requirements.txt version)
211+
PYBF_REF="${{ inputs.pybatfish_ref || 'master' }}"
212+
pip install git+https://github.com/batfish/pybatfish.git@${PYBF_REF}
213+
fi
214+
168215
pip install -r requirements-dev.txt
169216
pip install -e .
170217
171-
- name: Start Batfish service
218+
- name: Start Batfish service (Docker)
219+
if: inputs.batfish_docker != ''
172220
run: |
173-
# Extract questions archive
221+
# Start Batfish using Docker container
222+
docker run -d --name batfish \
223+
-p 9996:9996 \
224+
${{ inputs.batfish_docker }}
225+
226+
# Wait for service to start
227+
sleep 15
228+
229+
- name: Start Batfish service (JAR)
230+
if: inputs.batfish_docker == ''
231+
run: |
232+
# Extract questions archive and start from JAR
174233
tar -xzf questions.tgz
175234
176235
# Start Batfish using simple allinone mode (like pybatfish CI)
@@ -196,7 +255,13 @@ jobs:
196255
if: failure()
197256
run: |
198257
echo "=== Batfish logs ==="
199-
tail -100 batfish.log || echo "No batfish log found"
258+
if [ "${{ inputs.batfish_docker }}" != "" ]; then
259+
# Save docker logs to batfish.log for consistency, then display
260+
docker logs batfish > batfish.log 2>&1 || echo "No Docker container logs found"
261+
tail -100 batfish.log || echo "No batfish log found"
262+
else
263+
tail -100 batfish.log || echo "No batfish log found"
264+
fi
200265
201266
# Summary job for required status checks
202267
all-labs-passed:

‎.github/workflows/reusable-lab-validation.yml‎

Lines changed: 227 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,227 @@
1+
name: Lab Validation (reusable)
2+
3+
# NOTE: This reusable workflow is kept in sync with the lab validation job
4+
# in ci.yml. Changes to lab validation logic should be applied to both files
5+
# to maintain consistency.
6+
7+
on:
8+
workflow_call:
9+
inputs:
10+
batfish_ref:
11+
description: "Git ref for Batfish version to test"
12+
required: false
13+
default: "master"
14+
type: string
15+
batfish_docker:
16+
description: "Docker image for Batfish (e.g., batfish/batfish:2024.12.20.1234)"
17+
required: false
18+
type: string
19+
pybatfish_ref:
20+
description: "Git ref for Pybatfish version to test"
21+
required: false
22+
default: "master"
23+
type: string
24+
pybatfish_version:
25+
description: "PyPI version for Pybatfish (e.g., 2025.7.7.2423)"
26+
required: false
27+
type: string
28+
pybatfish_pypi_repo:
29+
description: "PyPI repository (test or prod)"
30+
required: false
31+
default: "prod"
32+
type: string
33+
lab_filter:
34+
description: "Optional lab filter pattern (e.g., 'eos_*')"
35+
required: false
36+
type: string
37+
38+
jobs:
39+
# First job to discover available labs
40+
discover-labs:
41+
runs-on: ubuntu-latest
42+
outputs:
43+
labs: ${{ steps.discover.outputs.labs }}
44+
steps:
45+
- name: Checkout lab-validation
46+
uses: actions/checkout@v4
47+
48+
- name: Discover available labs
49+
id: discover
50+
run: |
51+
if [ "${{ inputs.lab_filter }}" != "" ]; then
52+
# Apply filter pattern
53+
labs=$(ls -1 snapshots/ | grep -E "${{ inputs.lab_filter }}" | jq -R -s -c 'split("\n")[:-1]')
54+
else
55+
# Find all directories in snapshots/
56+
labs=$(ls -1 snapshots/ | jq -R -s -c 'split("\n")[:-1]')
57+
fi
58+
echo "labs=$labs" >> $GITHUB_OUTPUT
59+
echo "Discovered labs: $labs"
60+
61+
# Build Batfish JAR once and cache it (only if not using Docker)
62+
build-batfish:
63+
runs-on: ubuntu-latest
64+
if: inputs.batfish_docker == ''
65+
steps:
66+
- name: Checkout Batfish
67+
uses: actions/checkout@v4
68+
with:
69+
repository: batfish/batfish
70+
path: batfish
71+
ref: ${{ inputs.batfish_ref || 'master' }}
72+
73+
- name: Set up Java 17
74+
uses: actions/setup-java@v4
75+
with:
76+
distribution: 'temurin'
77+
java-version: '17'
78+
79+
- name: Cache Bazel
80+
uses: actions/cache@v3
81+
with:
82+
path: ~/.cache/bazel
83+
key: ${{ runner.os }}-bazel-17-${{ hashFiles('batfish/.bazelversion', 'batfish/WORKSPACE', 'batfish/maven_install.json') }}-${{ github.run_id }}
84+
restore-keys: |
85+
${{ runner.os }}-bazel-17-${{ hashFiles('batfish/.bazelversion', 'batfish/WORKSPACE', 'batfish/maven_install.json') }}-
86+
87+
- name: Build Batfish JAR and questions
88+
working-directory: batfish
89+
run: |
90+
bazel build //projects/allinone:allinone_main_deploy.jar
91+
cp bazel-bin/projects/allinone/allinone_main_deploy.jar allinone.jar
92+
# Create questions archive following pybatfish pattern
93+
tar -czf questions.tgz questions/
94+
95+
- name: Upload Batfish artifacts
96+
uses: actions/upload-artifact@v4
97+
with:
98+
name: batfish-artifacts
99+
path: |
100+
batfish/allinone.jar
101+
batfish/questions.tgz
102+
retention-days: 1
103+
104+
# Matrix job to run each lab in parallel
105+
lab-integration-test:
106+
runs-on: ubuntu-latest
107+
timeout-minutes: 20
108+
needs: [discover-labs, build-batfish]
109+
if: always() # Run even if build-batfish was skipped (for Docker mode)
110+
strategy:
111+
matrix:
112+
lab: ${{ fromJson(needs.discover-labs.outputs.labs) }}
113+
fail-fast: false # Don't cancel other jobs if one fails
114+
115+
steps:
116+
- name: Checkout lab-validation
117+
uses: actions/checkout@v4
118+
with:
119+
path: lab-validation
120+
121+
- name: Set up Python 3.10
122+
uses: actions/setup-python@v4
123+
with:
124+
python-version: "3.10"
125+
cache: 'pip'
126+
cache-dependency-path: 'requirements*.txt'
127+
128+
- name: Set up Java 17
129+
uses: actions/setup-java@v4
130+
with:
131+
distribution: 'temurin'
132+
java-version: '17'
133+
134+
- name: Download Batfish artifacts
135+
if: inputs.batfish_docker == ''
136+
uses: actions/download-artifact@v4
137+
with:
138+
name: batfish-artifacts
139+
path: .
140+
141+
- name: Install lab-validation dependencies
142+
working-directory: lab-validation
143+
run: |
144+
python -m pip install --upgrade pip
145+
146+
# Install base dependencies from requirements.txt
147+
pip install -r requirements.txt
148+
149+
# Install Pybatfish based on input parameters (may override version from requirements.txt)
150+
if [ "${{ inputs.pybatfish_version }}" != "" ]; then
151+
if [ "${{ inputs.pybatfish_pypi_repo }}" = "test" ]; then
152+
pip install -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple pybatfish==${{ inputs.pybatfish_version }}
153+
else
154+
pip install pybatfish==${{ inputs.pybatfish_version }}
155+
fi
156+
else
157+
# Default: install from git ref (override requirements.txt version)
158+
PYBF_REF="${{ inputs.pybatfish_ref || 'master' }}"
159+
pip install git+https://github.com/batfish/pybatfish.git@${PYBF_REF}
160+
fi
161+
162+
pip install -r requirements-dev.txt
163+
pip install -e .
164+
165+
- name: Start Batfish service (Docker)
166+
if: inputs.batfish_docker != ''
167+
run: |
168+
# Start Batfish using Docker container
169+
docker run -d --name batfish \
170+
-p 9996:9996 \
171+
${{ inputs.batfish_docker }}
172+
173+
# Wait for service to start
174+
sleep 15
175+
176+
- name: Start Batfish service (JAR)
177+
if: inputs.batfish_docker == ''
178+
run: |
179+
# Extract questions archive and start from JAR
180+
tar -xzf questions.tgz
181+
182+
# Start Batfish using simple allinone mode (like pybatfish CI)
183+
coordinator_args=(\
184+
-templatedirs=questions \
185+
-periodassignworkms=5 \
186+
)
187+
allinone_args=(\
188+
-runclient=false \
189+
-coordinatorargs="$(echo -n "${coordinator_args[@]}")" \
190+
)
191+
java -cp allinone.jar org.batfish.allinone.Main "${allinone_args[@]}" > batfish.log 2>&1 &
192+
193+
# Wait for service to start
194+
sleep 10
195+
196+
- name: Run lab integration test - ${{ matrix.lab }}
197+
working-directory: lab-validation
198+
run: |
199+
python -m pytest lab_tests/test_labs.py -v --tb=short --labname=${{ matrix.lab }}
200+
201+
- name: Show Batfish logs on failure
202+
if: failure()
203+
run: |
204+
echo "=== Batfish logs ==="
205+
if [ "${{ inputs.batfish_docker }}" != "" ]; then
206+
# Save docker logs to batfish.log for consistency, then display
207+
docker logs batfish > batfish.log 2>&1 || echo "No Docker container logs found"
208+
tail -100 batfish.log || echo "No batfish log found"
209+
else
210+
tail -100 batfish.log || echo "No batfish log found"
211+
fi
212+
213+
# Summary job for required status checks
214+
all-labs-passed:
215+
runs-on: ubuntu-latest
216+
needs: [lab-integration-test]
217+
if: always() # Run even if some lab tests failed
218+
219+
steps:
220+
- name: Check all lab tests passed
221+
run: |
222+
if [ "${{ needs.lab-integration-test.result }}" != "success" ]; then
223+
echo "❌ Some lab integration tests failed"
224+
exit 1
225+
else
226+
echo "✅ All lab integration tests passed"
227+
fi

‎.spr.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
githubRepoOwner: batfish
2+
githubRepoName: lab-validation
3+
githubHost: github.com
4+
githubRemote: origin
5+
githubBranch: main
6+
requireChecks: true
7+
requireApproval: false
8+
mergeMethod: rebase
9+
mergeQueue: false
10+
forceFetchTags: false
11+
showPrTitlesInStack: false
12+
branchPushIndividually: false

0 commit comments

Comments
 (0)