Skip to content

Commit 3f53bed

Browse files
Update cloud experiment documentation (#2094)
This pull request updates the FuzzBench documentation to provide clearer and more comprehensive instructions for running and submitting experiments, especially when contributing via pull requests. The main focus is on clarifying the workflow differences, experiment naming requirements, and artifact submission process for external contributors. Documentation improvements for experiment submission: * Added a new "Submitting Experiments via Pull Requests" section to `running_an_experiment.md`, detailing the workflow, experiment name requirements, artifact submission, and log access for contributors submitting experiments through GitHub PRs. * Updated `adding_a_new_fuzzer.md` to reference the new instructions for submitting experiments via pull requests, improving discoverability and guidance for new contributors. General documentation enhancements: * Expanded the introduction to `run_experiment.py` in `running_an_experiment.md` to explain its purpose and workflow, helping users understand how experiments are started and managed in FuzzBench. --------- Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
1 parent bf0b238 commit 3f53bed

2 files changed

Lines changed: 68 additions & 0 deletions

File tree

docs/getting-started/adding_a_new_fuzzer.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,9 @@ you are adding): add an experiment request to
308308
[service/experiment-requests.yaml](https://github.com/google/fuzzbench/blob/master/service/experiment-requests.yaml).
309309
`service/experiment-requests.yaml` explains how to do this.
310310
311+
For detailed instructions on submitting experiments via pull requests, refer to the [Submitting Experiments via Pull Requests](../running-a-cloud-experiment/running_an_experiment.md) section in the Running an Experiment documentation.
312+
313+
311314
At the end of the experiment, FuzzBench will generate a report comparing your
312315
fuzzer to the latest versions of other fuzzers, so you only need to include
313316
fuzzers that you've modified in a meaningful way (i.e. fuzzers whose results are

docs/running-a-cloud-experiment/running_an_experiment.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,15 @@ FuzzBench.
4040

4141
# run_experiment.py
4242

43+
The `run_experiment.py` script is the core script used to start experiments in FuzzBench. It is also the script that the parameters passed to `gcbrun` are fed into. This script creates a dispatcher instance on Google Compute Engine which runs the experiment, including:
44+
45+
1. Building desired fuzzer-benchmark combinations.
46+
1. Starting instances to run fuzzing trials with the fuzzer-benchmark builds and stopping them when they are done.
47+
1. Measuring the coverage from these trials.
48+
1. Generating reports based on these measurements.
49+
50+
The rest of this page will assume all commands are run from the root of FuzzBench.
51+
4352
## Experiment configuration file
4453

4554
You need to create an experiment configuration yaml file.
@@ -163,3 +172,59 @@ env:
163172

164173
Currently values in `env` are only set before running the fuzzer, not before
165174
building the benchmarks or the fuzzer itself.
175+
176+
# Submitting Experiments via Pull Requests
177+
This section complements the existing instructions for running experiments locally or on your own cloud infrastructure, providing additional guidance for contributing experiments via pull requests.
178+
When contributing an experiment through a GitHub Pull Request (PR), note that the workflow and naming rules may differ from running experiments on your own infrastructure.
179+
180+
## Maintainer Workflow
181+
182+
FuzzBench maintainers may run your experiment manually using an internal tool (`/gcbrun`) instead of the `run_experiment.py` script you’d use on your own project. For example:
183+
184+
```bash
185+
/gcbrun run_experiment.py -a \
186+
--experiment-config /opt/fuzzbench/service/experiment-config.yaml \
187+
--experiment-name 2025-01-22-freetype2-mts \
188+
--fuzzers afl aflplusplus libfuzzer \
189+
--benchmarks freetype2_ftfuzzer
190+
```
191+
192+
Therefore, you **should** provide all needed artifacts (e.g., benchmarks, seed corpora) in the PR.
193+
194+
## Experiment Name Requirements
195+
196+
Your experiment name **must**:
197+
198+
- Use only lowercase letters (`a-z`), numbers (`0–9`), and hyphens (`-`).
199+
- Be ≤ 30 characters long.
200+
- Avoid underscores (`_`).
201+
202+
Invalid:
203+
```bash
204+
2025-01-22-freetype2_ftfuzzer-mutants
205+
```
206+
207+
Valid:
208+
```bash
209+
2025-01-22-freetype2-mts
210+
```
211+
212+
If the name is invalid, maintainers will rename it before running.
213+
214+
## Seeds and Custom Inputs
215+
216+
If your experiment relies on a custom seed corpus:
217+
218+
- Place it in a `.tar.gz` file.
219+
- Link to it in your PR (GitHub repo, GCS URL, etc.).
220+
- Make sure your benchmark supports loading it correctly.
221+
222+
## Logs and Debugging
223+
224+
- Fuzzer output logs are available. As outlined above.
225+
- **Cloud logs are not publicly accessible**.
226+
- To debug deeper issues, you can:
227+
- Log directly to `stdout`/`stderr` in your benchmark or fuzzer script.
228+
- Include lightweight instrumentation in the fuzzer (e.g., seed counts, coverage summaries).
229+
230+

0 commit comments

Comments
 (0)