diff --git a/docs/integration_process/checklist.rst b/docs/integration_process/checklist.rst new file mode 100644 index 00000000000..3c2df6c50ba --- /dev/null +++ b/docs/integration_process/checklist.rst @@ -0,0 +1,48 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Checklist +========= + +Use this as a final review before opening your pull request: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - Step + - Done when… + * - 1 · Add module (register, regenerate, patches) + - module entry added under ``modules.target_sw`` with ``repo`` + ``hash``; + ``update_module_from_known_good.py`` ran and generated files committed; + patches referenced if needed; ``known_good_correct`` and ``bzlmod-lock`` + checks green. + * - 2 · Documentation + - ``needs_json`` added to ``//:docs`` data and toctree entry in + ``sw_components.rst``; ``//:docs_combo`` shows the module. + * - 3 · Unit tests (default platform) + - module compiles and unit tests pass on ``--config=linux-x86_64`` via + ``quality_runners.py``; ``metadata`` tuned if needed. + * - 4 · Platforms (build & test) + - ``score_pkg_bundle`` + ``*.score.json`` added and registered in + ``showcases/BUILD`` (ships into every image); ``bazel test + @score_my_module//...`` passes. + * - 5 · Component tests + - *not available yet — to be introduced.* + * - 6 · Integration tests + - tests added under ``feature_integration_tests/itf``. + * - 7 · Reporting + - Rust coverage generated; verification/status tables updated. + * - 8 · Code quality + - formatting, Bazel Clippy (Rust) and the CodeQL/MISRA C++ scan pass. diff --git a/docs/integration_process/integration_process.rst b/docs/integration_process/integration_process.rst index 2468fba0860..55a21d828af 100644 --- a/docs/integration_process/integration_process.rst +++ b/docs/integration_process/integration_process.rst @@ -12,5 +12,77 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* +################### Integration Process ################### + +This document is a step-by-step *how-to* for module owners who want to integrate +their own S-CORE module into the **Reference Integration** — whether for the +first time, when bringing it back after an exclusion, or to make it part of the +S-CORE releases in general. + +It walks you through everything from registering the module in +``known_good.json`` over wiring it into the Bazel build, the showcases, the +images and the CLI, up to adding it to the CI/CD pipelines and the consolidated +reporting (documentation, coverage and verification reports). + + +What "integration" means here +============================= + +The Reference Integration is a single `Bazel (bzlmod) `_ +build that pulls every S-CORE module together at a defined commit, builds them +against each other and runs their tests, showcases and reports. + +Integrating a module is **incremental**: each step below builds on the previous +one and *extends the reach* of your module a little further into the +integration. You climb the ladder only as far as your module needs — a pure +library may stop after it builds in-tree, while a module with a runnable example +goes all the way to running on every target platform and being exercised by +integration tests. + +The chapters that follow walk this ladder in order. Stop at whatever rung makes +sense for your module. + + +Prerequisites +============= + +Before you start, make sure that: + +* Your module lives in its own GitHub repository under the ``eclipse-score`` + organization and is a **bzlmod module**, i.e. it has a top-level + ``MODULE.bazel`` with a ``module(name = "score_")`` declaration. +* The module name follows the ``score_`` convention (e.g. + ``score_communication``). The same name is used as the Bazel + ``bazel_dep`` / repository name (``@score_//...``). +* Your module is registered in the + `S-CORE Bazel registry `_ so + that it can be resolved as a ``bazel_dep`` during the integration. +* The module builds and tests pass standalone on all four currently supported + target platforms: Linux ``x86_64``, QNX 8 ``x86_64``, Red Hat AutoSD + ``x86_64`` and EB corbos Linux (Safety Apps) ``aarch64``. +* You know the commit hash you want to pin (the integration always pins an exact + commit, never a floating branch). + + +The integration steps +====================== + +Work through the chapters in order — each one extends the reach of your module a +little further into the integration: + +.. toctree:: + :maxdepth: 2 + :numbered: + + step_1_add_module + step_2_documentation + step_3_unit_tests + step_4_platforms + step_5_component_tests + step_6_integration_tests + step_7_reporting + step_8_code_quality + checklist + reference diff --git a/docs/integration_process/reference.rst b/docs/integration_process/reference.rst new file mode 100644 index 00000000000..9b01c4653c6 --- /dev/null +++ b/docs/integration_process/reference.rst @@ -0,0 +1,148 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Reference +========= + +This section is a catalogue of *what already exists* in the integration — the +CI checks that run on every change and the consolidated reports — together with +the touch-point a new module has in each. Use it to understand where your module +shows up once it is wired in via the steps above. The supported target platforms +and their images are listed in :ref:`supported_platforms` (Step 4). + +.. _ci_checks: + +CI checks +--------- + +The pipelines live under `.github/workflows `_. Most +operate on ``//...`` and pick up your module without changes; the table says +what each check verifies and whether you typically need to touch anything. + +**Gating** marks the checks that run in the merge queue +(``merge_group``) and must therefore be green before a pull request can be +merged. The remaining checks are informational, run only on ``push``/schedule, +or gate a different event (release approvals). The definitive list of *required* +status checks is configured in the repository's GitHub branch-protection / +merge-queue settings; the column below reflects merge-queue participation +declared in the workflows. + +.. list-table:: + :header-rows: 1 + :widths: 30 36 12 22 + + * - Check (workflow) + - What it verifies + - Gating + - Your action + * - `build_and_test_qnx.yml `_ + - Builds the QNX IFS and runs integration tests on QEMU. + - **yes** + - none (graph-wide) + * - `build_and_test_autosd.yml `_ + - Builds the AutoSD OCI image with the Automotive Image Builder. + - **yes** + - none (graph-wide) + * - `build_and_test_ebclfsa.yml `_ + - Builds the EBcLfSA image and runs the high-integrity safety tests. + - **yes** + - none (graph-wide) + * - `known_good_correct.yml `_ + - Fails if the generated Bazel fragments drift from ``known_good.json``. + - **yes** + - commit step 1.2 output + * - `bzlmod-lock.yml `_ + - Verifies ``MODULE.bazel.lock`` is consistent with the module graph. + - **yes** + - update lockfile if it fails + * - `format.yml `_ + - Runs the code-formatting checks. + - **yes** + - run the format targets + * - `codeql-multiple-repo-scan.yml `_ + - Multi-repository CodeQL security scan across the integrated modules. + - **yes** + - none + * - `test_and_docs.yml `_ + - Code-quality checks, builds the docs and reports, publishes to Pages. + - **yes** + - wire docs in Step 2, reports in Step 7 + * - `check_release_approvals.yml `_ + - Enforces required approvals on PRs targeting ``releases/*`` branches. + - release branches only + - none + * - `build_and_test_linux.yml `_ + - Builds the Linux x86_64 image and runs feature integration tests on Docker. + - no + - none (graph-wide) + * - `internal_tests.yml `_ + - Runs the integration tooling tests (``//scripts/tooling:tooling_tests``). + - no + - only if you change scripts + * - `docs_cleanup.yml `_ + - Scheduled cleanup of published documentation versions. + - no + - none + * - `test_integration.yml `_ / + `reusable_smoke-test.yml `_ / + `reusable_integration-build.yml `_ + - Smoke-test / reusable build of the latest module ``main`` branches. + - no + - add runtime targets to + `ci/showcase_targets_run.txt `_ + +.. _reports: + +Reports +------- + +The consolidated outputs published by the integration. They are built by +``test_and_docs.yml`` and rendered into the documentation site. + +.. list-table:: + :header-rows: 1 + :widths: 30 40 30 + + * - Report + - Contents + - Source / target + * - Consolidated documentation + - All integrated module docs merged into one Sphinx site. + - `BUILD `_ ``docs(...)`` → ``bazel run //:docs_combo`` + * - Platform verification report + - Per-release requirements/architecture verification, safety analyses and + per-test-case results. + - `docs/verification_report/platform_verification_report.rst `_ + * - Unit test summary + - Per-module unit-test execution table (generated at build time by + ``quality_runners.py``). + - `docs/verification_report/unit_test_summary.md `_ + * - Coverage summary + - Per-module C++ and Rust coverage table (generated at build time by + ``quality_runners.py``). + - `docs/verification_report/coverage_summary.md `_ + * - C++ coverage (per module) + - lcov/genhtml line/function/branch report for modules declaring ``cpp``. + - `scripts/quality_runners.py `_ → + ``python3 scripts/quality_runners.py --modules-to-test `` + * - Rust coverage reports + - Per-module Rust line coverage (C0/C1) for modules declaring ``rust``. + - `rust_coverage/BUILD `_ → + ``bazel run //rust_coverage:rust_coverage_`` + * - Overall feature & process status + - Feature/process completion dashboard derived from the pinned module repos. + - `docs/s_core_v_1/roadmap/overall_status.rst `_ + * - Integration status dashboard + - Live build/health overview of the integration. + - `status_dashboard.html `_ diff --git a/docs/integration_process/step_1_add_module.rst b/docs/integration_process/step_1_add_module.rst new file mode 100644 index 00000000000..21f415b2659 --- /dev/null +++ b/docs/integration_process/step_1_add_module.rst @@ -0,0 +1,194 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Step 1 — Adding module to reference integration +================================================= + +.. admonition:: What it unlocks + :class: tip + + **Module in the graph** — Your module becomes a *known* part of the + integration, pinned at an exact commit, enters the **Bazel module graph** and + builds cleanly against every other module. This is the foundation every later + step builds on — afterwards ``@score_my_module//...`` resolves and any other + target can depend on your module. + +1.1 Register the module in ``known_good.json`` +---------------------------------------------- + +``known_good.json`` is the single source of truth for which modules — and at +which commit — are part of the integration. Every other generated file is +derived from it, so this is always the first change. + +Open `known_good.json `_. It has two top-level groups +under ``modules``: + +* ``target_sw`` — the S-CORE software modules that are built and shipped + (baselibs, communication, persistency, logging, …). **Your module goes here.** +* ``tooling`` — supporting tooling repos (docs-as-code, process, platform, …). + +Add an entry for your module under ``modules.target_sw``: + +.. code-block:: json + + "score_my_module": { + "repo": "https://github.com/eclipse-score/my_module.git", + "hash": "0000000000000000000000000000000000000000", + "metadata": { + "code_root_path": "//src/...", + "langs": ["rust"], + "extra_test_config": [], + "exclude_test_targets": [], + "rust_coverage_config": "ferrocene-coverage" + } + } + +Field reference +~~~~~~~~~~~~~~~ + +``repo`` (required) + HTTPS clone URL of the module repository. + +``hash`` (required, mutually exclusive with ``version``) + Exact commit to pin. Use ``version`` instead if the module is published to a + Bazel registry (this emits a ``single_version_override`` rather than a + ``git_override``). + +``bazel_patches`` (optional) + List of patch labels applied on top of the checked-out commit (see + :ref:`patches`). + +``metadata.code_root_path`` (default ``//score/...``) + Bazel target pattern that points at the module's source root. Used to scope + test discovery and Rust coverage queries (e.g. ``//src/...``). + +``metadata.langs`` (default ``["cpp", "rust"]``) + Languages used by the module. ``rust`` enables a generated Rust coverage + report (see :ref:`reporting`). + +``metadata.extra_test_config`` + Extra ``--//flag=value`` build settings injected when building/testing this + module (feature flags, config selections, …). + +``metadata.exclude_test_targets`` + Test targets that must **not** run in the integration (flaky, environment + specific, or not meaningful out-of-tree). Supports wildcards such as + ``//score/json/examples:*``. + +``metadata.rust_coverage_config`` (default ``ferrocene-coverage``) + The Bazel coverage config to use for the generated Rust coverage report. + +``pin_version`` (optional, default ``false``) + When ``true``, the automatic "update to latest HEAD" scripts will leave this + module's hash untouched. + +.. note:: + + To bump an existing entry to the latest commit on a branch you can use + ``scripts/known_good/update_module_latest.py`` instead of editing the hash by + hand. The first integration, however, is added manually. + + +1.2 Regenerate the Bazel module fragments +----------------------------------------- + + Your module now enters the **Bazel module graph**. From now on + ``@score_my_module//...`` resolves and any other target — showcases, docs, + other modules — can depend on it. (If it declares ``rust``, its Rust coverage + target is generated here too.) + +The Bazel files are **generated** from ``known_good.json`` — never edit them by +hand. After editing the JSON, regenerate them: + +.. code-block:: bash + + python3 scripts/known_good/update_module_from_known_good.py + +This (re)writes: + +* `bazel_common/score_modules_target_sw.MODULE.bazel `_ + — a ``bazel_dep`` + ``git_override`` block for your module. This file is + pulled into the build via ``include(...)`` in + `MODULE.bazel `_. +* `rust_coverage/BUILD `_ — a + ``rust_coverage_report`` target for every module that declares ``rust`` in its + ``langs`` (skipped for C++-only modules). + +Next, refresh the Bazel lockfile so it reflects the updated module graph: + +.. code-block:: bash + + bazel mod deps --lockfile_mode=update + +Adding your module introduces a new ``bazel_dep`` / ``git_override`` into the +module graph, which changes the set of resolved dependencies. Running ``bazel +mod deps`` with ``--lockfile_mode=update`` re-resolves the graph and writes the +new dependencies into `MODULE.bazel.lock `_. Committing +an up-to-date lockfile keeps the dependency resolution reproducible for everyone +and is required by the ``Bzlmod Lockfile Check`` CI job (see section 1.4), which +fails if the committed lockfile is out of sync with the module graph. + +Commit the regenerated files together with the ``known_good.json`` change. CI +verifies that the generated files are in sync with the JSON (see the +``known_good_correct`` workflow), so a manual edit that drifts from the JSON +will fail. + +.. _patches: + +1.3 (optional) Add Bazel patches +-------------------------------- + +If the module does not build cleanly out-of-tree (e.g. label visibility issues), +add patch files under ``patches//`` and reference them from the +``bazel_patches`` list of the module entry in ``known_good.json``. See +`patches/communication `_ for a worked example. The +generator emits ``patches = [...]`` with ``patch_strip = 1`` automatically. + +.. note:: + + Patching a module should be the **exception, not the rule**. A patch keeps a + local divergence from the upstream module that has to be maintained and + re-based on every update. Prefer fixing the underlying issue in the module's + own repository so it builds cleanly out-of-tree, and only reach for a patch + as a temporary, last-resort workaround. + +1.4 CI checks that the module was added correctly +------------------------------------------------- + +Once the changes above are pushed, two **gating** CI jobs verify that the module +was wired in consistently. They do not build your module's code — they only +check that the generated metadata is in sync with ``known_good.json``: + +.. list-table:: + :header-rows: 1 + :widths: 28 48 24 + + * - Check (workflow) + - What it verifies + - Fix if it fails + * - Known Good Matches Bazel + (`known_good_correct.yml `_) + - Re-runs ``update_module_from_known_good.py`` and fails if the generated + Bazel fragments differ from what you committed — i.e. ``known_good.json`` + and the generated ``*.MODULE.bazel`` / ``rust_coverage/BUILD`` are out of + sync. + - re-run step 1.2 and commit the regenerated files + * - Bzlmod Lockfile Check + (`bzlmod-lock.yml `_) + - Verifies ``MODULE.bazel.lock`` is consistent with the updated module + graph (your new ``bazel_dep`` / ``git_override``). + - refresh and commit ``MODULE.bazel.lock`` + +Both jobs run in the merge queue, so they must be green before the pull request +can be merged. The full catalogue of checks is under :ref:`ci_checks`. diff --git a/docs/integration_process/step_2_documentation.rst b/docs/integration_process/step_2_documentation.rst new file mode 100644 index 00000000000..3584547f1fc --- /dev/null +++ b/docs/integration_process/step_2_documentation.rst @@ -0,0 +1,135 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Step 2 — Join the documentation and process-compliance checks +============================================================= + +.. admonition:: What it unlocks + :class: tip + + **Docs & process checks** — Your module's documentation appears in the + single, combined docs site that is built and published for the whole + integration — and, just as importantly, your module's **requirements, + architecture and other process artifacts are validated against the S-CORE + process** as part of the same build. This is usually the first thing you + *extend the integration with* after the module is in the graph. + +Building the docs is not only about rendering pages. The integration uses the +docs-as-code toolchain (`score_docs_as_code +`_, pulled in via +``score_sphinx_bundle`` in `docs/conf.py `_), which runs the +**S-CORE metamodel checks** on every ``needs`` object: requirements, +architecture elements, safety artifacts and their links must conform to the +process model defined in `score_process +`_. If your module's +requirements/architecture are malformed, unlinked or violate the metamodel, the +docs build (and therefore CI) fails — so wiring your module in here is what gets +its process artifacts continuously checked, not just published. + +The integration builds one Sphinx site that merges the docs of every integrated +module. To pull yours in: + +#. Add your module's ``needs_json`` to the ``docs(...)`` rule's ``data`` list in + the top-level `BUILD `_ file: + + .. code-block:: python + + docs( + data = [ + "@score_persistency//:needs_json", + # ... + "@score_my_module//:needs_json", + ], + known_good = "known_good.json", + source_dir = "docs", + ) + + This requires your module to expose a ``//:needs_json`` target (i.e. it is a + docs-as-code module). + +#. Add a toctree entry so the module shows up in the navigation. **Which page + you edit depends on what kind of module it is** — the site groups modules + into two top-level sections, each backed by its own ``.rst`` page: + + .. list-table:: + :header-rows: 1 + :widths: 22 30 48 + + * - Section + - Page to edit + - Put your module here if it is… + * - **Modules** + - `docs/sw_components.rst `_ + - an S-CORE **software module** that ships in the integration — i.e. it + lives under ``modules.target_sw`` in ``known_good.json`` (communication, + persistency, logging, orchestrator, baselibs, …). This is the common + case. + * - **Process, Methods & Tools** + - `docs/process_methods_tools.rst `_ + - a **tooling / process** repo from ``modules.tooling`` (platform, process, + docs-as-code) rather than a shipped software module. + + For a normal software module, add the entry to the ``Modules`` toctree in + `docs/sw_components.rst `_, next to the existing + modules: + + .. code-block:: rst + + Modules + ======= + + .. toctree:: + :titlesonly: + :maxdepth: 1 + + _collections/score_persistency/docs/index + _collections/score_logging/docs/index + # ... + _collections/score_my_module/docs/index + + The ``_collections/score_my_module/docs/index`` path is **not** a file in this + repository — the docs build mounts every module's documentation under + ``_collections//`` at build time. So the path must be: + + * ``score_my_module`` — the **module/repository name** as registered in + ``known_good.json`` (the ``@score_my_module`` repo), and + * ``docs/index`` — the path of the **documentation root** *inside that + module's repository* (most modules use ``docs/index``; match whatever your + module actually exposes via its ``needs_json`` docs target). + +Build the full docs locally to verify your module shows up — or use the +live-preview server which rebuilds on every change: + +.. code-block:: bash + + # one-shot full build incl. all modules + bazel run //:docs_combo + + # live preview in the browser (auto-rebuild) + bazel run //:live_preview_combo_experimental + +The docs are built and published by the ``test_and_docs`` workflow (see +:ref:`ci_checks`). + +.. note:: + + **Planned refactoring.** Today documentation generation is entangled with + unit tests, coverage and feature integration tests in the single + ``test_and_docs`` workflow + (`test_and_docs.yml `_). This + should be refactored: the documentation build (including the metamodel / + process-compliance checks) belongs in a **dedicated, reusable workflow shared + across all S-CORE repositories**, hosted centrally in the + `cicd-workflows `_ + repository, instead of being duplicated and maintained per repo. diff --git a/docs/integration_process/step_3_unit_tests.rst b/docs/integration_process/step_3_unit_tests.rst new file mode 100644 index 00000000000..149fcc31db5 --- /dev/null +++ b/docs/integration_process/step_3_unit_tests.rst @@ -0,0 +1,88 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Step 3 — Get compiled and unit-tested on the default platform +============================================================= + +.. admonition:: What it unlocks + :class: tip + + **Default-platform unit tests** — Every module registered in + ``known_good.json`` is automatically **compiled and run through its unit + tests** on the default platform (Linux ``x86_64``, ``--config=linux-x86_64``) + — no per-module wiring is needed. As soon as your module is in the graph + (Step 1), its unit tests are part of the integration's quality gate. + +Unit tests are driven by +`scripts/quality_runners.py `_, which loads +``known_good.json`` and iterates over **every** module under +``modules.target_sw``. For each module it runs ``bazel coverage`` with the +``unit-tests`` config, which builds and tests the module's source tree on the +default Linux ``x86_64`` platform: + +.. code-block:: bash + + # what the runner effectively executes per module + bazel coverage --config=unit-tests \ + --instrumentation_filter=@score_my_module \ + @score_my_module + + # reproduce locally for a single module + python3 scripts/quality_runners.py --modules-to-test score_my_module + +``--config=unit-tests`` expands to ``--config=linux-x86_64`` (plus +``--build_tests_only`` and a ``-manual`` tag filter), so this is the **default +platform** baseline that every module must pass — independent of the +target-platform images in Step 4. + +What you can tune from ``known_good.json`` +------------------------------------------ + +You do not edit ``quality_runners.py`` to control what is tested — the per-module +``metadata`` in ``known_good.json`` (Step 1) drives it: + +* ``metadata.code_root_path`` — the target pattern that is built and tested + (``@score_my_module``), e.g. ``//src/...``. +* ``metadata.extra_test_config`` — extra ``--flag=value`` build/test settings + injected for this module (feature flags, config selections). +* ``metadata.exclude_test_targets`` — test targets that must **not** run + (flaky, environment-specific or not meaningful out-of-tree). You list plain + module-relative labels such as ``//src/cpp/tests:bm_kvs_cpp``; the runner + automatically turns each into a ``-@score_my_module`` exclusion. + Supports wildcards such as ``//score/json/examples:*``. +* ``metadata.rust_coverage_config`` (default ``ferrocene-coverage``) — selects + the Rust coverage configuration applied when running ``bazel coverage`` for + this module (e.g. ``ferrocene-coverage-per``). + +So enrolling your module into unit testing requires no extra step beyond Step 1 +— you only adjust these fields when the defaults do not fit. + +The unit-test results and the code-coverage numbers produced here are exported +into the consolidated reports (markdown summaries on the docs site, HTML +coverage reports and the CI job summary). How and where exactly is described in +Step 7, see :ref:`reporting`. + +This runs in the ``test_and_docs`` workflow +(`test_and_docs.yml `_), in the same +job as the docs build (see :ref:`ci_checks`). + +.. note:: + + **Planned refactoring.** As with the documentation build, unit testing is + currently bundled into the monolithic ``test_and_docs`` workflow together with + coverage and feature integration tests. This should be split out into a + **dedicated, reusable unit-test workflow shared across all S-CORE + repositories**, hosted centrally in the + `cicd-workflows `_ + repository, instead of being duplicated per repo. diff --git a/docs/integration_process/step_4_platforms.rst b/docs/integration_process/step_4_platforms.rst new file mode 100644 index 00000000000..6fa7d4f598f --- /dev/null +++ b/docs/integration_process/step_4_platforms.rst @@ -0,0 +1,220 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Step 4 — Get built and tested on the target platforms +===================================================== + +.. admonition:: What it unlocks + :class: tip + + **Target-platform builds & tests** — Your module is actually **compiled and + tested by the pipelines** — built for and deployed to every target platform + (Linux x86_64, QNX, AutoSD, EBcLfSA aarch64), offered in the interactive CLI, + and exercised by CI. This is the rung that turns a library into something + that *runs* inside the integration images. + +The integration provides build definitions for four target images, each built +and tested by its own gating CI job — it does not distribute ready-to-flash +images; you build them yourself from these definitions. The table lists the +platforms, the image target, the runner used to execute it, and the per-platform +CI check. + +.. _supported_platforms: + +.. list-table:: Supported target platforms + :header-rows: 1 + :widths: 22 14 22 22 20 + + * - Platform + - Arch + - Image + - Runner + - Build & test workflow + * - Linux + - x86_64 + - `images/linux_x86_64 `_ + - Docker (`runners/docker_x86_64 `_) + - `build_and_test_linux.yml `_ + * - QNX 8 + - x86_64 + - `images/qnx_x86_64 `_ + - QEMU (`runners/qemu_x86_64 `_) + - `build_and_test_qnx.yml `_ + * - Red Hat AutoSD + - x86_64 + - `images/autosd `_ + - Docker (`runners/docker_x86_64 `_) + - `build_and_test_autosd.yml `_ + * - EB corbos Linux (Safety Apps) + - aarch64 + - `images/ebclfsa_aarch64 `_ + - QEMU (`runners/qemu_aarch64 `_) + - `build_and_test_ebclfsa.yml `_ + +.. note:: + + The EB corbos *Safety Applications* image is special: it runs a + high-integrity application demo under additional constraints. Read + `images/ebclfsa_aarch64/README.md `_ + before targeting it. + +Run any image locally in QEMU +----------------------------- + +Every image exposes a ``:run`` target that boots it in an emulator, so you can +try your module on the real target without hardware: + +.. code-block:: bash + + bazel run //images/linux_x86_64:run + bazel run //images/qnx_x86_64:run + bazel run //images/autosd:run + bazel run //images/ebclfsa_aarch64:run + +The QNX and aarch64 images boot under QEMU, which needs a one-time host setup +(TUN device, bridge helper, libvirt network). Follow the QEMU runner how-to in +`runners/qemu_x86_64/README.md `_ before the +first run. + +Two ways to get your module onto the images +------------------------------------------- + +There are two ways to make your binary part of the images: + +* **As a showcase (recommended).** Bundle your binary once into the aggregate + ``//showcases`` package; it is then deployed into **every** image + automatically — one change covers all four platforms, and the binary is also + offered in the interactive CLI. See :ref:`add_showcase` below. +* **Directly into each image.** Wire the binary into each image's build + description by hand. This is **per-image work that you repeat four times** + (once per image) and is only needed for components that are not run as a CLI + showcase — typically a daemon. The ``datarouter`` is integrated this way; see + how the QNX image does it in + `images/qnx_x86_64/build/BUILD `_ (the + ``datarouter`` source and the ``DATAROUTER_PATH`` mapping). See + :ref:`add_directly` below. Prefer the showcase route unless you have a reason + not to. + +.. _add_showcase: + +Add a runtime showcase +---------------------- +A *showcase* is the vehicle that gets your module onto the platforms: you bundle +a runnable binary, and the aggregate ``//showcases`` package is deployed into +**every** image automatically. You do this once — there is no per-platform work. + +There are two patterns under `showcases/ `_: + +* **Standalone** (`showcases/standalone `_) — bundle + a binary that already exists in your module's repo. +* **Composed** (e.g. + `showcases/orchestration_persistency `_) + — a small binary defined *here* that combines several modules. + +**1. Define the bundle.** Add a ``score_pkg_bundle`` target. To re-use a binary +that lives in your module: + +.. code-block:: python + + load("//bazel_common:bundlers.bzl", "score_pkg_bundle") + + score_pkg_bundle( + name = "my_module", + bins = ["@score_my_module//examples:my_example"], + config_data = ["//showcases/standalone:my_module.score.json"], + package_dir = "standalone", + ) + +**2. Add a CLI descriptor.** Create a ``*.score.json`` next to the bundle. The +CLI auto-discovers every ``*.score.json`` deployed under ``/showcases`` and +offers it in the interactive menu, so **no code change to the CLI is required**: + +.. code-block:: json + + { + "name": "My Module Example", + "description": "Demonstrates my_module at runtime", + "apps": [ + { + "path": "/showcases/bin/my_example", + "args": [], + "env": {}, + "dir": "/showcases/data/my_module" + } + ] + } + +**3. Register the bundle in the top-level showcase package.** Add your bundle to +the aggregate ``score_pkg_bundle`` in +`showcases/BUILD `_ so it is packaged into the images: + +.. code-block:: python + + score_pkg_bundle( + name = "showcases", + bins = ["//showcases/cli"], + other_package_files = [ + "//showcases/standalone:comm_pkg_files", + # ... + "//showcases/standalone:my_module_pkg_files", + ], + package_dir = "showcases", + ) + +Once your bundle is part of ``//showcases``, it is reachable from +``//images/:image`` and is therefore **built by every platform CI +job** automatically. + +.. _add_directly: + +Add a binary directly to an image +--------------------------------- + +If your component is not a CLI showcase — for example a background daemon that +must always be present — you wire it straight into each image's build +description instead. Unlike the showcase route, **this is per-image work: you +repeat it for every image you target** (Linux, QNX, AutoSD, EBcLfSA). + +The reference example is the ``datarouter``. In the QNX image it is added as a +source of the image target and exposed to the image's build description via a +location mapping, then placed into the filesystem by ``system.build``: + +.. code-block:: python + + # images/qnx_x86_64/build/BUILD + qnx_ifs( + name = "init", + srcs = [ + # ... + "//showcases", + "@score_logging//score/datarouter", + "//feature_integration_tests/configs/datarouter:etc_configs", + ], + ext_repo_maping = { + "BUNDLE_PATH": "$(location //showcases:showcases)", + "DATAROUTER_PATH": "$(location @score_logging//score/datarouter:datarouter)", + }, + ) + +See `images/qnx_x86_64/build/BUILD `_ and the +matching deployment lines in +`images/qnx_x86_64/build/system.build `_ +for the full picture, and repeat the equivalent wiring in the other images you +need (`images/linux_x86_64 `_, +`images/autosd `_, +`images/ebclfsa_aarch64 `_). + +If your module simply cannot run on a given platform, exclude its targets via +``metadata.exclude_test_targets`` / ``metadata.extra_test_config`` in +``known_good.json`` (Step 1) rather than editing the workflow. diff --git a/docs/integration_process/step_5_component_tests.rst b/docs/integration_process/step_5_component_tests.rst new file mode 100644 index 00000000000..24a8ef8e845 --- /dev/null +++ b/docs/integration_process/step_5_component_tests.rst @@ -0,0 +1,32 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Step 5 — Add component tests +============================ + +.. admonition:: What it unlocks + :class: tip + + **Component-level verification** — Your module is verified **in isolation as + a component** — its public interfaces are tested against their specification, + independently of the other modules and of the full integration images. + +.. warning:: + + **Currently missing — needs to be introduced.** There is no component-test + stage in the Reference Integration yet. Component tests (testing a single + module's components/interfaces in isolation, between unit tests in Step 3 and + the runtime feature integration tests in Step 6) still need to be designed and + wired into the integration and its CI. This step is a placeholder so the + intended test pyramid is documented; update it once the mechanism exists. diff --git a/docs/integration_process/step_6_integration_tests.rst b/docs/integration_process/step_6_integration_tests.rst new file mode 100644 index 00000000000..01f9835486c --- /dev/null +++ b/docs/integration_process/step_6_integration_tests.rst @@ -0,0 +1,81 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Step 6 — Add feature integration tests +====================================== + +.. admonition:: What it unlocks + :class: tip + + **Runtime feature tests** — Your showcase is **exercised at runtime** on the + deployed images, not just built. This is the difference between "it ships" + and "it provably works on the target". + +Two test frameworks +------------------- + +The integration currently uses **two** separate frameworks for integration +tests, each driven by its own Bazel target and run in a different CI workflow: + +* **FIT — scenario-based feature integration tests** + (`feature_integration_tests/test_cases `_, + target ``//feature_integration_tests/test_cases:fit``). Python/Pytest test + cases that orchestrate Rust and C++ scenarios. They are executed as part of the + ``test_and_docs`` workflow on the default Linux platform: + + .. code-block:: bash + + bazel test --config=linux-x86_64 //feature_integration_tests/test_cases:fit + +* **ITF — Integration Test Framework** + (`feature_integration_tests/itf `_, + target ``//feature_integration_tests/itf``). Black-box tests that run against a + booted image. They are executed for **two** images only — QNX (on QEMU) by + `build_and_test_qnx.yml `_ and + Linux x86_64 by + `build_and_test_linux.yml `_: + + .. code-block:: bash + + # QNX image, booted under QEMU + bazel test --config=itf-qnx-x86_64 //feature_integration_tests/itf + # Linux x86_64 image + bazel test --config=linux-x86_64 //feature_integration_tests/itf + +Add black-box tests that exercise your showcase on a running target under +`feature_integration_tests/itf `_. They run +against the deployed image and assert that the binary is present and behaves as +expected: + +.. code-block:: python + + def test_my_example_is_deployed(target): + exit_code, _ = target.execute("test -f /showcases/bin/my_example") + assert exit_code == 0 + + def test_my_example_runs(target): + exit_code, out = target.execute("/showcases/bin/my_example") + assert exit_code == 0 + +The existing ``test_run_all_showcases`` test invokes ``cli --examples=all``, so +any showcase you registered in Step 4 is already covered by the end-to-end run. + +.. note:: + + **Planned consolidation.** Having two integration-test frameworks (FIT and + ITF) executed by different workflows on different platforms is a known gap. + The execution should be consolidated so that **all integration tests — both + FIT and ITF — run against all four target images** (Linux x86_64, QNX, AutoSD, + EBcLfSA aarch64), instead of FIT running only in ``test_and_docs`` on Linux + and ITF running only for the QNX and Linux images. diff --git a/docs/integration_process/step_7_reporting.rst b/docs/integration_process/step_7_reporting.rst new file mode 100644 index 00000000000..f41ca81aee1 --- /dev/null +++ b/docs/integration_process/step_7_reporting.rst @@ -0,0 +1,130 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. _reporting: + +Step 7 — Add coverage and verification reports +============================================== + +.. admonition:: What it unlocks + :class: tip + + **Consolidated reports** — Your module shows up in the **consolidated + coverage and verification / status reports** that are published alongside the + docs — the evidence side of the integration. + +.. note:: + + **No extra action is required here.** As already explained in + :doc:`Step 3 `, the ``test_and_docs`` workflow already + executes your module's unit tests and gathers and publishes the code-coverage + data. If your module passes Step 3, it is automatically part of the reports + described below — this chapter only explains *how* that collection, + aggregation and publishing happens. + +How coverage is collected +------------------------- + +The ``test_and_docs`` workflow runs +`scripts/quality_runners.py `_, which iterates +over **every** module under ``modules.target_sw`` in ``known_good.json`` and, per +module, runs its unit tests with coverage (``bazel coverage``) and then extracts +a per-language summary. The driver is the module's ``metadata.langs``: + +* ``"cpp"`` → a C++ (lcov/genhtml) coverage run. +* ``"rust"`` → a Rust coverage run. + +Both write into the two consolidated reports +`docs/verification_report/unit_test_summary.md `_ +and +`docs/verification_report/coverage_summary.md `_. +The same scoping metadata applies to both languages: + +* ``metadata.code_root_path`` — the target pattern that is instrumented and + tested (``@score_my_module``). +* ``metadata.extra_test_config`` — extra ``--flag`` settings for the coverage run. +* ``metadata.exclude_test_targets`` — targets excluded from the run. + +So for most modules **no extra wiring is needed** — declaring the language in +``metadata.langs`` (Step 1) is what enrolls the module into coverage. + +Where the results are exported +------------------------------ + +``quality_runners.py`` does not only run the tests and coverage — it parses the +output per module and **exports the results** in three forms: + +* **Markdown summaries committed into the docs.** A unit-test execution table + and a coverage table are written to + `docs/verification_report/unit_test_summary.md `_ + (columns: module, passed, failed, skipped, total) and + `docs/verification_report/coverage_summary.md `_ + (columns: module, lines, functions, branches). These two files are pulled into + the published documentation by + `docs/verification_report/platform_verification_report.rst `_, + so every module's unit-test and coverage numbers appear in the + **Platform Verification Report** on the docs site. +* **Detailed HTML coverage reports.** The per-module ``genhtml`` / Rust coverage + HTML is written under the coverage output directory + (``artifacts/coverage/cpp/`` and ``artifacts/coverage/rust/`` + by default; override with ``--coverage-output-dir``). +* **CI job summary.** In CI the ``test_and_docs`` workflow additionally appends + the two markdown summaries to the GitHub Actions **job summary** + (``$GITHUB_STEP_SUMMARY``), so the pass/fail and coverage tables are visible + directly on the workflow run. + +C++ coverage +------------ + +If your module declares ``cpp`` in ``metadata.langs``, ``quality_runners.py`` +runs ``bazel coverage`` over ``@score_my_module`` and produces an +HTML/lcov report via ``genhtml`` (the ``lcov`` package must be present — CI +installs it; locally ``sudo apt-get install -y lcov``). The line/function/branch +numbers land in ``coverage_summary.md``. To make your module's C++ coverage +meaningful: + +#. Ensure ``metadata.langs`` contains ``"cpp"`` and ``metadata.code_root_path`` + points at the instrumented source root. +#. Use ``metadata.exclude_test_targets`` to drop tests that should not count + (e.g. examples, flaky or environment-specific targets). +#. Reproduce a single module's run locally with: + + .. code-block:: bash + + python3 scripts/quality_runners.py --modules-to-test score_my_module + +Rust coverage +------------- + +If your module declares ``rust`` in ``metadata.langs``, the +``rust_coverage_report`` target is generated automatically into +`rust_coverage/BUILD `_ by step 1.2 — no manual action +is needed. Tune ``metadata.exclude_test_targets`` and +``metadata.rust_coverage_config`` in ``known_good.json`` to control what is +measured. Build a single module's report with: + +.. code-block:: bash + + bazel run //rust_coverage:rust_coverage_score_my_module + +Verification & status reports +----------------------------- + +The consolidated reports live under +`docs/verification_report `_ and the roadmap / +status trackers under +`docs/s_core_v_1/roadmap `_ (e.g. +``overall_status.rst``). Add your module to the relevant tables/rows so it shows +up in the platform verification report and the feature/process status overview. +The full catalogue of reports is under :ref:`reports`. diff --git a/docs/integration_process/step_8_code_quality.rst b/docs/integration_process/step_8_code_quality.rst new file mode 100644 index 00000000000..667b01205fb --- /dev/null +++ b/docs/integration_process/step_8_code_quality.rst @@ -0,0 +1,67 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +Step 8 — Code quality +===================== + +.. admonition:: What it unlocks + :class: tip + + **Static analysis & style checks** — Your module's source is **statically + analysed and style-checked** on every pull request, on top of the unit, + component and integration tests of the previous steps. + +The integration runs several language-specific code-quality checks. They operate +on ``//...`` (or on the pinned module checkouts), so they pick up your module +automatically once it is in the graph — there is nothing module-specific to wire +in. + +What runs today +--------------- + +* **Formatting checks** (all languages) — + `format.yml `_ runs the shared S-CORE + formatter (a reusable workflow from + `cicd-workflows `_) and fails + if any file is not formatted. Reproduce / fix locally by running the format + targets before pushing. +* **Bazel Clippy** (Rust) — Rust static analysis runs through the ``rules_rust`` + Clippy aspect over the Rust targets, flagging lint violations as build errors. +* **CodeQL multi-repo scan** (C++) — + `codeql-multiple-repo-scan.yml `_ + checks out every module pinned in ``known_good.json`` and runs CodeQL with the + **MISRA C++ coding standards** pack across all of them. + +.. note:: + + **Missing — to be introduced.** There is no **clang-tidy** stage for C++ yet. + clang-tidy is planned to complement the CodeQL / MISRA scan for C++ static + analysis, but it is currently not wired into the integration or its CI. Until + then, C++ static analysis relies on the CodeQL scan only. + +Generated reports +----------------- + +The CodeQL scan exports its findings as build artifacts on the workflow run: + +* a **SARIF** result file (``sarif-results/cpp.sarif``), uploaded as the + ``codeql-sarif-results`` artifact, and +* a human-readable **HTML report** (``codeql-report.html``), generated from the + SARIF with ``sarif-tools`` and uploaded as the ``codeql-html-report`` + artifact. + +Download these from the *Artifacts* section of the CodeQL workflow run to review +the C++ findings. The formatting and Clippy checks do not produce a separate +report — they pass or fail directly on the workflow run, with the offending +files / lints shown in the job log.