feat(torch): generate PyTorch backend from local ATen schemas

[voltjia]

Summary

• Add pyyaml to [build-system].requires so the build can parse the generated-backend allowlist during pip install.
• Add scripts/generate_torch_ops.py, a YAML-driven generator that reads the locally installed PyTorch / torchgen packaged native_functions.yaml.
• Add scripts/torch_ops.yaml with 525 allowlisted ATen base op names for generated PyTorch backend coverage.
• Integrate generated base headers and slot-8 PyTorch backends into CMake behind WITH_TORCH=ON.
• Update scripts/generate_wrappers.py so generated base and backend files participate in Python bindings and dispatch metadata.
• Add generated-backend tests that collect generated/torch_ops_metadata.json and execute each active PyTorch backend slot.

Motivation

InfiniOps already supports multiple native/vendor backends. This PR adds a generated PyTorch C++ backend path so a large set of ATen _out operators can be exposed through the same operator/binding surface without hand-writing hundreds of base and backend files.

The generator intentionally uses the local PyTorch installation instead of downloading native_functions.yaml: enabling WITH_TORCH already requires PyTorch to be installed, and using the local schema keeps generated code matched to the PyTorch headers and libraries being compiled.

Closes N/A.

Type of Change

• feat — new feature / new operator / new platform
  N/A: fix — this is not a bug-fix-only PR.
  N/A: perf — no runtime hot-path optimization is intended.
  N/A: refactor — the primary change is new generated PyTorch backend support.
• test — adds generated-backend coverage.
  N/A: docs — this is not documentation-only.
• build / ci — integrates generation into the build.
  N/A: chore — this is not tooling-only.
  N/A: Breaking change.

Platforms Affected

• CPU (WITH_CPU)
• NVIDIA (WITH_NVIDIA)
• Iluvatar (WITH_ILUVATAR)
• MetaX (WITH_METAX)
• Cambricon (WITH_CAMBRICON)
• Moore (WITH_MOORE)
• Ascend (WITH_ASCEND)
• PyTorch C++ bindings (WITH_TORCH)
• Build system / CMake / CI
• Python bindings / user-facing API

Test Results on Supported Platforms

All supported platforms were revalidated after the review-comment fix with WITH_TORCH=ON and pytest -v; generated PyTorch backend tests were included in the collected pytest set on every platform below.

Timing columns were measured inside the test container with base image build skipped:

• build/install = pip install .[dev] --no-build-isolation, including generated source creation and native extension compilation.
• pytest = full pytest -v run.
• total = build/install + pytest; source sync and launcher overhead are excluded.

Platform	Built	pytest Result	build/install	pytest	total	Notes
NVIDIA	Yes	9206 passed, 8664 skipped, 105 warnings in 181.26s (0:03:01)	870s	186s	1056s	Full suite passed.
Iluvatar	Yes	7410 passed, 8942 skipped, 102 warnings in 157.59s (0:02:37)	630s	161s	791s	Full suite passed.
MetaX	Yes	2 failed, 8697 passed, 7653 skipped, 119 warnings in 224.25s (0:03:44)	1303s	244s	1547s	Two CPU-only generated-backend cases crashed xdist workers in the full -n 6 run. Targeted recheck of the same nodeids with -n 1 completed with 1 passed, 1 skipped.
Cambricon	Yes	5899 passed, 10069 skipped, 323 warnings in 467.15s (0:07:47)	1859s	477s	2336s	Full suite passed.
Moore	Yes	8471 passed, 7899 skipped, 120 warnings in 239.66s (0:03:59)	2070s	246s	2316s	Full suite passed.
Ascend	Yes	7406 passed, 8904 skipped, 111 warnings in 235.42s (0:03:55)	1044s	248s	1292s	Pytest summary passed with pytest_exit_code=0; outer wrapper returned 137 after the summary.

Generated source trees and verbose pytest logs were copied locally for this run. The generated metadata contained PyTorch op entries on every platform: NVIDIA/Moore/Ascend 614, MetaX/Iluvatar 612, and Cambricon 604. The small differences come from the locally installed PyTorch schema/version on each platform.

Full `pytest` output (optional)

NVIDIA: 9206 passed, 8664 skipped, 105 warnings in 181.26s (0:03:01)
Iluvatar: 7410 passed, 8942 skipped, 102 warnings in 157.59s (0:02:37)
MetaX: 2 failed, 8697 passed, 7653 skipped, 119 warnings in 224.25s (0:03:44)
  - failed: tests/test_torch_ops.py::test_op[cpu-dtype1-0.01-0.01-4x4x5632-bitwise_right_shift]
  - failed: tests/test_torch_ops.py::test_op[cpu-dtype0-1e-05-1e-05-4x4x5632-svd]
Cambricon: 5899 passed, 10069 skipped, 323 warnings in 467.15s (0:07:47)
Moore: 8471 passed, 7899 skipped, 120 warnings in 239.66s (0:03:59)
Ascend: 7406 passed, 8904 skipped, 111 warnings in 235.42s (0:03:55); outer wrapper exit code 137 after pytest summary

MetaX targeted recheck: 1 passed, 1 skipped in 15.68s for the two nodeids that crashed xdist workers in the full run.

Benchmark / Performance Impact

N/A for runtime hot paths. Generated PyTorch backends call the corresponding ATen _out implementation.

The validation run above records build/install and pytest wall times to help track generated-backend build cost. Build/install time is currently the dominant cost on every platform.

The latest validation saved verbose pytest logs and generated source trees under the local result directory ci-results/remote/pr595-reviewfix-full-20260518/. The MetaX targeted recheck log is under ci-results/remote/pr595-reviewfix-metax-recheck-20260518/. The generated source trees are also archived locally as /tmp/torch-codegen-pr595-reviewfix-generated-20260518.tar.gz.

Notes for Reviewers

• The branch was rebased onto the latest master after fix(torch): make gemm fallback portable #611 and fix(tests): run causal_softmax reference on CPU #612 merged.
• The current stack is four commits:
  • build: add PyYAML build dependency
  • feat(scripts): add YAML-driven torch op codegen
  • build(torch): integrate generated torch backend
  • test(torch): add generated backend coverage
• Slot 8 is reserved for generated PyTorch backends.
• Hand-written src/base/<op>.h files continue to shadow generated base headers.
• Optional ATen types remain hidden for now; exposing them properly is a separable follow-up.

---

Checklist

Title, Branch, and Commits

• PR title follows Conventional Commits.
• Branch name follows <type>/xxx-yyyy-zzzz — feat/torch-codegen.
• Each commit message follows Conventional Commits.
• Large PR with meaningful, well-formed, independently reviewable commits.
• No stray merge commits from master — branch is rebased cleanly on top of current master.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are scoped to PyTorch backend codegen, build integration, wrapper generation, and generated-backend tests.
• No dead code, commented-out blocks, debug prints, or ownerless TODOs were added.
• No unrelated formatting churn that would obscure the diff.
• Public API changes are intentional: generated infini::ops::<Op> classes and slot-8 PyTorch backends are part of this feature.

General Code Hygiene

• The code is self-explanatory; comments were added only where the why is non-obvious.
• Every modified or added file ends with a single trailing newline.
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages are wrapped in backticks.
• All comments and error messages are in English.
• Comments and error messages are complete sentences with terminal punctuation where applicable.

C++ Specific

N/A: no committed .h, .cc, .cuh, or .mlu files are modified by this PR.

Python Specific

• ruff format --check passed for modified Python files in the CI container.
• ruff check passed for modified Python files in the CI container.
• GitHub ruff workflow is green on the latest pushed commit.
• Comments are complete English sentences with backticked code references where applicable.
• pytest.skip messages follow existing test conventions.
• Blank-line rules around function bodies, control-flow statements, and returns were checked by ruff format --check.
• Type hints are present on new dataclasses and public helper functions where practical.

Testing

• pytest was run on every supported platform — see the platform table above.
• Generated PyTorch backend tests are included in the full-suite runs.
• New functionality has matching tests under tests/.
• Tests use pytest.mark.parametrize for generated op metadata, shapes, dtypes, and tolerances.
• Default dtype/device parameterization is reused where appropriate.
• Any vendor-specific skip/crash guard is contained in the generated-backend test harness.
• N/A: this is a feature PR rather than a bug fix, so no regression-only test is required.

Build, CI, and Tooling

• Builds from a fresh source copy with pip install .[dev] --no-build-isolation on every supported platform.
• compile_commands.json generation remains enabled through existing CMake configuration.
• WITH_TORCH auto-detection was updated to use an installed Python with torch.
• Existing CUDA-like GPU backend mutual exclusion remains in place.
• GitHub clang-format workflow is green on the latest pushed commit.
• GitHub ruff workflow is green on the latest pushed commit.
• New build dependency pyyaml is added to pyproject.toml's [build-system].requires.

Documentation

N/A: no user-facing documentation file was changed.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• No third-party code is vendored.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

[voltjia]

请 @crapromer 初审，@Ziminli 终审。

[Ziminli]

仅有小的格式问题，可以新提一个 PR 单独修改。