buck2-based build system

[andreiltd]

Hi everyone 👋

I've been working on an experimental fork of this repo that replaces the python build scripts with buck2, and I'd love to get your thoughts on whether this direction could be useful for the project.

I hope this won't start a "which build system is better" debate, they're all equally bad in their own ways 😄 I ended up using buck2 simply because I already had a collection of wasm component model rules written for other stuff, and starlark's syntax felt like a good fit given that the existing project infrastructure is python.

My primary goal with introducing a build system was to improve the dx for contributors working on wasi-testsuite itself: making it easy to build, test, and iterate without manual toolchain setup. The existing Python test runner and adapter architecture already serves great as a generic framework for downstream embedders.

You can start here to get a feel for how it works:

https://github.com/andreiltd/wasi-testsuite?tab=readme-ov-file#using-buck2

Some of the highlights:

• Buck2 downloads all toolchains automatically using http_archive macro as a build step, for example runtimes, are pinned using bzl file
• C and Rust tests are compiled using Buck2's built-in polyglot cxx_binary and rust_binary rules, for example
• wasi-sdk, wasm-tools, nodejs and asc compiler are also downloaded hermetically as build step, see toolchian definition.
• Build and test on Linux, macOS, and Windows from a single workflow. Results match upstream.
• The existing python adapters and runners are reused as-is,
• I went an extra mile and wrote a simple build step ./buck2 build //tests:dist that produces a self-contained .tar.gz with manifest, prebuilt wasm binaries, configs, and fixtures. This can be used instead of prod/testsuite-base branch, after extracting, the runner can just do: ./run-tests wasi-testsuite/ -r adapters/wasmtime.py. I've used the package to create a rolling GH release of test suite.

I'm genuinely curious whether this direction would be interesting for the project. It's not finished yet, but if there is an interest I can polish it and materialize some PRs. Thanks! 🙏

cc: @saulecabrera @alexcrichton @yoshuawuyts

[alexcrichton]

I've got no experience with buck2 previously myself, but this all sounds reasonable to me! Do you have some CI configuration/runs that you can link to which showcases the builds/tests/runs/etc across windows/mac/linux? I'm curious to see what it looks like to execute buck, timings involved,e tc.

[andreiltd]

Hey Alex,

https://github.com/andreiltd/wasi-testsuite/actions/runs/23906721249

This is the latest CI run on the fork.

[alexcrichton]

Thanks! Some questions:

• Is dotslash necessary? Is buck2 something that's frequently pinned? For example is buck2 difficult to install otherwise? Or are there frequent breaking changes such that pinning is necessary?
• I see that CI is still setting up Python, downloading wasmtime manually, etc. Are these tasks that are now extraneous and/or could they be folded into Buck?
• Could tasks such as formatting/miscellaneous stuff be moved from CI into Buck2? For example for running it locally and reproducing the same result as CI.
• The dist action took ~1 minute, how come that took so long?
• Does Buck2 have a concept of a built-in unit test system, and if so is this using that? For example are Python scripts still used or is it possible to have Buck2 print nice summaries and such for running tests, which ones failed, running with filters, etc?
• For the "build and test dist archive" step, it looks like we've still got the nromal Python runner for tests? Is that still needed and/or is that mistaken?

[andreiltd]

Is dotslash necessary? Is buck2 something that's frequently pinned? For example is buck2 difficult to install otherwise? Or are there frequent breaking changes such that pinning is necessary

Not strictly required but it's just convenient for few reasons: buck2 currently does not have a stable release tag and downloading a bi-monthly version or the latest built binary is recommended. It is also possible to build from source but it just takes long time. I haven't seen breaking changes in Starlark API so far, so I was using dotslash for all the other reasons. It's also worth to note that on unix you don't need to call dostslash ./buck2 explicitly, ./buck2 is just enough and dispatching to dostlash is handled by shebang. This only exists in CI to not introduce windows specific code.

I see that CI is still setting up Python, downloading wasmtime manually, etc. Are these tasks that are now extraneous and/or could they be folded into Buck

The wasmtime/python download in CI is only for the "test dist archive" smoke test: that step simulates what a consumer would do with the dist tarball outside our build system (extract tarball, run tests with system wasmtime). The ./buck2 test //tests:wasmtime etc uses hermetic wasmtime.

Could tasks such as formatting/miscellaneous stuff be moved from CI into Buck2? For example for running it locally and reproducing the same result as CI.

Yes, it's possible, some of those like code formatting are trivial rules. Python liniting/formatting (mypy, pylint, flake8) etc could be written as simple rules that call a system installed tools. Handling the same rules hermetically would be a bit more involved because it requires pip-installed packages.

The dist action took ~1 minute, how come that took so long?

That's a bug, I have fix for that locally. Long story short is that conformance rule was applying wasm_transition to component targets that already handle their own transition internally, causing them to be built under a different platform hash.

Does Buck2 have a concept of a built-in unit test system, and if so is this using that? For example are Python scripts still used or is it possible to have Buck2 print nice summaries and such for running tests, which ones failed, running with filters, etc?

Buck2 has ExternalRunnerTestInfo which is what our conformance tests use. It runs each test as a subprocess and reports pass/fail per target. Buck2 shows a summary like:

 Tests finished: Pass 104. Fail 0. Fatal 0. Skip 0. Omit 0.`

or Individual test (stdout, stderr comes from python test runner)

✓ Pass: root//tests:wasmtime_rust-wasip3_filesystem-mkdir-rmdir (0.4s)
---- STDOUT ----
Running test suite test with wasmtime 43.0.0
Test filesystem-mkdir-rmdir_component passed

===== Test results =====
wasmtime 43.0.0: PASS: 1 tests passed

---- STDERR ----

It shows individual failures with stdout/stderr. You can filter with buck2 test //tests:wasmtime_rust-wasip1 by suite or run a single test, or regex/wildcard: https://buck2.build/docs/users/commands/test/

For the "build and test dist archive" step, it looks like we've still got the nromal Python runner for tests? Is that still needed and/or is that mistaken?
Yes, the dist test step intentionally uses the standalone ./run-tests CLI (not buck2) to verify that the tarball works for consumers who don't have buck2.

If you have any suggestions about the workflow or have a strong opinion that other build system would be a better fit, I'm happy to explore!

[alexcrichton]

Not strictly required but it's just convenient for few reasons: buck2 currently does not have a stable release tag and downloading a bi-monthly version or the latest built binary is recommended.

Ok thanks yeah this definitely answers my question. Effectively this helps manage the lack of published releases. It's a bit annoying to ask contributors to install a dotslash utility to work with the repo, but not the end of the world.

Yes, the dist test step intentionally uses the standalone ./run-tests CLI (not buck2) to verify that the tarball works for consumers who don't have buck2.

Personally for this and the "test dist archive" step I'd advocate for basically skipping it outright. I feel it'll be too much of a drag to maintain both a build system and a custom python script to do something similar-but-not-as-complete as the build system. This does mean that the final artifact isn't tested per-se, but that may also be validatable via a Buck2 rule which performs some "simple" manual testing which could be relatively easily seen to translate to an external script of some form.

If you have any suggestions about the workflow or have a strong opinion that other build system would be a better fit, I'm happy to explore!

I'm mostly trying to vicariously through you feel out buck2 myself as I've got no experience with it. It sounds like it solves all the problems I'd imagine coming up, however, and everything looks pretty reasonable to me. Ideally in my opinion a transition to buck2 would remove the need for scattered python scripts throughout the report (as they'd otherwise be a bit of a pain to maintain IMO).

One possibly-still-open question in my mind is integration into downstream repositories for this test suite. For example if not a submodule, then what? This is a bit of a broader question than how to build this repo itself, however, and thus relatively orthogonal to a decision on buck2, so I'll table it for now to come up later if need be.

Basically tl;dr; I think this is all reasonable, but others might have thoughts too.

[yoshuawuyts]

I've talked this over with @andreiltd sync a few weeks ago, and this seems like it would remove some of the steepest barriers we have to contributing to the test suite. From both my own experience and that of others the current artifact/branch setup is not easy to get a hang of. Replacing that with a purpose-built build system would be great. And Buck2 specifically seems like it gets a lot of things right, so I'm happy that's what is being proposed here.

I'm a big fan of us trying this out and would love to see a PR put up for this.

[saulecabrera]

@andreiltd thanks for putting this together.

I have no experience with Buck2 and no strong opinions about which build system to use (we could debate all day about pros/cons IMO). However, I think the improvements to the long-term maintenance story are reasonable enough for us to try moving in this direction.

One question from my end:

The existing python adapters and runners are reused as-is

Is this the final state or are there other existing parts from the runner/adapters that you think that could be replaced with a buck2-native alternative?

I'm going to cc @wingo for any additional thoughts as well, since (if I'm not wrong!) is the author of the initial build pieces.

I'd be happy to help review any PRs.