Using this repository as a submodule has a large git history

[alexcrichton]

Right now Wasmtime pulls in tests from this repository as a submodule, pinned to the prod/testsuite-base branch. This effectively means, though, that when cloning Wasmtime contributors have to clone the full history of this branch, which is quite large. This branch is added to whenever a wasm binary changes, is added, etc, and wasm binaries are all blobs so there's not the ability to share much between commits (it's a lot of big objects). I wanted to open this as an issue because while it's not the biggest of problems now I think this is going to become a scaling problem at some point.

One possible alternative would be to produce downloadable artifacts from CI or similar from this repository rather than committing all wasm binaries to a branch. Wasmtime's tests could then download and unpack this artifact to run all the tests, which would avoid the need to check binaries into Wasmtime's repo as well. This also isn't necessarily great though as fetching artifacts can be annoying, and there might be a lot of artifacts in this repo (e.g. if they were per-commit).

Overall I'm not sure how best to handle this myself, but I wanted to file an issue regardless in case others have better ideas on integration.

[saulecabrera]

This also isn't necessarily great though as fetching artifacts can be annoying, and there might be a lot of artifacts in this repo (e.g. if they were per-commit).

If the artifact packaging is not per-commit but according to a particular release cadence instead (e.g., week, month), the number of artifacts to deal with becomes more manageable, I'd think.

Is the per-commit granularity something that you find useful?

One thing that comes to mind with the approach of downloading artifacts is the local development experience; anyone wanting to locally run tests that depend on this repository might need to explicitly download the right artifacts (could be through a script or equivalent of course). Might not be a big of a deal, since anyone wanting to run tests with the current approach needs to explicitly initialiaze the submodules anyway.

[alexcrichton]

Per week/month I think is reasonable yeah, it's a bit of a tradeoff where per-commit is no longer available but I don't think that's the end of the world.

And yeah I'm not sure exactly what this would end up looking like in terms of downstream scripting. I'm hesitant to host code in this repo that's considered the only way to download the artifacts/run the suite insofar as downstream projects likely have their own requirement/flavor of doing things, so I'd ideally prefer to consider the artifact and its layout more of an API of sorts. This repo could host an example script to download/run everything but I think there inevitably needs to be some reliance on downstream projects to do some work to integrate this test suite into their project's workflows

[saulecabrera]

I'm hesitant to host code in this repo that's considered the only way to download the artifacts/run the suite insofar as downstream projects likely have their own requirement/flavor of doing things, so I'd ideally prefer to consider the artifact and its layout more of an API of sorts.

Apologies if my comment hinted that this repository would hold the canonical way of downloading the artifacts. What I was trying to get to is exactly what you're describing, each project that would want to make use of the test suite would need to define the best way to download the artifacts. I think we are aligned there.