SUP-5805: Add shared plugin path, caching, and file locking for plugins when used in agent-stack-k8s

[Mykematt]

Description

Problem: In Kubernetes environments with ephemeral pods, multiple jobs starting simultaneously would redundantly download the same plugin, wasting time and bandwidth.

Solution: Implemented file locking for shared plugin storage in Kubernetes agents to prevent race conditions during plugin downloads. Added enhanced logging to show:

• When shared storage is being used
• When a job is waiting for another job to finish downloading a plugin
• When a plugin is successfully reused from cache (with commit hash)

Context

Linear: SUP-5805

Changes

Core Implementation (internal/job/plugin.go):

• Added openCachedPlugin() helper function to DRY up duplicated cached plugin handling code
• Enhanced acquirePluginLock() with logging for lock acquisition wait states
• Updated checkoutPlugin() to support shared plugin storage with file locking when BUILDKITE_PLUGINS_PATH_INCLUDES_AGENT_NAME=false

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go tool gofumpt -extra -w .)

Disclosures / Credits

I consulted Claude for potential approaches, then wrote the implementation myself

[Mykematt]

Tested with EKS cluster using EFS-backed PVC for /workspace/plugins: https://buildkite.com/olabuildkite/buildkite-kubernetes/builds/346/steps/table

Job 1: checks to ensure shared plugin path is empty
Job 2: downloads the plugin binary since workspace is empty
Job 3: could not acquire lock so waited for lock to be released; once released, saw binary already downloaded and skipped downloading
Job 4: saw the binary immediately

Fix is mainly between Jobs 2 and 3. No race condition between them, and only one of them downloaded the binary by locking the process.

[DrJosh9000]

Hi @Mykematt, thanks for this. I realise it's been a while since you raised it, but it deserves consideration.

Aside from the syscall dependency, which breaks the Windows build, I have reservations about introducing more file-based locking (the prior art for file locks here is in git mirrors, which, let's just say, is not my favourite part of the agent).

I would like to solve caching more holistically, especially for the k8s stack, and I imagine plugins would fit into whatever solution we arrive at.

I'm going to remove reviewers from this. This isn't "we're not doing this". Maybe it can be made to work well - if that's the case feel free to re-add me.