CoInitializeEx failed crash opening dialogs on Windows when the Dispatchers.IO thread is already in a different COM apartment

[Wavesonics]

Summary

On Windows (JVM / Compose Desktop), opening any FileKit dialog intermittently crashes with java.lang.RuntimeException: CoInitializeEx failed. The crash is non-deterministic — it depends on which Dispatchers.IO thread the coroutine lands on and whether that thread was already COM-initialized in a different apartment model.

Root cause

In WindowsFilePicker.kt, useFileDialog runs on withContext(Dispatchers.IO) (line ~155) and calls initCom():

Ole32.INSTANCE
    .CoInitializeEx(null, COINIT_APARTMENTTHREADED or Ole32.COINIT_DISABLE_OLE1DDE)
    .verify("CoInitializeEx failed")

Dispatchers.IO is a shared thread pool. If the thread it picks was already initialized as MTA by other code (or a prior dialog left it in a conflicting state), CoInitializeEx returns RPC_E_CHANGED_MODE (0x80010106). .verify() treats any non-S_OK HRESULT as fatal and throws.

Two problems:

1. RPC_E_CHANGED_MODE is not a real failure — COM is usable, just in the already-established apartment. It should be tolerated, not thrown.
2. The finally block calls CoUninitialize() unconditionally, even when CoInitializeEx did not succeed. Per the Win32 docs, a caller that receives RPC_E_CHANGED_MODE must not call CoUninitialize. This over-uninitializes a thread another caller owns, which is likely why a retry on a freshly-reset thread then succeeds.

Note: #593 (merged) moved the file picker onto Dispatchers.IO to match the saver / directory pickers — which widens the surface for this on Windows, since all three dialog types now contend for shared IO threads.

Reproduction

Intermittent; more likely under an app that does concurrent IO work (so IO threads are already initialized). Calling rememberFilePickerLauncher / FileKit.openFileSaver() etc. and opening dialogs repeatedly will eventually hit it.

java.lang.RuntimeException: CoInitializeEx failed
    at io.github.vinceglb.filekit.dialogs.platform.windows.WindowsFilePicker.initCom(...)
    at io.github.vinceglb.filekit.dialogs.platform.windows.WindowsFilePicker.useFileDialog(...)

Environment

• FileKit 0.14.2 (latest)
• Compose Multiplatform 1.11.1, Kotlin 2.4.0, JVM target 21
• Windows 11

Suggested fix

In initCom(), treat S_OK, S_FALSE, and RPC_E_CHANGED_MODE as success, and track whether this call actually initialized COM so the finally only calls CoUninitialize() when it did. Alternatively, pin dialog work to a dedicated single STA thread rather than a shared Dispatchers.IO thread.

Current workaround

We catch the exception and retry up to 4× — the unconditional CoUninitialize() in the finally resets the thread's state, so a subsequent attempt succeeds.