[feature] hdf5 mpi for load_state

[csskene]

If hdf5 is built with mpi, then files can be written using hdf5 in mpi mode. I was wondering if an option should be made available to also read files in mpi mode in load_state if available?
I guess this would involve checking the config file and then specifying driver='mpio' when opening the file?

[kburns]

If things are working correctly, I don't think this is supposed to be necessary. Reading from a joined or virtual file is supposed to be lazy, I believe, so each core should only be loading into memory its local portion of the data when it slices it from the dataset object. If this isn't working correctly we should definitely try to fix it, but I'm not sure it involves mpio. @jsoishi?

[jsoishi]

@kburns that's right. If it isn't working right, send a test case and I'll have a look.

[csskene]

@kburns and @jsoishi I'm pretty sure this is true, that each core opens the file and then reads only the portion you ask for. I don't think this requires mpio, so it is fine as it is. For some reason I thought that using mpio for reading may scale better for reading on large numbers of cores, but I cannot find any information on this when I look now. Sorry for any confusion, happy for the issue to be closed.