External functions working with file paths on Windows lack encoding conversions

[henrikt-ma]

Breaking out what remains of #3946 as a separate issue in order to get a more focused discussion and getting some progress.

As of Modelica 3.6, the language specification makes it clear that strings in Modelica shall use UTF-8 in the external C interface. On Windows, this is not the what the stdlib functions working with file paths expect, but the MSL is currently getting away with it by breaking the external C interface both in functions producing file paths and functions consuming file paths.

While it is clear to me that something needs to be done to make these functions comply with the language specification, I am not an expert on Windows development, so I cannot make a concrete proposal for which combinations of Windows-specific functions and character encoding conversions to use.

ModelicaInternal_removeFile is an example of consuming a file path:

ModelicaStandardLibrary/Modelica/Resources/C-Sources/ModelicaInternal.c

Line 497 in 2c0d22b

if ( remove(file) != 0 ) {

ModelicaInternal_readDirectory is an example of producing a file path:

ModelicaStandardLibrary/Modelica/Resources/C-Sources/ModelicaInternal.c

Line 612 in 2c0d22b

files[iFiles] = pName;

If these functions would to the proper conversions between whatever encoding Windows uses and UTF-8, vendors would also need to make corresponding corrections to loadResource, but that's outside the scope of this issue.

Could someone who knows windows please provide some references or examples of how to write the code?

[HansOlsson]

There are two ways of handling that:

• Set Windows to UTF-8 mode (newer, some part in Beta?)
• Use wide-character support in Windows, i.e., convert UTF-8 to UTF-16 and then use _wremove (etc).

[henrikt-ma]

Good! Then I am initiating an informal poll here on which approach to take:

• 🚀 Set Windows to UTF-8 mode
• 👀 Use wide-character support in Windows, i.e., use MultiByteToWideChar, WideCharToMultiByte and _wremove (etc).

(If more alternatives come up in the discussion, I'll add them to the poll.)

[HansOlsson]

Good! Then I am initiating an informal poll here on which approach to take:

• 🚀 Set Windows to UTF-8 mode
• 👀 Use wide-character support in Windows

(If more alternatives come up in the discussion, I'll add them to the poll.)

Well, setting Windows to UTF-8 mode is something that each user has to do, so not something we can rely, or some setting when building/running the binaries - and that may depend on other considerations as well; so the best we can do for that in MSL is likely to have it as a configuration option in the Makefiles - with the alternative to switch to wide-character function. We can certainly poll for which one to have as default.

[maltelenz]

Relying on user configuration of their OS to a non-default setting, which from a quick Google search might have unexpected consequences, seems unreasonable.

Using wide-character (MultiByteToWideChar, WideCharToMultiByte, and then use the w-api functions) seems the only reasonable choice from my point of view.

[henrikt-ma]

I have updated the 👀 description based on @maltelenz's comment above.

[beutlich]

Note that ModelicaMatIO already supports path encoding conversion on Win, whereas ModelicaIO does not.

[henrikt-ma]

Note that ModelicaMatIO already supports path encoding conversion on Win, whereas ModelicaIO does not.

How come this inconsistency isn't causing problems? Are the two sets of functions used with two different sets of functions producing the file paths?

[beutlich]

Note that ModelicaMatIO already supports path encoding conversion on Win, whereas ModelicaIO does not.

How come this inconsistency isn't causing problems? Are the two sets of functions used with two different sets of functions producing the file paths?

I cannot tell, maybe users avoid using Modelica.Utilities.Files functions at all or on non-ASCII paths.