It is currently October 7th, 2022, 10:18 am

[Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Report bugs with the Rainmeter application and suggest features.
User avatar
tass_co
Posts: 379
Joined: May 4th, 2020, 3:01 pm
Location: Ankara, TURKEY

[Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by tass_co »

Hello everyone.
Can't we put an end to this mess?
We encounter errors while writing skin code.
Although we are sure of our code, the skin is not working properly.
You are trying everything but the result is negative.
You're going crazy.
Then you realize that your code is not working because of this ridiculous situation(of course if you can).
It's really frustrating...
We encounter such errors a lot on the forum.
Is there anything we can do about this issue?
I don't know where i going from here, but i promise it won't be boring... :great:
User avatar
SilverAzide
Rainmeter Sage
Posts: 2064
Joined: March 23rd, 2015, 5:26 pm

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by SilverAzide »

tass_co wrote: August 10th, 2022, 10:46 pm Hello everyone.
Can't we put an end to this mess?
We encounter errors while writing skin code.
Although we are sure of our code, the skin is not working properly.
You are trying everything but the result is negative.
You're going crazy.
Then you realize that your code is not working because of this ridiculous situation(of course if you can).
It's really frustrating...
We encounter such errors a lot on the forum.
Is there anything we can do about this issue?
Sure. Just always encode all your skin files as UTF-16 LE. ...Aaaaand, problem solved! You're welcome! :bow:

But this doesn't help, you say? This won't work, you say? Well...

Is it Rainmeter's fault that Notepad doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that Notepad++, Visual Studio / VS Code, and practically every other text editor doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that LUA doesn't read or write files using UTF-16 LE? ...Nope.
Is it Rainmeter's fault that the Windows APIs that Rainmeter needs to function all use UTF-16 LE? ...Nope.

Pretty discouraging, eh? Joking aside, most every app these days uses or can use UTF-8 by default... but not the Windows OS. And there's your problem. So the fix is rewriting Rainmeter to ditch the Windows .INI file format and come up with something else, and -- oh yeah -- break every skin ever written.

Hmm, I guess that's never going to happen either, is it? So, it is what it is, and everyone is just going to have to deal with it... :(
Gadgets Wiki GitHub More Gadgets...
User avatar
tass_co
Posts: 379
Joined: May 4th, 2020, 3:01 pm
Location: Ankara, TURKEY

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by tass_co »

SilverAzide wrote: August 11th, 2022, 1:06 am Sure. Just always encode all your skin files as UTF-16 LE. ...Aaaaand, problem solved! You're welcome! :bow:

But this doesn't help, you say? This won't work, you say? Well...

Is it Rainmeter's fault that Notepad doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that Notepad++, Visual Studio / VS Code, and practically every other text editor doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that LUA doesn't read or write files using UTF-16 LE? ...Nope.
Is it Rainmeter's fault that the Windows APIs that Rainmeter needs to function all use UTF-16 LE? ...Nope.

Pretty discouraging, eh? Joking aside, most every app these days uses or can use UTF-8 by default... but not the Windows OS. And there's your problem. So the fix is rewriting Rainmeter to ditch the Windows .INI file format and come up with something else, and -- oh yeah -- break every skin ever written.

Hmm, I guess that's never going to happen either, is it? So, it is what it is, and everyone is just going to have to deal with it... :(
I think someone is very bored :lol:

First off, I didn't say it was Rainmeter fault.
I did not say to change Rainmeter's code system completely. I was just asking if it could be done as a plugin or option.
Maybe the file where the code is written can be checked or the returned data can be examined.
In other words, i think this problem can be solved with a distinctive feature. Am I wrong?

I did some research. There are ready-made codes for c++ and c# encode. It even exists for powershell.

The script could be:
I think a parameter can be added to the [Rainmeter] section.
EncodingWarning or CheckEncode
The skin and the files with code content that are requested to work with the skin can be checked (inc,lua,txt etc)and the result can be given as !Log.
In this way, it is immediately obvious what the problem is caused by.
I don't know where i going from here, but i promise it won't be boring... :great:
User avatar
SilverAzide
Rainmeter Sage
Posts: 2064
Joined: March 23rd, 2015, 5:26 pm

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by SilverAzide »

tass_co wrote: August 11th, 2022, 2:16 am I think someone is very bored :lol:

First off, I didn't say it was Rainmeter fault.
I did not say to change Rainmeter's code system completely. I was just asking if it could be done as a plugin or option.
Maybe the file where the code is written can be checked or the returned data can be examined.
In other words, i think this problem can be solved with a distinctive feature. Am I wrong?

I did some research. There are ready-made codes for c++ and c# encode. It even exists for powershell.

The script could be:
I think a parameter can be added to the [Rainmeter] section.
EncodingWarning or CheckEncode
The skin and the files with code content that are requested to work with the skin can be checked (inc,lua,txt etc)and the result can be given as !Log.
In this way, it is immediately obvious what the problem is caused by.
I'm sorry, none of your points are relevant to this issue. And you missed what I was trying to explain.

You need to understand that at the very lowest level, which is Rainmeter reading and writing to .INI-formatted files, it requires using Win32 APIs. Specifically, if you are curious, the APIs are GetPrivateProfileString and WritePrivateProfileString (and other related ones used for reading/writing sections).

These APIs come in two flavors: ASCII and "wide char" (Unicode). In order to support all languages and encodings, Windows (thus Rainmeter) internally uses the Unicode APIs. Note that I did not say UTF-8; that is not a choice. It's either ASCII or Unicode, period.

All the plugins, C, C#, !Logging, blah blah doesn't matter one bit. The issue boils down to the Win32 APIs Rainmeter uses to read and write to INI-formatted files. Microsoft made a choice, a well-intentioned one in my estimation, way back in the Windows NT/2000 days to use Unicode internally. This hasn't worked out too well 20+ years later, but the code base is pretty much locked in at this point and we have what we have.

Adding a !Log output as a warning can't be done reasonably because Rainmeter has to read the file first, right? And once the file is read, it's too late. Windows isn't going to tell you the file isn't the right format, because the INI spec says "ASCII or Unicode". So maybe you have a "format checker" function that checks to see if the file is OK, but now you have to parse through all the files in a skin looking for every include file and every Lua script and etc. And "UTF-8 without BOM" can look identical to ASCII... until it isn't, so then what? Now you got to read every line and test every character in every file to see if it's OK or not.

To really resolve this issue, what you are asking for is for the Rainmeter devs to essentially re-create from scratch an encoding-aware drop-in replacement for the Windows GetPrivateProfileString/WritePrivateProfileString APIs. I don't want to speak for the devs, but I think this is highly unlikely, to say the least. You really need to just learn to live with the limitations we have and move on.
Gadgets Wiki GitHub More Gadgets...
User avatar
tass_co
Posts: 379
Joined: May 4th, 2020, 3:01 pm
Location: Ankara, TURKEY

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by tass_co »

SilverAzide wrote: August 11th, 2022, 3:40 am I'm sorry, none of your points are relevant to this issue. And you missed what I was trying to explain.

You need to understand that at the very lowest level, which is Rainmeter reading and writing to .INI-formatted files, it requires using Win32 APIs. Specifically, if you are curious, the APIs are GetPrivateProfileString and WritePrivateProfileString (and other related ones used for reading/writing sections).

These APIs come in two flavors: ASCII and "wide char" (Unicode). In order to support all languages and encodings, Windows (thus Rainmeter) internally uses the Unicode APIs. Note that I did not say UTF-8; that is not a choice. It's either ASCII or Unicode, period.

All the plugins, C, C#, !Logging, blah blah doesn't matter one bit. The issue boils down to the Win32 APIs Rainmeter uses to read and write to INI-formatted files. Microsoft made a choice, a well-intentioned one in my estimation, way back in the Windows NT/2000 days to use Unicode internally. This hasn't worked out too well 20+ years later, but the code base is pretty much locked in at this point and we have what we have.

Adding a !Log output as a warning can't be done reasonably because Rainmeter has to read the file first, right? And once the file is read, it's too late. Windows isn't going to tell you the file isn't the right format, because the INI spec says "ASCII or Unicode". So maybe you have a "format checker" function that checks to see if the file is OK, but now you have to parse through all the files in a skin looking for every include file and every Lua script and etc. And "UTF-8 without BOM" can look identical to ASCII... until it isn't, so then what? Now you got to read every line and test every character in every file to see if it's OK or not.

To really resolve this issue, what you are asking for is for the Rainmeter devs to essentially re-create from scratch an encoding-aware drop-in replacement for the Windows GetPrivateProfileString/WritePrivateProfileString APIs. I don't want to speak for the devs, but I think this is highly unlikely, to say the least. You really need to just learn to live with the limitations we have and move on.
Im sorry, I never thought it would be like this.
It seemed like something that could be fixed with a little patch :oops:
Thank you :thumbup:
I don't know where i going from here, but i promise it won't be boring... :great:
User avatar
Brian
Developer
Posts: 2512
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by Brian »

Just to chime in...
SilverAzide wrote: August 11th, 2022, 3:40 am To really resolve this issue, what you are asking for is for the Rainmeter devs to essentially re-create from scratch an encoding-aware drop-in replacement for the Windows GetPrivateProfileString/WritePrivateProfileString APIs. I don't want to speak for the devs, but I think this is highly unlikely, to say the least. You really need to just learn to live with the limitations we have and move on.
It isn't just the use of GetPrivateProfileString/WritePrivateProfileString (as well as the "Section" variants) - it is almost all the interactions with the winapi that expects UTF-16LE. Things like D2D (or DIrectWrite), and all the functions that retrieve "system" information (like the Wifi and SysInfo plugins).

This dependence on the winapi is the main reason why Rainmeter isn't portable to other OS's.

SilverAzide wrote: August 11th, 2022, 3:40 am And "UTF-8 without BOM" can look identical to ASCII... until it isn't, so then what?
The is the main problem with "encoding checking". I am not aware of a method (or library) to correctly detect encoding 100% of the time. There is always outliers in all the encoding formats that can get it wrong. But that also doesn't mean there aren't libraries that aren't "good enough", which I am sure there is.


At the end of the day, since all the internal winapi functions use UTF-16LE, we recommend using this encoding in your skins.

-Brian
User avatar
tass_co
Posts: 379
Joined: May 4th, 2020, 3:01 pm
Location: Ankara, TURKEY

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by tass_co »

Brian wrote: August 11th, 2022, 4:27 am Just to chime in...



It isn't just the use of GetPrivateProfileString/WritePrivateProfileString (as well as the "Section" variants) - it is almost all the interactions with the winapi that expects UTF-16LE. Things like D2D (or DIrectWrite), and all the functions that retrieve "system" information (like the Wifi and SysInfo plugins).

This dependence on the winapi is the main reason why Rainmeter isn't portable to other OS's.




The is the main problem with "encoding checking". I am not aware of a method (or library) to correctly detect encoding 100% of the time. There is always outliers in all the encoding formats that can get it wrong. But that also doesn't mean there aren't libraries that aren't "good enough", which I am sure there is.


At the end of the day, since all the internal winapi functions use UTF-16LE, we recommend using this encoding in your skins.

-Brian
Thank you for information. I wasted your time...
I don't know where i going from here, but i promise it won't be boring... :great:
User avatar
Brian
Developer
Posts: 2512
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by Brian »

tass_co wrote: August 11th, 2022, 4:43 am Thank you for information. I wasted your time...
No, not at all. We've all been there, wondering why things are done they way they are. :)

-Brian
User avatar
Active Colors
Moderator
Posts: 1187
Joined: February 16th, 2012, 3:32 am
Location: Berlin, Germany

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by Active Colors »

SilverAzide wrote: August 11th, 2022, 1:06 am Sure. Just always encode all your skin files as UTF-16 LE. ...Aaaaand, problem solved! You're welcome! :bow:

Is it Rainmeter's fault that Notepad doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that Notepad++, Visual Studio / VS Code, and practically every other text editor doesn't default to UTF-16 LE? ...Nope.
By the way, while not an quick method, you can default to UTF 16 LE system wide https://forum.rainmeter.net/viewtopic.php?t=40002
User avatar
SilverAzide
Rainmeter Sage
Posts: 2064
Joined: March 23rd, 2015, 5:26 pm

Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8

Post by SilverAzide »

Active Colors wrote: August 11th, 2022, 5:53 am By the way, while not an quick method, you can default to UTF 16 LE system wide https://forum.rainmeter.net/viewtopic.php?t=40002
Very handy! I knew about the NP++ one, but the secret Notepad tricks are good ones! :thumbup: Thanks!
Gadgets Wiki GitHub More Gadgets...