Hello everyone.
Can't we put an end to this mess?
We encounter errors while writing skin code.
Although we are sure of our code, the skin is not working properly.
You are trying everything but the result is negative.
You're going crazy.
Then you realize that your code is not working because of this ridiculous situation(of course if you can).
It's really frustrating...
We encounter such errors a lot on the forum.
Is there anything we can do about this issue?
It is currently September 8th, 2024, 1:13 am
[Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
-
- Posts: 522
- Joined: May 4th, 2020, 3:01 pm
- Location: Ankara, TURKEY
[Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
I don't know where i going from here, but i promise it won't be boring... ![Great :great:](./images/smilies/dn.gif)
![Great :great:](./images/smilies/dn.gif)
-
- Rainmeter Sage
- Posts: 2734
- Joined: March 23rd, 2015, 5:26 pm
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
Sure. Just always encode all your skin files as UTF-16 LE. ...Aaaaand, problem solved! You're welcome!tass_co wrote: ↑August 10th, 2022, 10:46 pm Hello everyone.
Can't we put an end to this mess?
We encounter errors while writing skin code.
Although we are sure of our code, the skin is not working properly.
You are trying everything but the result is negative.
You're going crazy.
Then you realize that your code is not working because of this ridiculous situation(of course if you can).
It's really frustrating...
We encounter such errors a lot on the forum.
Is there anything we can do about this issue?
![Bow :bow:](./images/smilies/Dziekuje.gif)
But this doesn't help, you say? This won't work, you say? Well...
Is it Rainmeter's fault that Notepad doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that Notepad++, Visual Studio / VS Code, and practically every other text editor doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that LUA doesn't read or write files using UTF-16 LE? ...Nope.
Is it Rainmeter's fault that the Windows APIs that Rainmeter needs to function all use UTF-16 LE? ...Nope.
Pretty discouraging, eh? Joking aside, most every app these days uses or can use UTF-8 by default... but not the Windows OS. And there's your problem. So the fix is rewriting Rainmeter to ditch the Windows .INI file format and come up with something else, and -- oh yeah -- break every skin ever written.
Hmm, I guess that's never going to happen either, is it? So, it is what it is, and everyone is just going to have to deal with it...
![Sad :(](./images/smilies/ac.gif)
-
- Posts: 522
- Joined: May 4th, 2020, 3:01 pm
- Location: Ankara, TURKEY
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
I think someone is very boredSilverAzide wrote: ↑August 11th, 2022, 1:06 am Sure. Just always encode all your skin files as UTF-16 LE. ...Aaaaand, problem solved! You're welcome!![]()
But this doesn't help, you say? This won't work, you say? Well...
Is it Rainmeter's fault that Notepad doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that Notepad++, Visual Studio / VS Code, and practically every other text editor doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that LUA doesn't read or write files using UTF-16 LE? ...Nope.
Is it Rainmeter's fault that the Windows APIs that Rainmeter needs to function all use UTF-16 LE? ...Nope.
Pretty discouraging, eh? Joking aside, most every app these days uses or can use UTF-8 by default... but not the Windows OS. And there's your problem. So the fix is rewriting Rainmeter to ditch the Windows .INI file format and come up with something else, and -- oh yeah -- break every skin ever written.
Hmm, I guess that's never going to happen either, is it? So, it is what it is, and everyone is just going to have to deal with it...![]()
![LOL :lol:](./images/smilies/ap.gif)
First off, I didn't say it was Rainmeter fault.
I did not say to change Rainmeter's code system completely. I was just asking if it could be done as a plugin or option.
Maybe the file where the code is written can be checked or the returned data can be examined.
In other words, i think this problem can be solved with a distinctive feature. Am I wrong?
I did some research. There are ready-made codes for c++ and c# encode. It even exists for powershell.
The script could be:
I think a parameter can be added to the [Rainmeter] section.
EncodingWarning or CheckEncode
The skin and the files with code content that are requested to work with the skin can be checked (inc,lua,txt etc)and the result can be given as !Log.
In this way, it is immediately obvious what the problem is caused by.
I don't know where i going from here, but i promise it won't be boring... ![Great :great:](./images/smilies/dn.gif)
![Great :great:](./images/smilies/dn.gif)
-
- Rainmeter Sage
- Posts: 2734
- Joined: March 23rd, 2015, 5:26 pm
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
I'm sorry, none of your points are relevant to this issue. And you missed what I was trying to explain.tass_co wrote: ↑August 11th, 2022, 2:16 am I think someone is very bored![]()
First off, I didn't say it was Rainmeter fault.
I did not say to change Rainmeter's code system completely. I was just asking if it could be done as a plugin or option.
Maybe the file where the code is written can be checked or the returned data can be examined.
In other words, i think this problem can be solved with a distinctive feature. Am I wrong?
I did some research. There are ready-made codes for c++ and c# encode. It even exists for powershell.
The script could be:
I think a parameter can be added to the [Rainmeter] section.
EncodingWarning or CheckEncode
The skin and the files with code content that are requested to work with the skin can be checked (inc,lua,txt etc)and the result can be given as !Log.
In this way, it is immediately obvious what the problem is caused by.
You need to understand that at the very lowest level, which is Rainmeter reading and writing to .INI-formatted files, it requires using Win32 APIs. Specifically, if you are curious, the APIs are GetPrivateProfileString and WritePrivateProfileString (and other related ones used for reading/writing sections).
These APIs come in two flavors: ASCII and "wide char" (Unicode). In order to support all languages and encodings, Windows (thus Rainmeter) internally uses the Unicode APIs. Note that I did not say UTF-8; that is not a choice. It's either ASCII or Unicode, period.
All the plugins, C, C#, !Logging, blah blah doesn't matter one bit. The issue boils down to the Win32 APIs Rainmeter uses to read and write to INI-formatted files. Microsoft made a choice, a well-intentioned one in my estimation, way back in the Windows NT/2000 days to use Unicode internally. This hasn't worked out too well 20+ years later, but the code base is pretty much locked in at this point and we have what we have.
Adding a !Log output as a warning can't be done reasonably because Rainmeter has to read the file first, right? And once the file is read, it's too late. Windows isn't going to tell you the file isn't the right format, because the INI spec says "ASCII or Unicode". So maybe you have a "format checker" function that checks to see if the file is OK, but now you have to parse through all the files in a skin looking for every include file and every Lua script and etc. And "UTF-8 without BOM" can look identical to ASCII... until it isn't, so then what? Now you got to read every line and test every character in every file to see if it's OK or not.
To really resolve this issue, what you are asking for is for the Rainmeter devs to essentially re-create from scratch an encoding-aware drop-in replacement for the Windows GetPrivateProfileString/WritePrivateProfileString APIs. I don't want to speak for the devs, but I think this is highly unlikely, to say the least. You really need to just learn to live with the limitations we have and move on.
-
- Posts: 522
- Joined: May 4th, 2020, 3:01 pm
- Location: Ankara, TURKEY
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
Im sorry, I never thought it would be like this.SilverAzide wrote: ↑August 11th, 2022, 3:40 am I'm sorry, none of your points are relevant to this issue. And you missed what I was trying to explain.
You need to understand that at the very lowest level, which is Rainmeter reading and writing to .INI-formatted files, it requires using Win32 APIs. Specifically, if you are curious, the APIs are GetPrivateProfileString and WritePrivateProfileString (and other related ones used for reading/writing sections).
These APIs come in two flavors: ASCII and "wide char" (Unicode). In order to support all languages and encodings, Windows (thus Rainmeter) internally uses the Unicode APIs. Note that I did not say UTF-8; that is not a choice. It's either ASCII or Unicode, period.
All the plugins, C, C#, !Logging, blah blah doesn't matter one bit. The issue boils down to the Win32 APIs Rainmeter uses to read and write to INI-formatted files. Microsoft made a choice, a well-intentioned one in my estimation, way back in the Windows NT/2000 days to use Unicode internally. This hasn't worked out too well 20+ years later, but the code base is pretty much locked in at this point and we have what we have.
Adding a !Log output as a warning can't be done reasonably because Rainmeter has to read the file first, right? And once the file is read, it's too late. Windows isn't going to tell you the file isn't the right format, because the INI spec says "ASCII or Unicode". So maybe you have a "format checker" function that checks to see if the file is OK, but now you have to parse through all the files in a skin looking for every include file and every Lua script and etc. And "UTF-8 without BOM" can look identical to ASCII... until it isn't, so then what? Now you got to read every line and test every character in every file to see if it's OK or not.
To really resolve this issue, what you are asking for is for the Rainmeter devs to essentially re-create from scratch an encoding-aware drop-in replacement for the Windows GetPrivateProfileString/WritePrivateProfileString APIs. I don't want to speak for the devs, but I think this is highly unlikely, to say the least. You really need to just learn to live with the limitations we have and move on.
It seemed like something that could be fixed with a little patch
![Embarrassed :oops:](./images/smilies/ah.gif)
Thank you
![Thumbs Up :thumbup:](./images/smilies/ay.gif)
I don't know where i going from here, but i promise it won't be boring... ![Great :great:](./images/smilies/dn.gif)
![Great :great:](./images/smilies/dn.gif)
-
- Developer
- Posts: 2725
- Joined: November 24th, 2011, 1:42 am
- Location: Utah
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
Just to chime in...
This dependence on the winapi is the main reason why Rainmeter isn't portable to other OS's.
At the end of the day, since all the internal winapi functions use UTF-16LE, we recommend using this encoding in your skins.
-Brian
It isn't just the use of GetPrivateProfileString/WritePrivateProfileString (as well as the "Section" variants) - it is almost all the interactions with the winapi that expects UTF-16LE. Things like D2D (or DIrectWrite), and all the functions that retrieve "system" information (like the Wifi and SysInfo plugins).SilverAzide wrote: ↑August 11th, 2022, 3:40 am To really resolve this issue, what you are asking for is for the Rainmeter devs to essentially re-create from scratch an encoding-aware drop-in replacement for the Windows GetPrivateProfileString/WritePrivateProfileString APIs. I don't want to speak for the devs, but I think this is highly unlikely, to say the least. You really need to just learn to live with the limitations we have and move on.
This dependence on the winapi is the main reason why Rainmeter isn't portable to other OS's.
The is the main problem with "encoding checking". I am not aware of a method (or library) to correctly detect encoding 100% of the time. There is always outliers in all the encoding formats that can get it wrong. But that also doesn't mean there aren't libraries that aren't "good enough", which I am sure there is.SilverAzide wrote: ↑August 11th, 2022, 3:40 am And "UTF-8 without BOM" can look identical to ASCII... until it isn't, so then what?
At the end of the day, since all the internal winapi functions use UTF-16LE, we recommend using this encoding in your skins.
-Brian
-
- Posts: 522
- Joined: May 4th, 2020, 3:01 pm
- Location: Ankara, TURKEY
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
Thank you for information. I wasted your time...Brian wrote: ↑August 11th, 2022, 4:27 am Just to chime in...
It isn't just the use of GetPrivateProfileString/WritePrivateProfileString (as well as the "Section" variants) - it is almost all the interactions with the winapi that expects UTF-16LE. Things like D2D (or DIrectWrite), and all the functions that retrieve "system" information (like the Wifi and SysInfo plugins).
This dependence on the winapi is the main reason why Rainmeter isn't portable to other OS's.
The is the main problem with "encoding checking". I am not aware of a method (or library) to correctly detect encoding 100% of the time. There is always outliers in all the encoding formats that can get it wrong. But that also doesn't mean there aren't libraries that aren't "good enough", which I am sure there is.
At the end of the day, since all the internal winapi functions use UTF-16LE, we recommend using this encoding in your skins.
-Brian
I don't know where i going from here, but i promise it won't be boring... ![Great :great:](./images/smilies/dn.gif)
![Great :great:](./images/smilies/dn.gif)
-
- Developer
- Posts: 2725
- Joined: November 24th, 2011, 1:42 am
- Location: Utah
-
- Moderator
- Posts: 1297
- Joined: February 16th, 2012, 3:32 am
- Location: Berlin, Germany
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
By the way, while not an quick method, you can default to UTF 16 LE system wide https://forum.rainmeter.net/viewtopic.php?t=40002SilverAzide wrote: ↑August 11th, 2022, 1:06 am Sure. Just always encode all your skin files as UTF-16 LE. ...Aaaaand, problem solved! You're welcome!![]()
Is it Rainmeter's fault that Notepad doesn't default to UTF-16 LE? ...Nope.
Is it Rainmeter's fault that Notepad++, Visual Studio / VS Code, and practically every other text editor doesn't default to UTF-16 LE? ...Nope.
-
- Rainmeter Sage
- Posts: 2734
- Joined: March 23rd, 2015, 5:26 pm
Re: [Feature] Mess: ASCII/UTF-8/UTF-16 LE/BOM UTF-8
Very handy! I knew about the NP++ one, but the secret Notepad tricks are good ones!Active Colors wrote: ↑August 11th, 2022, 5:53 am By the way, while not an quick method, you can default to UTF 16 LE system wide https://forum.rainmeter.net/viewtopic.php?t=40002
![Thumbs Up :thumbup:](./images/smilies/ay.gif)