It is currently September 21st, 2020, 12:25 pm

[BUG or MISTAKE] Issue with substitution in WebParser string

Report bugs with the Rainmeter application and suggest features.
User avatar
Yincognito
Posts: 2565
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

[BUG or MISTAKE] Issue with substitution in WebParser string

Post by Yincognito »

I posted this here because I've encountered this problem before, and although being a mistake in my code(s) is a possibility, the fact that it happened to me in multiple circumstances made me consider that it might be something wrong here with the parsing.

Skin:
Test_1.0.0.rmskin
Code:
@Resources\Variables.inc:

Code: Select all

[Variables]

a=5

Test.ini:

Code: Select all

[Variables]

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1
BackgroundMode=2
SolidColor=47,47,47,255

---Measures---

[MeasureParseFile]
Measure=WebParser
Url="file://#@#Variables.inc"
CodePage=1200
RegExp=(?siU)^(.*)$
StringIndex=1
UpdateRate=600
FinishAction=[!UpdateMeasureGroup "FileGroup"][!UpdateMeter *][!Redraw]
OnConnectErrorAction=[!UpdateMeasureGroup "FileGroup"][!UpdateMeter *][!Redraw]
OnRegExpErrorAction=[!UpdateMeasureGroup "FileGroup"][!UpdateMeter *][!Redraw]

[MeasureFileString]
Group=FileGroup
Measure=String
String="[MeasureParseFile]"
RegExpSubstitute=1
Substitute="(?:^\s+|\s+$)":"","(?si)^\s*\[Variables\]\s*(?=\R)":"","(?siU)\R(\N*)=(\N*?)":'[!SetVariable \1 \2]'
;Substitute="(?:^\s+|\s+$)":"","(?si)\s*\[Variables\]\s*(?=\R)":"","(?siU)\R(\N*)=(\N*?)":'[!SetVariable \1 \2]'
;Substitute="(?:^\s+|\s+$)":"","(?si)^\s*\[Variables\]\s*(?=\R)":"","(?siU)\R(\N*)=(\N*?)":'[!SetVariable \1 \2]',"(?si)^.(.*)$":"\1"
;Substitute="(?:^\s+|\s+$)":"","(?si)\s*\[Variables\]\s*(?=\R)":"","(?siU)\R(\N*)=(\N*?)":'[!SetVariable \1 \2]',"(?si)^.(.*)$":"\1"
UpdateDivider=-1
OnUpdateAction=[MeasureFileString]
DynamicVariables=1

---Meters---

[MeterTest]
Meter=STRING
X=0
Y=0
FontFace=Consolas
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
FontSize=16
AntiAlias=1
Text="a = #a#"
DynamicVariables=1
What the skin does: Replicate what @Include does, by parsing the Variables.inc file locally using WebParser
Expected behavior: The first 2 substitutes should work, the last 2 should not
Actual behavior: The first 2 susbtitutes don't work, but the last 2 do work
Problem Assumption: A character is "lost" in the string due to the first execution of the [MeasureFileString] measure
Questions: Why this happens and how to fix this? Is this a mistake on my part or something else going on? If the latter, is this the expected behavior?

P.S. It isn't necessarily my intention to use this approach instead of @Include - it just happens that I posted a reply in another slightly related thread and noticed the issue again. Since when I encountered the problem before the code(s) were much more complicated and this was basic enough to be understood easily, I took the opportunity of mentioning the (potential) problem on the forum. The substitute is fairly basic as well: removes the leading and trailing whitespace characters, removes the section name (i.e. [Variables]) and whitespaces up until the last new line, and finally replaces each SomeVar=SomeValue line with [!SetVariable SomeVar SomeValue] (didn't covered the string values in this sample, as it didn't bring anything new to the table anyway).

P.S.S. The Log, although helpful, doesn't quite help with these issues, since you can't "see" line breaks and other such characters in the log.
You do not have the required permissions to view the files attached to this post.
Last edited by Yincognito on August 21st, 2020, 12:03 pm, edited 1 time in total.
User avatar
Brian
Developer
Posts: 2062
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: [BUG or MISTAKE] Issue with substitution in WebParser string

Post by Brian »

While this is technically a mistake, it's an easy one to make, especially with non-printable characters.

Your "Possible Assumption" is close. What is happening is Variables.inc file is UTF-16LE encoded...meaning it has a BOM placed before the text (as I am sure you already know). When the Substitute is executed (in the OnUpdateAction), our bang parser tries to execute that command starting with the BOM from the file. It fails, but it isn't obvious since we usually do not log execution errors since we cannot know if a command passed to Windows is successful or not.

In other words, the Substitute returns the string X[Variables]\r\n\r\n[!SetVariable a 5] (with X being the BOM, which does not print). It actually sends that entire string to Windows to execute. Yikes. It's a good thing Windows doesn't produce an error in these cases.

You can see what Substitute returns in the About dialog (Skins tab).

The second set of substitutes work because they ignore that first character (the BOM). However, this too is partially incorrect since the Substitute now makes the string value [Variables]\r\n\r\n[!SetVariable a 5]...basically the same as above without the BOM. So, the bang parser now sends the command Variables to Windows ( 8-) ), and then executes the !SetVariable bang. While you get the results you want, it really isn't correct.
Space
Space
Space
Space
So, there are 2 solutions here. Either account for the BOM (either UTF-16LE or UTF-8) in your Substitute....or just use ANSI/UTF8(without BOM) encoding in the Variables.inc file.

-Brian

PS - You should use OnChangeAction instead of OnUpdateAction for cases like these. (I understand it's just an example.)
User avatar
Yincognito
Posts: 2565
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: [BUG or MISTAKE] Issue with substitution in WebParser string

Post by Yincognito »

Brian wrote: August 20th, 2020, 9:17 pm While this is technically a mistake, it's an easy one to make, especially with non-printable characters.

Your "Possible Assumption" is close. What is happening is Variables.inc file is UTF-16LE encoded...meaning it has a BOM placed before the text (as I am sure you already know). When the Substitute is executed (in the OnUpdateAction), our bang parser tries to execute that command starting with the BOM from the file. It fails, but it isn't obvious since we usually do not log execution errors since we cannot know if a command passed to Windows is successful or not.

In other words, the Substitute returns the string X[Variables]\r\n\r\n[!SetVariable a 5] (with X being the BOM, which does not print). It actually sends that entire string to Windows to execute. Yikes. It's a good thing Windows doesn't produce an error in these cases.

You can see what Substitute returns in the About dialog (Skins tab).

The second set of substitutes work because they ignore that first character (the BOM). However, this too is partially incorrect since the Substitute now makes the string value [Variables]\r\n\r\n[!SetVariable a 5]...basically the same as above without the BOM. So, the bang parser now sends the command Variables to Windows ( 8-) ), and then executes the !SetVariable bang. While you get the results you want, it really isn't correct.
Space
Space
Space
Space
So, there are 2 solutions here. Either account for the BOM (either UTF-16LE or UTF-8) in your Substitute....or just use ANSI/UTF8(without BOM) encoding in the Variables.inc file.

-Brian

PS - You should use OnChangeAction instead of OnUpdateAction for cases like these. (I understand it's just an example.)
Yeah, the encoding was among my list of suspects too, along with some hypothetical EOF funky character, or some EOL lost somewhere (however the latter was out of the question since I tried to remove all the \s beforehand). I was really hoping that it wasn't the encoding though, as I don't like to encode files in something else other than UCS-2 LE BOM when it comes to Rainmeter files. The other solution is tricky as well, since I don't have any control over what some user decides the encoding will be, I can only control my own original files. Huh, that's kind of an unfortunate situation, but at least now I know the cause of it - many thanks for letting me know and confirming one of my possible assumptions. Will have to investigate a bit what else can be done in this matter, and knowing "my enemy" makes me confident I'll eventually find an acceptable solution.

Thanks again for answering and looking into it. :thumbup:
User avatar
Yincognito
Posts: 2565
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: [BUG or MISTAKE] Issue with substitution in WebParser string

Post by Yincognito »

Brian wrote: August 20th, 2020, 9:17 pmSo, there are 2 solutions here. Either account for the BOM (either UTF-16LE or UTF-8) in your Substitute....or just use ANSI/UTF8(without BOM) encoding in the Variables.inc file.

-Brian
Yep, solved this by using a "^[\xFEFF]":"", at the start of the substitute. Many thanks for pointing me in the right direction. ;-)
User avatar
Brian
Developer
Posts: 2062
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: [BUG or MISTAKE] Issue with substitution in WebParser string

Post by Brian »

You're welcome, and I am glad you got it sorted out.

One day we will move on from retrofitting old technology and have no need for "invisible" characters and different text encoding.

-Brian
User avatar
Yincognito
Posts: 2565
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: [BUG or MISTAKE] Issue with substitution in WebParser string

Post by Yincognito »

Brian wrote: August 21st, 2020, 4:50 am You're welcome, and I am glad you got it sorted out.

One day we will move on from retrofitting old technology and have no need for "invisible" characters and different text encoding.

-Brian
Yeah, well, I didn't mind the invisible characters, I did however mind that I didn't know exactly which one was it so I can deal with it. After all, I also use invisible characters in my skins quite extensively, [\x200B] and [\x2006] being my favorites. I use the former to manually "mark" parts of strings that I want to use a different inline setting on in my tooltips, and the latter to fit a couple of strings in a limited location without having to increase the skin's width.