It is currently March 29th, 2024, 2:19 am

⭐ Weather.com - Parsing the V3 JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Yincognito wrote: August 25th, 2020, 3:43 am I don't follow. To me "when the parent hasn't even updated yet" is pretty much the same as "JUST BEFORE the parent measure updates the values", since children are a function of the parent. I understand the subtle difference in what you said, but from a practical point of view, it shouldn't make any difference: once that child measure is 0, it can't be set back to the "past" value of 4 or whatever if we're talking about a leftover from the previous WebParser update, can it? I mean, unless that non zero value is set (or kept) because it remains there in the weather.com source itself (which would be unfortunate), the following parent update will just skip both the number and the string value of the child measure since "if there are no alerts present, no child measures will be populated", like you said earlier.
Yeah, what I'm saying is that if you have some measure that is unrelated to the parent measure for the alerts, you need to be careful when it does the enable/disable or "Reset", as you don't want all the child measures to display as "zero" while they wait for the alert parent measure to actually execute. What I'm saying is that you really want one single process that:

1) Reset all child values to an empty string and a zero number.
2) Update the parent measure
3) The parent measure will populate all the child measures

I don't see any way to have that be a single, connected series of actions, that ensures they are done in the right order at the right time.
User avatar
Brian
Developer
Posts: 2674
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Brian »

Just to chime in here. It looks as if the old Webparser plugin would default the "number" value to 0 on every update. This was omitted (probably by accident) when we converted the plugin to a measure.

Took a few years to catch!

If you look at the committed code from when we switched the plugin to a measure, you will see the plugin used to default the "number" value to 0.
https://github.com/rainmeter/rainmeter/blob/6ae599238785dcd5676db1b0b16b3f954ecd167a/Plugins/PluginWebParser/WebParser.cpp#L759-L827

However, in the same commit, you can see the "new" measure code does not default the number value to "0".
https://github.com/rainmeter/rainmeter/blob/1f8b3e0434eef8a0fa196e4510accf2d27e18939/Library/MeasureWebParser.cpp#L378-L441
This will be fixed in the next beta.

-Brian
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Brian wrote: August 25th, 2020, 4:49 am Just to chime in here. It looks as if the old Webparser plugin would default the "number" value to 0 on every update. This was omitted (probably by accident) when we converted the plugin to a measure.

Took a few years to catch!

If you look at the committed code from when we switched the plugin to a measure, you will see the plugin used to default the "number" value to 0.
https://github.com/rainmeter/rainmeter/blob/6ae599238785dcd5676db1b0b16b3f954ecd167a/Plugins/PluginWebParser/WebParser.cpp#L759-L827

However, in the same commit, you can see the "new" measure code does not default the number value to "0".
https://github.com/rainmeter/rainmeter/blob/1f8b3e0434eef8a0fa196e4510accf2d27e18939/Library/MeasureWebParser.cpp#L378-L441
This will be fixed in the next beta.

-Brian
Ah great. I'll get that out later this morning. That's good news actually, as the "work around" for this behavior was a bad Rube Goldberg. The mouse kept getting out of the trap...
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Beta r3404 is out to address this.
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

jsmorley wrote: August 25th, 2020, 1:13 pm Beta r3404 is out to address this.
It seems changes in weather.com result in an increased number of Rainmeter updates and fixes - who would have thought? ;-)
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by SilverAzide »

As of this moment there are 9 simultaneous weather alerts in Lake Charles, LA. I've never seen that many at once. I had to tweak the JSONAlerts file to get them all. Time to get out of town, folks! On the upside, Beta r3404 is working perfectly now.
WxMeter Alert Severe 9a.png
WxMeter Alert Severe 9.png
You do not have the required permissions to view the files attached to this post.
Gadgets Wiki GitHub More Gadgets...
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

SilverAzide wrote: August 27th, 2020, 12:33 am As of this moment there are 9 simultaneous weather alerts in Lake Charles, LA. I've never seen that many at once. I had to tweak the JSONAlerts file to get them all. Time to get out of town, folks! On the upside, Beta r3404 is working perfectly now.

WxMeter Alert Severe 9a.pngWxMeter Alert Severe 9.png
Thanks for mentioning it - even though I don't use the alerts (I use qualifierPhrase instead, it's simpler to manipulate and such events reflect there as well), I'm interested in how the source looks like during extreme situations (e.g. max number of alerts, max number of chars in a phrase, various formats for the snow amount, that kind of things).
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
Cariboudjan
Posts: 264
Joined: May 12th, 2019, 8:55 am

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Cariboudjan »

Could the RegExp for Weather.com be located on Rainmeter.net? That way authors can parse Rainmeter.net for the RegExp to use to parse Weather.com with a double webparser.

Then authors do not need to keep updating their skins. When jsmorley updates the xml on Rainmeter.net, it will filter out to all of the Rainmeter skins out in the wild and start working again without any change to the skin itself.

I know I know it would mean that all weather Rainmeter skins would be constantly parsing Rainmeter.net. But it would be nice.

Maybe the template for the webparser could only parse on skin load, writekeyvalue the result to itself, and use the value when skin refreshed to reduce server pinging. That way the majority of pinging would only occur when most users restart their PCs or exit Game Mode.


Actually........ Why not include the weather.com RegExp as a built-in variable or something. That way if you want to get Weather.com working again you just need to update Rainmeter...
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

Cariboudjan wrote: January 18th, 2021, 2:26 am Could the RegExp for Weather.com be located on Rainmeter.net? That way authors can parse Rainmeter.net for the RegExp to use to parse Weather.com with a double webparser.

Then authors do not need to keep updating their skins. When jsmorley updates the xml on Rainmeter.net, it will filter out to all of the Rainmeter skins out in the wild and start working again without any change to the skin itself.

I know I know it would mean that all weather Rainmeter skins would be constantly parsing Rainmeter.net. But it would be nice.

Maybe the template for the webparser could only parse on skin load, writekeyvalue the result to itself, and use the value when skin refreshed to reduce server pinging. That way the majority of pinging would only occur when most users restart their PCs or exit Game Mode.


Actually........ Why not include the weather.com RegExp as a built-in variable or something. That way if you want to get Weather.com working again you just need to update Rainmeter...
This approach has already been suggested in various forms already by some users - I believe even this thread has some similar suggestions somewhere. The answer has been pretty straight forward - just saying. Maybe you're already aware of it, but if you aren't, this has already been discussed. ;-)
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
nek
Posts: 105
Joined: November 3rd, 2019, 12:00 am

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by nek »

I found something, tiny things about regular expressions in the
WeatherComJSON_August 4, 2020.rmskin > WeatherComJSONMeasures.inc | Updated July 31, 2020

There is a code RegExp=(?siU)^(.*)$ for WebParser measure, but RegExp=(?s)^(.*)$ is fewer steps and faster to parse.

Code: Select all

[@EntireSiteSuperParent]
Measure=WebParser
URL=#URLSite#
UpdateRate=#UpdateRate#
Flags=Resync | NoCookies
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
DecodeCharacterReference=1
LogSubstringErrors=0
RegExp=(?siU)^(.*)$
StringIndex=1
FinishAction=[!EnableMeasureGroup Parents][!Update]

What is the best Regular Expressions for WebParser measure to fetch the whole contents?

I have tested some regular expressions and here are the results.

1. Testing tool
https://regex101.com/
Regex101 settings for a WebParser RegExp FLAVOR PCRE(PHP <7.3) FUNCTION Match REGEX FLAGS none
regex101.WebParser.RegExp.png
FYI: Regex101 settings for a Measures Substitute FLAVOR PCRE(PHP <7.3) FUNCTION Substitution REGEX FLAGS g
regex101.Measure.Substitute.png
2. Sample text (300 byte, LF)

Code: Select all

Rainmeter allows you to display customizable skins on your desktop, from hardware usage meters to fully functional audio visualizers.

You are only limited by your imagination and creativity.


Rainmeter is open source software distributed free of charge under the terms of the GNU GPL v2 license.
xyz
3. Results
RegExp.match.steps.png
(?siU)
s modifier: single line. Dot matches newline characters
i modifier: insensitive. Case insensitive match (ignores case of [a-zA-Z])
U modifier: Ungreedy. The match becomes lazy by default. Now a ? following a quantifier makes it greedy

As we can see (?s)^(.*)$ is fewer steps than (?siU)^(.*)$.
(?siU) is great options to parse a part of contents like RegExp=(?siU)"timeZoneAbbreviation":"(.*)", but not suitable to get whole contents.
Basically, child measures of WebPaser are able to access the whole contents with parent WebParser measure's RegExp=(?s)^.*$ (non-captured) and get rid of the StringIndex=1.

Code: Select all

[@EntireSiteSuperParent]
Measure=WebParser
URL=#URLSite#
UpdateRate=#UpdateRate#
Flags=Resync | NoCookies
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
DecodeCharacterReference=1
LogSubstringErrors=0
RegExp=(?s)^.*$
;RegExp=(?siU)^(.*)$
;StringIndex=1
FinishAction=[!EnableMeasureGroup Parents][!Update]
4. .* and .+ are greedy by default
RegExp.greedy.png
5. References
PCRE Regex Cheatsheet
Resources for regular expressions - Rainmeter Docs
WebParser measure - Rainmeter Docs
You do not have the required permissions to view the files attached to this post.
Last edited by nek on February 13th, 2021, 11:27 pm, edited 3 times in total.