It is currently June 24th, 2021, 4:17 pm

⭐ Weather.com - Parsing the V3 JSON - NEW!

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
Yincognito
Posts: 3250
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

SilverAzide wrote: August 27th, 2020, 12:33 am As of this moment there are 9 simultaneous weather alerts in Lake Charles, LA. I've never seen that many at once. I had to tweak the JSONAlerts file to get them all. Time to get out of town, folks! On the upside, Beta r3404 is working perfectly now.

WxMeter Alert Severe 9a.pngWxMeter Alert Severe 9.png
Thanks for mentioning it - even though I don't use the alerts (I use qualifierPhrase instead, it's simpler to manipulate and such events reflect there as well), I'm interested in how the source looks like during extreme situations (e.g. max number of alerts, max number of chars in a phrase, various formats for the snow amount, that kind of things).
User avatar
Cariboudjan
Posts: 150
Joined: May 12th, 2019, 8:55 am

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Cariboudjan »

Could the RegExp for Weather.com be located on Rainmeter.net? That way authors can parse Rainmeter.net for the RegExp to use to parse Weather.com with a double webparser.

Then authors do not need to keep updating their skins. When jsmorley updates the xml on Rainmeter.net, it will filter out to all of the Rainmeter skins out in the wild and start working again without any change to the skin itself.

I know I know it would mean that all weather Rainmeter skins would be constantly parsing Rainmeter.net. But it would be nice.

Maybe the template for the webparser could only parse on skin load, writekeyvalue the result to itself, and use the value when skin refreshed to reduce server pinging. That way the majority of pinging would only occur when most users restart their PCs or exit Game Mode.


Actually........ Why not include the weather.com RegExp as a built-in variable or something. That way if you want to get Weather.com working again you just need to update Rainmeter...
User avatar
Yincognito
Posts: 3250
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

Cariboudjan wrote: January 18th, 2021, 2:26 am Could the RegExp for Weather.com be located on Rainmeter.net? That way authors can parse Rainmeter.net for the RegExp to use to parse Weather.com with a double webparser.

Then authors do not need to keep updating their skins. When jsmorley updates the xml on Rainmeter.net, it will filter out to all of the Rainmeter skins out in the wild and start working again without any change to the skin itself.

I know I know it would mean that all weather Rainmeter skins would be constantly parsing Rainmeter.net. But it would be nice.

Maybe the template for the webparser could only parse on skin load, writekeyvalue the result to itself, and use the value when skin refreshed to reduce server pinging. That way the majority of pinging would only occur when most users restart their PCs or exit Game Mode.


Actually........ Why not include the weather.com RegExp as a built-in variable or something. That way if you want to get Weather.com working again you just need to update Rainmeter...
This approach has already been suggested in various forms already by some users - I believe even this thread has some similar suggestions somewhere. The answer has been pretty straight forward - just saying. Maybe you're already aware of it, but if you aren't, this has already been discussed. ;-)
nek
Posts: 42
Joined: November 3rd, 2019, 12:00 am

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by nek »

I found something, tiny things about regular expressions in the
WeatherComJSON_August 4, 2020.rmskin > WeatherComJSONMeasures.inc | Updated July 31, 2020

There is a code RegExp=(?siU)^(.*)$ for WebParser measure, but RegExp=(?s)^(.*)$ is fewer steps and faster to parse.

Code: Select all

[@EntireSiteSuperParent]
Measure=WebParser
URL=#URLSite#
UpdateRate=#UpdateRate#
Flags=Resync | NoCookies
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
DecodeCharacterReference=1
LogSubstringErrors=0
RegExp=(?siU)^(.*)$
StringIndex=1
FinishAction=[!EnableMeasureGroup Parents][!Update]

What is the best Regular Expressions for WebParser measure to fetch the whole contents?

I have tested some regular expressions and here are the results.

1. Testing tool
https://regex101.com/
Regex101 settings for a WebParser RegExp FLAVOR PCRE(PHP <7.3) FUNCTION Match REGEX FLAGS none
regex101.WebParser.RegExp.png
FYI: Regex101 settings for a Measures Substitute FLAVOR PCRE(PHP <7.3) FUNCTION Substitution REGEX FLAGS g
regex101.Measure.Substitute.png
2. Sample text (300 byte, LF)

Code: Select all

Rainmeter allows you to display customizable skins on your desktop, from hardware usage meters to fully functional audio visualizers.

You are only limited by your imagination and creativity.


Rainmeter is open source software distributed free of charge under the terms of the GNU GPL v2 license.
xyz
3. Results
RegExp.match.steps.png
(?siU)
s modifier: single line. Dot matches newline characters
i modifier: insensitive. Case insensitive match (ignores case of [a-zA-Z])
U modifier: Ungreedy. The match becomes lazy by default. Now a ? following a quantifier makes it greedy

As we can see (?s)^(.*)$ is fewer steps than (?siU)^(.*)$.
(?siU) is great options to parse a part of contents like RegExp=(?siU)"timeZoneAbbreviation":"(.*)", but not suitable to get whole contents.
Basically, child measures of WebPaser are able to access the whole contents with parent WebParser measure's RegExp=(?s)^.*$ (non-captured) and get rid of the StringIndex=1.

Code: Select all

[@EntireSiteSuperParent]
Measure=WebParser
URL=#URLSite#
UpdateRate=#UpdateRate#
Flags=Resync | NoCookies
UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
DecodeCharacterReference=1
LogSubstringErrors=0
RegExp=(?s)^.*$
;RegExp=(?siU)^(.*)$
;StringIndex=1
FinishAction=[!EnableMeasureGroup Parents][!Update]
4. .* and .+ are greedy by default
RegExp.greedy.png
5. References
PCRE Regex Cheatsheet
Resources for regular expressions - Rainmeter Docs
WebParser measure - Rainmeter Docs
You do not have the required permissions to view the files attached to this post.
Last edited by nek on February 13th, 2021, 11:27 pm, edited 3 times in total.
User avatar
jsmorley
Developer
Posts: 21787
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

You are right! I'll change that the next time I modify the @Include files. Thanks!
User avatar
Yincognito
Posts: 3250
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

nek wrote: January 30th, 2021, 7:29 pm I found something, tiny things about regular expressions in the
WeatherComJSON_August 4, 2020.rmskin > WeatherComJSONMeasures.inc | Updated July 31, 2020

There is a code RegExp=(?siU)^(.*)$ for WebParser measure, but RegExp=(?s)^(.*)$ is fewer steps and faster to parse.
You'll find a lot of such things in any skin that uses regex though, not just here. While you're absolutely right that it takes more steps (I'm also careful in that regard, never using something if it's not functionally required), my feeling is that the (?siU) flags are used just as a way to "set and forget" in most skins, where this form is preferred "just in case", even though some of the individual flags are not always needed.

Just curious, why didn't you suggested ^([\s\S]*)$ instead? In your table it's the most efficient, and this is supported by various analyses by regex experts online, who point out that specifying the pattern as close as possible yields the best results in terms of speed. Some even suggest avoiding the . (dot, every character) pattern as often and whenever possible, and "narrow down" the pattern to make it faster.
nek
Posts: 42
Joined: November 3rd, 2019, 12:00 am

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by nek »

Yincognito wrote: January 30th, 2021, 9:03 pm Just curious, why didn't you suggested ^([\s\S]*)$ instead?
Because...
(?s)^.*$ is cool :thumbup: , and ^([\s\S]*)$ is not.
There are not any technical reasons at all.
I am not expert of the regular expressions, and I don't remember whitch \s \S is match with whitespace character.
I just feel that (?s)^.*$ is more easier to understand for people. That's it.
Last edited by nek on March 23rd, 2021, 6:45 pm, edited 1 time in total.
User avatar
Yincognito
Posts: 3250
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

nek wrote: January 30th, 2021, 10:09 pm Because...
(?s)^.*$ is cool :thumbup: , and ^([\s\S]*)$ is not.
There are not any technical reasons at all.
I am not expert of the regular expressions, and I don't remember whitch \s \S is match with whitespace character.
I just feel that (?s)^.*$ is easy to understand for everyone. That it.
Ah, ok, no problem - I'm not an expert either. I guess I was just confused as to why a technical reason was appropriate for a certain choice, while it wasn't for the other, that's all. :confused:

P.S. Remembering what \s and \S stand for is easy: in regex, lower-case are affirmations, while upper-case are negations (e.g. \d = digit, \D = not digit; \s = whitespace. \S = not whitespace; \w = word, \W= not word; etc).