It is currently March 28th, 2024, 9:59 pm

⭐ Weather.com - Parsing the V3 JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the JSON

Post by Yincognito »

OnyxBlack wrote: June 3rd, 2020, 10:39 am
  • Unit measures have been disabled.
    The new json doesn't include unit information (C°/F°, pressure, etc.) about the data, so I've commented out those measures. In the future this might change, so they can be re-enabled.
Yes, it does (include unit information). It's on every unit related section in the JSON (i.e. observation, all kinds of forecast, etc). Look for the units: string in the new JSON and you'll see what I mean. It's only 3 systems, m=metric, e=imperial and h=hybrid (aka UK). These look like this (presented this way for the sake of convenience with the "old" data - in practice you'd create some variables or a measure to hold the values or toggle between the systems based on the value of the character after the units: string above):

Code: Select all

"unitOfMeasurement":{"units":"m","temp":"C","speed":"km\u002Fh","distance":"km","accumulation":"cm","precip":"mm","pressure":"mb"},
"unitOfMeasurement":{"units":"e","temp":"F","speed":"mph","distance":"mi","accumulation":"in","precip":"in","pressure":"in"},
"unitOfMeasurement":{"units":"h","temp":"C","speed":"mph","distance":"mi","accumulation":"cm","precip":"mm","pressure":"mb"},
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the JSON

Post by jsmorley »

We certainly can parse any JSON format they provide, I'm not too concerned about that.

What I am concerned about is any rush to solve this before we understand the problem. I have not yet seen anything that makes me comfortable that we understand exactly what is going on, what triggers or doesn't trigger any different format(s), how you can or cannot achieve a consistent and reliable result, and assuming that things are in some kind of "flux" right now, where it is all going to end up.

So by all means carry on, but for myself, I'm not going to even look at how to parse any "different" JSON format until I'm happy that there is something reliable that I can count on. If we have to, or desire to, use a UserAgent string to force one format or another, I'm happy to cross that bridge when we get to it, but I'm not confident that we have proved that there is ANY way to get a single, consistent format yet, and I'm going to wait a little longer to see if weather.com settles down.
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 2:17 pm Got it, did it. Thanks.

Curious though, if this is being served only to mobile devices based on useragent, why am I getting the same (new) code when I visit on a desktop browser?
This is the question we're all wondering about. No one knows why people with desktop browsers get the V3 JSON. It is random, and can go away with the cookie-clearing steps. JSMorley's suggestion to wait is a good one, because this situation isn't settled yet. But if you need a solution now... :)

P.S.: On your tweaked JSON measures, great job! My only recommendation (besides restoring the units per Yincognito's suggestion), is to handle the short day names using a Time measure like the way JSMorley's original file handles the month names (use the time stamp and return the short day using a format string per the user's locale). Your version only works for English.
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 27
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

Yincognito wrote: June 3rd, 2020, 2:21 pm Yes, it does (include unit information). It's on every unit related section in the JSON (i.e. observation, all kinds of forecast, etc). Look for the units: string in the new JSON and you'll see what I mean.
Ah, that explains things. I assumed in the old json (which I don't have access too obviously) there was some section or key that stored all those individually, based on the following regex in the original [@UnitParent]:

Code: Select all

RegExp=(?siU){"units":(.*),"temp":(.*),.*"speed":(.*),.*"distance":(.*),.*accumulation":(.*),.*"precip":(.*),.*"pressure":(.*)}

I didn't realise I had to create my own measure to provide all the right units based on that single "unit:[x]" string, e.g. "units:m" means provide all metric unit types. I can fix that.

jsmorley wrote:So by all means carry on, but for myself, I'm not going to even look at how to parse any "different" JSON format until I'm happy that there is something reliable that I can count on. If we have to, or desire to, use a UserAgent string to force one format or another, I'm happy to cross that bridge when we get to it, but I'm not confident that we have proved that there is ANY way to get a single, consistent format yet, and I'm going to wait a little longer to see if weather.com settles down.
Fair enough, but this is the site source I'm being provided via webparser and without my new measures and regex, my weather skin won't work. Others may (and seem to) have the same issue.

SilverAzide wrote:This is the question we're all wondering about. No one knows why people with desktop browsers get the V3 JSON. It is random, and can go away with the cookie-clearing steps. JSMorley's suggestion to wait is a good one, because this situation isn't settled yet. But if you need a solution now... :)

P.S.: On your tweaked JSON measures, great job! My only recommendation (besides restoring the units per Yincognito's suggestion), is to handle the short day names using a Time measure like the way JSMorley's original file handles the month names (use the time stamp and return the short day using a format string per the user's locale). Your version only works for English.
Yeah, I need it now. Otherwise, how will I know whether the sun is going to come up tomorrow? :) Seriously though, I've been using this for a few days and so far, it has worked continuously.
Good call on short day names, I was a bit short sighted there (and lazy, I guess).
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the JSON

Post by Yincognito »

OnyxBlack wrote: June 3rd, 2020, 2:54 pm Ah, that explains things. I assumed in the old json (which I don't have access too obviously) there was some section or key that stored all those individually, based on the following regex in the original [@UnitParent]:

Code: Select all

RegExp=(?siU){"units":(.*),"temp":(.*),.*"speed":(.*),.*"distance":(.*),.*accumulation":(.*),.*"precip":(.*),.*"pressure":(.*)}
There was, yes - some section similar to the format I posted above. But since such a section doesn't exist in the V3 JSON, well, one has to work with what he's got (i.e. setting the units locally based on the the units:[x] part).
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
OnyxBlack
Posts: 27
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

SilverAzide wrote: June 3rd, 2020, 2:40 pm My only recommendation (besides restoring the units per Yincognito's suggestion), is to handle the short day names using a Time measure like the way JSMorley's original file handles the month names (use the time stamp and return the short day using a format string per the user's locale). Your version only works for English.
Fixed, should work with all locales now (based on language set in the Variables file)

Unit types have been reenabled.
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 6:31 pm Fixed, should work with all locales now (based on language set in the Variables file)

Unit types have been reenabled.
Excellent, and in the nick of time too. Just got hit with V3 JSON again today for the first time since the first day this started (clearing the cookies has fixed it for me). Great work!!!
Gadgets Wiki GitHub More Gadgets...
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 6:31 pm Fixed, should work with all locales now (based on language set in the Variables file)

Unit types have been reenabled.
I think the "Narrative" regex is busted... if there is a comma in the text, it's chopping off everything before the comma. :( It's also grabbing the wrong narratives; instead of grabbing the ones in the "daypart" section, it's grabbing some from the data section and some from the 15 day forecast?
Last edited by SilverAzide on June 3rd, 2020, 9:10 pm, edited 1 time in total.
Gadgets Wiki GitHub More Gadgets...
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the JSON

Post by Yincognito »

SilverAzide wrote: June 3rd, 2020, 8:05 pm I think the "Narrative" regex is busted... if there is a comma in the text, it's chopping off everything before the comma. :(
But should be a comma in the narrative though? In other words, have you seen instances where there is a comma in those texts (e.g. phrase, narrative, etc)?
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
OnyxBlack
Posts: 27
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

SilverAzide wrote: June 3rd, 2020, 8:05 pm I think the "Narrative" regex is busted... if there is a comma in the text, it's chopping off everything before the comma. :(
True, if a value contains a comma, the regex will not work (or match prematurely). I hadn't thought of that, then again I have yet to encounter a comma in a value that is not a delimiter. On first thought I assumed it would suffice to simply match with the surrounding "" quotation marks, but then there are these pesky Nulls that don't fit that pattern. Not sure if "narrative" suffers from occasional nulls though.
Thoughts?