It is currently September 23rd, 2020, 2:50 pm

⭐ Weather.com - Parsing the V3 JSON - NEW!

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
SilverAzide
Posts: 955
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 1:44 pm Sure, obviously if everything is working there's no reason to do anything. If it ain't broke, don't fix it :)
Although this will hopefully come in handy for those of us who, for whatever reason, are receiving the new source code from the webparser.
I'll update my post accordingly.
Perhaps one handy thing you can do is set the user agent string to one that forces the mobile format. That way, you'll never get the old format, even if you are not having this issue. Thanks for your efforts!
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 19
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

SilverAzide wrote: June 3rd, 2020, 1:48 pm Perhaps one handy thing you can do is set the user agent string to one that forces the mobile format. That way, you'll never get the old format, even if you are not having this issue. Thanks for your efforts!
Yeah, I have no idea what such a UserAgent string would look like, I'm not a web develper :confused:
User avatar
SilverAzide
Posts: 955
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 1:51 pm Yeah, I have no idea what such a UserAgent string would look like, I'm not a web develper :confused:
Run Chrome (go to any website you want), press F12 to open the dev tools window. Click the "three dots" menu all the way at the right side of the menu bar (right of the settings gear icon). Select "More Tools", then "Network conditions". This will add a tab at the bottom of the developer tools window. For the "User agent" option, uncheck "Select automatically" and pick another browser. The user agent string will appear below the dropdown list. You can paste this string in the WeParser measure that gets the entire site, for example:

UserAgent="Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36"

Now you'll always get V3 JSON, as weather.com thinks you are using a mobile device.
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 19
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

Got it, did it. Thanks.

Curious though, if this is being served only to mobile devices based on useragent, why am I getting the same (new) code when I visit on a desktop browser?
User avatar
Yincognito
Posts: 2589
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the JSON

Post by Yincognito »

OnyxBlack wrote: June 3rd, 2020, 10:39 am
  • Unit measures have been disabled.
    The new json doesn't include unit information (C°/F°, pressure, etc.) about the data, so I've commented out those measures. In the future this might change, so they can be re-enabled.
Yes, it does (include unit information). It's on every unit related section in the JSON (i.e. observation, all kinds of forecast, etc). Look for the units: string in the new JSON and you'll see what I mean. It's only 3 systems, m=metric, e=imperial and h=hybrid (aka UK). These look like this (presented this way for the sake of convenience with the "old" data - in practice you'd create some variables or a measure to hold the values or toggle between the systems based on the value of the character after the units: string above):

Code: Select all

"unitOfMeasurement":{"units":"m","temp":"C","speed":"km\u002Fh","distance":"km","accumulation":"cm","precip":"mm","pressure":"mb"},
"unitOfMeasurement":{"units":"e","temp":"F","speed":"mph","distance":"mi","accumulation":"in","precip":"in","pressure":"in"},
"unitOfMeasurement":{"units":"h","temp":"C","speed":"mph","distance":"mi","accumulation":"cm","precip":"mm","pressure":"mb"},
User avatar
jsmorley
Developer
Posts: 21386
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the JSON

Post by jsmorley »

We certainly can parse any JSON format they provide, I'm not too concerned about that.

What I am concerned about is any rush to solve this before we understand the problem. I have not yet seen anything that makes me comfortable that we understand exactly what is going on, what triggers or doesn't trigger any different format(s), how you can or cannot achieve a consistent and reliable result, and assuming that things are in some kind of "flux" right now, where it is all going to end up.

So by all means carry on, but for myself, I'm not going to even look at how to parse any "different" JSON format until I'm happy that there is something reliable that I can count on. If we have to, or desire to, use a UserAgent string to force one format or another, I'm happy to cross that bridge when we get to it, but I'm not confident that we have proved that there is ANY way to get a single, consistent format yet, and I'm going to wait a little longer to see if weather.com settles down.
User avatar
SilverAzide
Posts: 955
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 2:17 pm Got it, did it. Thanks.

Curious though, if this is being served only to mobile devices based on useragent, why am I getting the same (new) code when I visit on a desktop browser?
This is the question we're all wondering about. No one knows why people with desktop browsers get the V3 JSON. It is random, and can go away with the cookie-clearing steps. JSMorley's suggestion to wait is a good one, because this situation isn't settled yet. But if you need a solution now... :)

P.S.: On your tweaked JSON measures, great job! My only recommendation (besides restoring the units per Yincognito's suggestion), is to handle the short day names using a Time measure like the way JSMorley's original file handles the month names (use the time stamp and return the short day using a format string per the user's locale). Your version only works for English.
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 19
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

Yincognito wrote: June 3rd, 2020, 2:21 pm Yes, it does (include unit information). It's on every unit related section in the JSON (i.e. observation, all kinds of forecast, etc). Look for the units: string in the new JSON and you'll see what I mean.
Ah, that explains things. I assumed in the old json (which I don't have access too obviously) there was some section or key that stored all those individually, based on the following regex in the original [@UnitParent]:

Code: Select all

RegExp=(?siU){"units":(.*),"temp":(.*),.*"speed":(.*),.*"distance":(.*),.*accumulation":(.*),.*"precip":(.*),.*"pressure":(.*)}

I didn't realise I had to create my own measure to provide all the right units based on that single "unit:[x]" string, e.g. "units:m" means provide all metric unit types. I can fix that.

jsmorley wrote:So by all means carry on, but for myself, I'm not going to even look at how to parse any "different" JSON format until I'm happy that there is something reliable that I can count on. If we have to, or desire to, use a UserAgent string to force one format or another, I'm happy to cross that bridge when we get to it, but I'm not confident that we have proved that there is ANY way to get a single, consistent format yet, and I'm going to wait a little longer to see if weather.com settles down.
Fair enough, but this is the site source I'm being provided via webparser and without my new measures and regex, my weather skin won't work. Others may (and seem to) have the same issue.

SilverAzide wrote:This is the question we're all wondering about. No one knows why people with desktop browsers get the V3 JSON. It is random, and can go away with the cookie-clearing steps. JSMorley's suggestion to wait is a good one, because this situation isn't settled yet. But if you need a solution now... :)

P.S.: On your tweaked JSON measures, great job! My only recommendation (besides restoring the units per Yincognito's suggestion), is to handle the short day names using a Time measure like the way JSMorley's original file handles the month names (use the time stamp and return the short day using a format string per the user's locale). Your version only works for English.
Yeah, I need it now. Otherwise, how will I know whether the sun is going to come up tomorrow? :) Seriously though, I've been using this for a few days and so far, it has worked continuously.
Good call on short day names, I was a bit short sighted there (and lazy, I guess).
User avatar
Yincognito
Posts: 2589
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the JSON

Post by Yincognito »

OnyxBlack wrote: June 3rd, 2020, 2:54 pm Ah, that explains things. I assumed in the old json (which I don't have access too obviously) there was some section or key that stored all those individually, based on the following regex in the original [@UnitParent]:

Code: Select all

RegExp=(?siU){"units":(.*),"temp":(.*),.*"speed":(.*),.*"distance":(.*),.*accumulation":(.*),.*"precip":(.*),.*"pressure":(.*)}
There was, yes - some section similar to the format I posted above. But since such a section doesn't exist in the V3 JSON, well, one has to work with what he's got (i.e. setting the units locally based on the the units:[x] part).
OnyxBlack
Posts: 19
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

SilverAzide wrote: June 3rd, 2020, 2:40 pm My only recommendation (besides restoring the units per Yincognito's suggestion), is to handle the short day names using a Time measure like the way JSMorley's original file handles the month names (use the time stamp and return the short day using a format string per the user's locale). Your version only works for English.
Fixed, should work with all locales now (based on language set in the Variables file)

Unit types have been reenabled.