It is currently March 28th, 2024, 9:19 pm

⭐ Weather.com - Parsing the V3 JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
Yincognito
Rainmeter Sage
Posts: 7027
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

Ok, so after examining and formatting the WebParserDump.txt that I was able to download using the Debug=2 option and wondering why the data seemed to be available but the regex failed, it appears that they indeed changed their format ... but only for accessing stuff from "outside" a browser. In other words, while in the browser the page source would appear the same as before, what is "served" to the WebParser measure is significantly different.

Long story short, this should work for you - let me know if it doesn't (replace the latitude and longitude values with your own):

Code: Select all

[Variables]
Latitude=0
Longitude=0

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1
BackgroundMode=2
SolidColor=47,47,47,255

---Measures---

[MeasureWebParser]
Measure=WebParser
Url=https://weather.com/en-GB/weather/today/l/#Latitude#,#Longitude#
;Debug=2
RegExp=^(?siU).*"getSunV3CurrentObservationsUrlConfig":\{.*"temperature":(.*),.*$
UpdateRate=600

[MeasureTemperature]
Measure=WebParser
Url=[MeasureWebParser]
StringIndex=1

---Meters---

[MeterTemperature]
Meter=STRING
X=0
Y=0
FontFace=Consolas
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
FontSize=16
AntiAlias=1
MeasureName=MeasureTemperature
Text="Temperature = %1 °C"
Basically, what you should do to build an "updated" regex is to set Debug=2 in your WebParser (you can uncomment it in the skin sample above, if you like) just one time in order to get how the source served to WebParser looks like in WebParserDump.txt, optionally format it for clarity in a clone file and then set your regex(es) accordingly.

Or ... you could wait for jsmorley to do that for you, of course. :D
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

I am very hesitant to jump to any conclusions, and certainly hesitant to try and "fix" an issue that at least for me, doesn't exist.


1.jpg


Over the years, I have found it pays to have a little patience with "glitches" with weather.com.
You do not have the required permissions to view the files attached to this post.
User avatar
Yincognito
Rainmeter Sage
Posts: 7027
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

jsmorley wrote: May 22nd, 2020, 10:59 am I am very hesitant to jump to any conclusions, and certainly hesitant to try and "fix" an issue that at least for me, doesn't exist.



1.jpg
Yes, that would be the wise choice indeed - you never know if they're going to revert this or not. Another option, if possible, of course, would be to add those "differences" as optional patterns (either using ? or the | or character) to the current regexes, so that it would work with either format. The latter, obviously, if the two formats aren't critically different from one another.

EDIT: Simple example, based on the sample skin I posted above:

Code: Select all

RegExp=^(?siU).*"(vt1observation|getSunV3CurrentObservationsUrlConfig)":\{.*"temperature":(.*),.*$
This should work with either format. Not sure if it's possible for all the fields, but then, it's just an example, after all.

EDIT 2: While I believe the above "or" method will work for every field, there is another issue that makes it unfunctional: since the names of the fields changed and they are ordered alphabetically in the weather.com source, this means that the order of the fields in the "new" regex(es) would have to change as well. In other words, if this change becomes definitive, one would have to rebuild the regex(es) completely - the "old" regex(es) do help as they can be used as a base for the "new" regex(es), but it's both the names and the order of the names that would have to be "updated" for everything to work flawlessly.

EDIT 3: This is where a stable and supported by the Rainmeter team JSON parser would come handy... 8-) Just change the names of the fields, without bothering with their order in the JSON source. Sure, this can be done using regex as well, but you'd have to waste resources by parsing the source string multiple times, once for each field (assuming they're unique).
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
OnyxBlack
Posts: 27
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

Hi guys,

I've been lurking on this thread hoping someone would post an updated, working measures file. That didn't happen so I went ahead and fixed it myself and registered so I could share my results.

A few things have changed:
  • Unit measures have been disabled.
    The new json doesn't include unit information (C°/F°, pressure, etc.) about the data, so I've commented out those measures. In the future this might change, so they can be re-enabled.
  • Unified the meaning of "Forecast[Day]Day..." measures.
    In the original measures, ForecastToday[Day/Night]Long would always be "today", where other days (Day1, Day2 etc) would return the full name of that day; Forecast...Short would return the abbreviated day name. The keys in the json this was based on have changed their value, so that presented an opportunity to make these more consistent:
    • Forecast[Day]DayLong: Always returns the full name of the day (Sunday, Monday, etc.)
    • Forecast[Day]DayShort: Always returns the abbreviated day name (Sun, Mon, Tue, etc.)
    • Forecast[Day]DayPart (new): Returns the day part name of the associated [Day] measures (Today, Tomorrow, Tuesday, Wednesday, etc.),
    • Forecast[Day]NightPart (new): Returns the night part name of the associated [Day] measures (Tonight, Tomorrow Night, Tuesday Night, etc.)
  • Added a 7th day forecast.
Apart from that, many key names in the json have changed, but I think I've associated all the right ones back with their measures.
If you're looking to work with the json further, note that now forecasts are bundled in arrays. Look at my regex to see how I split them up for extraction.

If you see any bugs or missing features, I'd appreciate contributions.

---------------
Notice: this is only for use if the original WeatherComJSONMeasures.inc from this thread aren't working for you.
---------------
Update (June 4, 16:00 GMT): Fixed forecast narrative, added Moon measures file.
---------------
WeatherComJSON.zip
You do not have the required permissions to view the files attached to this post.
Last edited by OnyxBlack on June 4th, 2020, 2:42 pm, edited 4 times in total.
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

Thanks! This is nice to have, but just be aware that this only works for people who are getting the "new" (V3?) version of the JSON. Most people seem to still be getting the old version, especially after doing the cookie-clearing stuff. Your code is great to have to create a "V3" version of a weather skin if weather.com starts sending that new format to everyone, or if you force the V3 using a user agent string for mobile.
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 27
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

SilverAzide wrote: June 3rd, 2020, 1:25 pm Thanks! This is nice to have, but just be aware that this only works for people who are getting the "new" (V3?) version of the JSON. Most people seem to still be getting the old version, especially after doing the cookie-clearing stuff. Your code is great to have to create a "V3" version of a weather skin if weather.com starts sending that new format to everyone, or if you force the V3 using a user agent string for mobile.
Sure, obviously if everything is working there's no reason to do anything. If it ain't broke, don't fix it :)
Although this will hopefully come in handy for those of us who, for whatever reason, are receiving the new source code from the webparser.
I'll update my post accordingly.
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 1:44 pm Sure, obviously if everything is working there's no reason to do anything. If it ain't broke, don't fix it :)
Although this will hopefully come in handy for those of us who, for whatever reason, are receiving the new source code from the webparser.
I'll update my post accordingly.
Perhaps one handy thing you can do is set the user agent string to one that forces the mobile format. That way, you'll never get the old format, even if you are not having this issue. Thanks for your efforts!
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 27
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

SilverAzide wrote: June 3rd, 2020, 1:48 pm Perhaps one handy thing you can do is set the user agent string to one that forces the mobile format. That way, you'll never get the old format, even if you are not having this issue. Thanks for your efforts!
Yeah, I have no idea what such a UserAgent string would look like, I'm not a web develper :confused:
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

OnyxBlack wrote: June 3rd, 2020, 1:51 pm Yeah, I have no idea what such a UserAgent string would look like, I'm not a web develper :confused:
Run Chrome (go to any website you want), press F12 to open the dev tools window. Click the "three dots" menu all the way at the right side of the menu bar (right of the settings gear icon). Select "More Tools", then "Network conditions". This will add a tab at the bottom of the developer tools window. For the "User agent" option, uncheck "Select automatically" and pick another browser. The user agent string will appear below the dropdown list. You can paste this string in the WeParser measure that gets the entire site, for example:

UserAgent="Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Mobile Safari/537.36"

Now you'll always get V3 JSON, as weather.com thinks you are using a mobile device.
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 27
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

Got it, did it. Thanks.

Curious though, if this is being served only to mobile devices based on useragent, why am I getting the same (new) code when I visit on a desktop browser?