It is currently September 23rd, 2020, 11:37 am

⭐ Weather.com - Parsing the V3 JSON - NEW!

Our most popular Tips and Tricks from the Rainmeter Team and others
dvo
Posts: 971
Joined: February 7th, 2016, 6:08 am

Re: ⭐ weather.com - Parsing the JSON

Post by dvo »

in NL he's down to... hopefully it jumps back...wait and see ;-) so it seems not be a local problem... :o
User avatar
Yincognito
Posts: 2587
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

Yep, just turned on my computer today and still no data received in the weather's WebParser. Just tried using various UserAgent strings and it didn't work, but I also tried downloading the file using Debug=2 option and it did download the page source (which can be then parsed).

I don't recommend using the aforementioned "workaround" just yet, maybe someone comes up with a more convenient alternative in the meantime. The "good news" is that there is a workaround for this, albeit a bit inconvenient.
User avatar
xenium
Posts: 554
Joined: January 4th, 2018, 9:52 pm

Re: ⭐ weather.com - Parsing the JSON

Post by xenium »

It works for me.
But WeatherComCodes doesn't work.
I need to manually edit the WeatherComJSONVariables.inc file
Capture.PNG
Capture2.PNG
Edit:
My skins that use the HTML version work for me.
But for other users they don't work, although their locations work for me
I don't know what could be causing this issue
You do not have the required permissions to view the files attached to this post.
User avatar
Yincognito
Posts: 2587
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

xenium wrote: May 22nd, 2020, 8:38 am It works for me.
But WeatherComCodes doesn't work.
I need to manually edit the WeatherComJSONVariables.inc file
Capture.PNG
Capture2.PNG

Edit:
My skins that use the HTML version work for me.
But for other users they don't work, although their locations work for me
I don't know what could be causing this issue
Yeah, found a way to get all the data, just having trouble processing it further (the same regexp error). Will explore some workarounds to handle this.
User avatar
Yincognito
Posts: 2587
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

Ok, so after examining and formatting the WebParserDump.txt that I was able to download using the Debug=2 option and wondering why the data seemed to be available but the regex failed, it appears that they indeed changed their format ... but only for accessing stuff from "outside" a browser. In other words, while in the browser the page source would appear the same as before, what is "served" to the WebParser measure is significantly different.

Long story short, this should work for you - let me know if it doesn't (replace the latitude and longitude values with your own):

Code: Select all

[Variables]
Latitude=0
Longitude=0

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1
BackgroundMode=2
SolidColor=47,47,47,255

---Measures---

[MeasureWebParser]
Measure=WebParser
Url=https://weather.com/en-GB/weather/today/l/#Latitude#,#Longitude#
;Debug=2
RegExp=^(?siU).*"getSunV3CurrentObservationsUrlConfig":\{.*"temperature":(.*),.*$
UpdateRate=600

[MeasureTemperature]
Measure=WebParser
Url=[MeasureWebParser]
StringIndex=1

---Meters---

[MeterTemperature]
Meter=STRING
X=0
Y=0
FontFace=Consolas
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
FontSize=16
AntiAlias=1
MeasureName=MeasureTemperature
Text="Temperature = %1 °C"
Basically, what you should do to build an "updated" regex is to set Debug=2 in your WebParser (you can uncomment it in the skin sample above, if you like) just one time in order to get how the source served to WebParser looks like in WebParserDump.txt, optionally format it for clarity in a clone file and then set your regex(es) accordingly.

Or ... you could wait for jsmorley to do that for you, of course. :D
User avatar
jsmorley
Developer
Posts: 21386
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

I am very hesitant to jump to any conclusions, and certainly hesitant to try and "fix" an issue that at least for me, doesn't exist.


1.jpg


Over the years, I have found it pays to have a little patience with "glitches" with weather.com.
You do not have the required permissions to view the files attached to this post.
User avatar
Yincognito
Posts: 2587
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

jsmorley wrote: May 22nd, 2020, 10:59 am I am very hesitant to jump to any conclusions, and certainly hesitant to try and "fix" an issue that at least for me, doesn't exist.



1.jpg
Yes, that would be the wise choice indeed - you never know if they're going to revert this or not. Another option, if possible, of course, would be to add those "differences" as optional patterns (either using ? or the | or character) to the current regexes, so that it would work with either format. The latter, obviously, if the two formats aren't critically different from one another.

EDIT: Simple example, based on the sample skin I posted above:

Code: Select all

RegExp=^(?siU).*"(vt1observation|getSunV3CurrentObservationsUrlConfig)":\{.*"temperature":(.*),.*$
This should work with either format. Not sure if it's possible for all the fields, but then, it's just an example, after all.

EDIT 2: While I believe the above "or" method will work for every field, there is another issue that makes it unfunctional: since the names of the fields changed and they are ordered alphabetically in the weather.com source, this means that the order of the fields in the "new" regex(es) would have to change as well. In other words, if this change becomes definitive, one would have to rebuild the regex(es) completely - the "old" regex(es) do help as they can be used as a base for the "new" regex(es), but it's both the names and the order of the names that would have to be "updated" for everything to work flawlessly.

EDIT 3: This is where a stable and supported by the Rainmeter team JSON parser would come handy... 8-) Just change the names of the fields, without bothering with their order in the JSON source. Sure, this can be done using regex as well, but you'd have to waste resources by parsing the source string multiple times, once for each field (assuming they're unique).
OnyxBlack
Posts: 19
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

Hi guys,

I've been lurking on this thread hoping someone would post an updated, working measures file. That didn't happen so I went ahead and fixed it myself and registered so I could share my results.

A few things have changed:
  • Unit measures have been disabled.
    The new json doesn't include unit information (C°/F°, pressure, etc.) about the data, so I've commented out those measures. In the future this might change, so they can be re-enabled.
  • Unified the meaning of "Forecast[Day]Day..." measures.
    In the original measures, ForecastToday[Day/Night]Long would always be "today", where other days (Day1, Day2 etc) would return the full name of that day; Forecast...Short would return the abbreviated day name. The keys in the json this was based on have changed their value, so that presented an opportunity to make these more consistent:
    • Forecast[Day]DayLong: Always returns the full name of the day (Sunday, Monday, etc.)
    • Forecast[Day]DayShort: Always returns the abbreviated day name (Sun, Mon, Tue, etc.)
    • Forecast[Day]DayPart (new): Returns the day part name of the associated [Day] measures (Today, Tomorrow, Tuesday, Wednesday, etc.),
    • Forecast[Day]NightPart (new): Returns the night part name of the associated [Day] measures (Tonight, Tomorrow Night, Tuesday Night, etc.)
  • Added a 7th day forecast.
Apart from that, many key names in the json have changed, but I think I've associated all the right ones back with their measures.
If you're looking to work with the json further, note that now forecasts are bundled in arrays. Look at my regex to see how I split them up for extraction.

If you see any bugs or missing features, I'd appreciate contributions.

---------------
Notice: this is only for use if the original WeatherComJSONMeasures.inc from this thread aren't working for you.
---------------
Update (June 4, 16:00 GMT): Fixed forecast narrative, added Moon measures file.
---------------
WeatherComJSON.zip
You do not have the required permissions to view the files attached to this post.
Last edited by OnyxBlack on June 4th, 2020, 2:42 pm, edited 4 times in total.
User avatar
SilverAzide
Posts: 954
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the JSON

Post by SilverAzide »

Thanks! This is nice to have, but just be aware that this only works for people who are getting the "new" (V3?) version of the JSON. Most people seem to still be getting the old version, especially after doing the cookie-clearing stuff. Your code is great to have to create a "V3" version of a weather skin if weather.com starts sending that new format to everyone, or if you force the V3 using a user agent string for mobile.
Gadgets Wiki GitHub More Gadgets...
OnyxBlack
Posts: 19
Joined: June 3rd, 2020, 10:06 am

Re: ⭐ Weather.com - Parsing the JSON

Post by OnyxBlack »

SilverAzide wrote: June 3rd, 2020, 1:25 pm Thanks! This is nice to have, but just be aware that this only works for people who are getting the "new" (V3?) version of the JSON. Most people seem to still be getting the old version, especially after doing the cookie-clearing stuff. Your code is great to have to create a "V3" version of a weather skin if weather.com starts sending that new format to everyone, or if you force the V3 using a user agent string for mobile.
Sure, obviously if everything is working there's no reason to do anything. If it ain't broke, don't fix it :)
Although this will hopefully come in handy for those of us who, for whatever reason, are receiving the new source code from the webparser.
I'll update my post accordingly.