It is currently September 24th, 2021, 3:08 pm

⭐ Weather.com - Parsing the V3 JSON - NEW!

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 22310
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

I know you are looking to build something that in theory won't break if they change the JSON, but I'm just not going to go on that journey with you... ;-)

If they change the JSON again, it won't be something simple like the order of the fields, or adding a space where there wasn't one before. If they change it again, I'm going to have to just get a dump, look at it, and rework the RegExp. I'm resigned to that, as I think trying to predict how they might change it is doomed to failure, and can only end in tears.
User avatar
jsmorley
Developer
Posts: 22310
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Ok...

Edit June 19, 2020 - Simplified the RegExp for "forecast" somewhat. This is just to make the regular expression a bit simpler and hopefully easier to follow, and doesn't impact any functionality. Also, re-ordered the measures in the "current" section, to make it a bit easier to deal with in About/Skins.

Get the .rmskin in the first post of this thread
User avatar
SilverAzide
Rainmeter Sage
Posts: 1564
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by SilverAzide »

OnyxBlack wrote: June 19th, 2020, 11:34 pm:confused:
jsmorley wrote: June 19th, 2020, 11:40 pm Trust me, I'm 99% in love with it... :thumbup:
...And I'm 110% in love with the fact that it all seems stable and working again. Massive thanks!
Gadgets Wiki GitHub More Gadgets...
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

jsmorley wrote: June 20th, 2020, 12:40 amI know you are looking to build something that in theory won't break if they change the JSON, but I'm just not going to go on that journey with you... ;-)
I know, I know. :lol: That's ok, I'm just going to enjoy the result alone. ;-) I'm never swayed or discouraged by the fact that the rest choose a different path than mine, so no worries.
jsmorley wrote: June 20th, 2020, 12:40 amIf they change the JSON again, it won't be something simple like the order of the fields, or adding a space where there wasn't one before. If they change it again, I'm going to have to just get a dump, look at it, and rework the RegExp. I'm resigned to that, as I think trying to predict how they might change it is doomed to failure, and can only end in tears.
Yeah, you used that "end in tears" a couple of times when we had such conversations, and apparently it never ended in tears for me when you said it would. :D We'll just have to wait and see, when it happens, now won't we? I, for once, am ready - way readier than the last time this occured. Are you ready to rework the regex at every minor change they could make? 8-)
User avatar
eclectic-tech
Rainmeter Sage
Posts: 4390
Joined: April 12th, 2012, 9:40 pm
Location: Cedar Point, Ohio, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by eclectic-tech »

SilverAzide wrote: June 20th, 2020, 1:25 am ...And I'm 110% in love with the fact that it all seems stable and working again. Massive thanks!
DITTO! :rosegift:
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

SilverAzide wrote: June 20th, 2020, 1:25 am ...And I'm 110% in love with the fact that it all seems stable and working again. Massive thanks!
I'm in love with the fact that OnyxBlack used the type of quantifiers that saved my regexes from a ton of otherwise impossible situations over the past years. Mine are a bit different, like {0,N}+ instead of just {N}. I used (and still use) them in all kinds of places in my skins, not just in the weather one. They are simply invaluable for (dynamic or not) iteration over a known or unknown quantity of items.

My sincere congratulations! Not just to him, but to everyone involved. Including you, SilverAzide. :D
User avatar
jsmorley
Developer
Posts: 22310
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Yincognito wrote: June 20th, 2020, 1:31 am I know, I know. :lol: That's ok, I'm just going to enjoy the result alone. ;-) I'm never swayed or discouraged by the fact that the rest choose a different path than mine, so no worries.


Yeah, you used that "end in tears" a couple of times when we had such conversations, and apparently it never ended in tears for me when you said it would. :D We'll just have to wait and see, when it happens, now won't we? I, for once, am ready - way readier than the last time this occured. Are you ready to rework the regex at every minor change they could make? 8-)
I'll cross that bridge when I come to it. Actually, the JSON code is not complicated at all, and pretty easy to read. The RegExp gets a little complicated, due to using Non-Capturing Groups in order to be able to just create the RegExp for one "day" and then just simply copy/paste it to all the others. However, the way I have worked this each element like IconCode:, windPhrase:, sunriseTimeLocal: and each and every other one all use exactly the same regular expression to capture them, so moving them around at least should be trivial. In that case, I just need to move the names around.

We will see how it goes. It's been stable for 2-3 weeks now, and I'm hopeful they are done for a while. If they decide to roll out a V4 at some point, I'll just quietly scream and get at it.
User avatar
Yincognito
Rainmeter Sage
Posts: 4052
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

jsmorley wrote: June 20th, 2020, 1:55 am I'll cross that bridge when I come to it. Actually, the JSON code is not complicated at all, and pretty easy to read. The RegExp gets a little complicated, due to using Non-Capturing Groups in order to be able to just create the RegExp for one "day" and then just simply copy/paste it to all the others. However, the way I have worked this each element like IconCode:, windPhrase:, sunriseTimeLocal: and each and every other one all use exactly the same regular expression to capture them, so moving them around at least should be trivial. In that case, I just need to move the names around.

We will see how it goes. It's been stable for 2-3 weeks now, and I'm hopeful they are done for a while. If they decide to roll out a V4 at some point, I'll just quietly scream and get at it.
Yep, there are some easily changeable things in your structure as well, so it's probably better than last time from this point of view. The V4 has a long way to go, I wouldn't worry about that for a couple of years. I mean, they just unified things on V3 (like some of us suspected right from the start), they won't switch to V4 yet, especially since I never saw it used for weather stuff yet - and I have looked in quite a lot of places, LOL.

So yeah, hold back your quiet scream on this for the time being - if changes happen, they won't be about V4. But changes will happen, as it's an IBM business for some time, and they probably want to put their mark on stuff and tweak them for their own benefit. ;-)
User avatar
jsmorley
Developer
Posts: 22310
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

So this was about as simple as I could make the regular expression to capture one "pair" of values for a day, the "day" and "night" values:

"iconCode":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]]

What it says is:

Search for the value type "iconCode":
Search for an \[ open bracket
Start a Non-Capturing Group (?:

Search for and skip over without capturing
Any number of any character .*
Followed by a character set [
Of a comma , OR | a close bracket \]
End the character set ]

End the Non-Capturing Group )
Do this skip-over 0 {0} times

Capture any number of any character (.*)
End the capture when a comma , is reached
Capture any number of any character (.*)
End the capture when a characters set [
Of a comma , OR | a close bracket \] is reached.
End the character set ]

This should properly account for the JSON format of:

"iconCode":[null,29,4,27,38,29,4,29,38,29,38,47,30,29,34,33,30,29,30,29,37,47,38,47,38,47,30,47,38,47]

Since the values are enclosed in [brackets], and are comma delimited, EXCEPT for the last one, the "night" part of the 15th day. Thus the reliance on [,|\]], a character set of either a comma , OR | a close bracket ] to end the searches/captures.

Again, Strings are enclosed in "quotes" and numbers are not, but that doesn't matter at all. Deal with the "quotes" later.

The number of {times} you skip-over without capturing gets you to the particular "day" you are interested in. So you skip to the first value for the given day, the "daytime" value, then capture it and the next one, the "nighttime" value.

Note that the first 8 types of data are for the entire day, and so there are only 15 entries each, not 30. Just don't treat those as "pairs". Just get one per day. Starting with cloudCover: it switches to "day" and "night" and there are 30 entries, in "pairs".

And you end up with:
RegExp=(?siU)"getSunV3DailyForecastUrlConfig":.*"duration:15day;.*"dayOfWeek":\[(?:.*[,|\]]){0}(.*)[,|\]].*"narrative":\[(?:.*[,|\]]){0}(.*)[,|\]].*"qpf":\[(?:.*[,|\]]){0}(.*)[,|\]].*"qpfSnow":\[(?:.*[,|\]]){0}(.*)[,|\]].*"sunriseTimeLocal":\[(?:.*[,|\]]){0}(.*)[,|\]].*"sunsetTimeLocal":\[(?:.*[,|\]]){0}(.*)[,|\]].*"temperatureMax":\[(?:.*[,|\]]){0}(.*)[,|\]].*"temperatureMin":\[(?:.*[,|\]]){0}(.*)[,|\]].*"cloudCover":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"dayOrNight":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"daypartName":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"iconCode":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"narrative":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"precipChance":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"precipType":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"qpf":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"qpfSnow":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"relativeHumidity":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"snowRange":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperature":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperatureHeatIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperatureWindChill":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"thunderCategory":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"thunderIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"uvDescription":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"uvIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windDirection":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windDirectionCardinal":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windPhrase":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windSpeed":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"wxPhraseLong":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"wxPhraseShort":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*
To get all the values, both day and night, for "today".
User avatar
JelleDekkers
Posts: 107
Joined: September 27th, 2017, 6:32 pm
Location: Netherlands

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by JelleDekkers »

Adding "?temp=" to the URL makes it possible to manually choose the units :)
Possible values for the parameter are "c", "f", "m", or "e".
I found out by clicking the little "weather.com" link below the widget when searching for "weather" on Google.
Image