It is currently September 25th, 2020, 10:51 am

⭐ Weather.com - Parsing the V3 JSON - NEW!

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 21387
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Yincognito wrote: June 20th, 2020, 1:31 am I know, I know. :lol: That's ok, I'm just going to enjoy the result alone. ;-) I'm never swayed or discouraged by the fact that the rest choose a different path than mine, so no worries.


Yeah, you used that "end in tears" a couple of times when we had such conversations, and apparently it never ended in tears for me when you said it would. :D We'll just have to wait and see, when it happens, now won't we? I, for once, am ready - way readier than the last time this occured. Are you ready to rework the regex at every minor change they could make? 8-)
I'll cross that bridge when I come to it. Actually, the JSON code is not complicated at all, and pretty easy to read. The RegExp gets a little complicated, due to using Non-Capturing Groups in order to be able to just create the RegExp for one "day" and then just simply copy/paste it to all the others. However, the way I have worked this each element like IconCode:, windPhrase:, sunriseTimeLocal: and each and every other one all use exactly the same regular expression to capture them, so moving them around at least should be trivial. In that case, I just need to move the names around.

We will see how it goes. It's been stable for 2-3 weeks now, and I'm hopeful they are done for a while. If they decide to roll out a V4 at some point, I'll just quietly scream and get at it.
User avatar
Yincognito
Posts: 2612
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

jsmorley wrote: June 20th, 2020, 1:55 am I'll cross that bridge when I come to it. Actually, the JSON code is not complicated at all, and pretty easy to read. The RegExp gets a little complicated, due to using Non-Capturing Groups in order to be able to just create the RegExp for one "day" and then just simply copy/paste it to all the others. However, the way I have worked this each element like IconCode:, windPhrase:, sunriseTimeLocal: and each and every other one all use exactly the same regular expression to capture them, so moving them around at least should be trivial. In that case, I just need to move the names around.

We will see how it goes. It's been stable for 2-3 weeks now, and I'm hopeful they are done for a while. If they decide to roll out a V4 at some point, I'll just quietly scream and get at it.
Yep, there are some easily changeable things in your structure as well, so it's probably better than last time from this point of view. The V4 has a long way to go, I wouldn't worry about that for a couple of years. I mean, they just unified things on V3 (like some of us suspected right from the start), they won't switch to V4 yet, especially since I never saw it used for weather stuff yet - and I have looked in quite a lot of places, LOL.

So yeah, hold back your quiet scream on this for the time being - if changes happen, they won't be about V4. But changes will happen, as it's an IBM business for some time, and they probably want to put their mark on stuff and tweak them for their own benefit. ;-)
User avatar
jsmorley
Developer
Posts: 21387
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

So this was about as simple as I could make the regular expression to capture one "pair" of values for a day, the "day" and "night" values:

"iconCode":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]]

What it says is:

Search for the value type "iconCode":
Search for an \[ open bracket
Start a Non-Capturing Group (?:

Search for and skip over without capturing
Any number of any character .*
Followed by a character set [
Of a comma , OR | a close bracket \]
End the character set ]

End the Non-Capturing Group )
Do this skip-over 0 {0} times

Capture any number of any character (.*)
End the capture when a comma , is reached
Capture any number of any character (.*)
End the capture when a characters set [
Of a comma , OR | a close bracket \] is reached.
End the character set ]

This should properly account for the JSON format of:

"iconCode":[null,29,4,27,38,29,4,29,38,29,38,47,30,29,34,33,30,29,30,29,37,47,38,47,38,47,30,47,38,47]

Since the values are enclosed in [brackets], and are comma delimited, EXCEPT for the last one, the "night" part of the 15th day. Thus the reliance on [,|\]], a character set of either a comma , OR | a close bracket ] to end the searches/captures.

Again, Strings are enclosed in "quotes" and numbers are not, but that doesn't matter at all. Deal with the "quotes" later.

The number of {times} you skip-over without capturing gets you to the particular "day" you are interested in. So you skip to the first value for the given day, the "daytime" value, then capture it and the next one, the "nighttime" value.

Note that the first 8 types of data are for the entire day, and so there are only 15 entries each, not 30. Just don't treat those as "pairs". Just get one per day. Starting with cloudCover: it switches to "day" and "night" and there are 30 entries, in "pairs".

And you end up with:
RegExp=(?siU)"getSunV3DailyForecastUrlConfig":.*"duration:15day;.*"dayOfWeek":\[(?:.*[,|\]]){0}(.*)[,|\]].*"narrative":\[(?:.*[,|\]]){0}(.*)[,|\]].*"qpf":\[(?:.*[,|\]]){0}(.*)[,|\]].*"qpfSnow":\[(?:.*[,|\]]){0}(.*)[,|\]].*"sunriseTimeLocal":\[(?:.*[,|\]]){0}(.*)[,|\]].*"sunsetTimeLocal":\[(?:.*[,|\]]){0}(.*)[,|\]].*"temperatureMax":\[(?:.*[,|\]]){0}(.*)[,|\]].*"temperatureMin":\[(?:.*[,|\]]){0}(.*)[,|\]].*"cloudCover":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"dayOrNight":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"daypartName":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"iconCode":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"narrative":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"precipChance":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"precipType":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"qpf":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"qpfSnow":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"relativeHumidity":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"snowRange":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperature":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperatureHeatIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperatureWindChill":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"thunderCategory":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"thunderIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"uvDescription":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"uvIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windDirection":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windDirectionCardinal":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windPhrase":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windSpeed":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"wxPhraseLong":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"wxPhraseShort":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*
To get all the values, both day and night, for "today".
User avatar
JelleDekkers
Posts: 58
Joined: September 27th, 2017, 6:32 pm

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by JelleDekkers »

Adding "?temp=" to the URL makes it possible to manually choose the units :)
Possible values for the parameter are "c", "f", "m", or "e".
I found out by clicking the little "weather.com" link below the widget when searching for "weather" on Google.
Image
User avatar
jsmorley
Developer
Posts: 21387
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

JelleDekkers wrote: June 20th, 2020, 9:30 am Adding "?temp=" to the URL makes it possible to manually choose the units :)
Possible values for the parameter are "c", "f", "m", or "e".
I found out by clicking the little "weather.com" link below the widget when searching for "weather" on Google.
Image
Right. Thanks. This might be useful if you wanted a combination not supported by just using Language. Like French and Fahrenheit.
User avatar
Yincognito
Posts: 2612
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

jsmorley wrote: June 20th, 2020, 5:29 am So this was about as simple as I could make the regular expression to capture one "pair" of values for a day, the "day" and "night" values:

"iconCode":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]]

[...]

This should properly account for the JSON format of:

"iconCode":[null,29,4,27,38,29,4,29,38,29,38,47,30,29,34,33,30,29,30,29,37,47,38,47,38,47,30,47,38,47]

[...]

Again, Strings are enclosed in "quotes" and numbers are not, but that doesn't matter at all. Deal with the "quotes" later.
Yep, I guess it looks alright. In my case, besides dealing with the quotes later (which I do as well), I was also interested in making the "data" capture work whatever the nature of the data was, so that I wouldn't have to write / use another pattern for strings or whatever else the data would look like. That was the main reason why I ended up with what I posted earlier. The way you approach things, while perfectly fine, requires: a) knowing the kind of data you look for, and b) using a different pattern in the case of strings ... especially if those strings contain commas (like in the case of some of the narratives), because an ungreedy regex will stop at the first comma and discard the rest of the string. I know OnyxBlack already handled that and his method is probably fine as well, but I always like things to be the "one size fits all" type (however discredited this approach is nowadays), to eliminate redundancy and doing the same work again for a different scenario.

Anyway, that's just me, and by no means I try to convince anyone to do as I do - just wanted to clarify the reasoning behind it. As long as the stuff is functional, any method is valid - it's just that personally I don't like to work twice when I can work once and be done with it.
User avatar
jsmorley
Developer
Posts: 21387
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Yincognito wrote: June 20th, 2020, 12:54 pm Yep, I guess it looks alright. In my case, besides dealing with the quotes later (which I do as well), I was also interested in making the "data" capture work whatever the nature of the data was, so that I wouldn't have to write / use another pattern for strings or whatever else the data would look like. That was the main reason why I ended up with what I posted earlier. The way you approach things, while perfectly fine, requires using a different pattern in the case of strings ... especially if those strings contain commas (like in the case of some of the narratives), because an ungreedy regex will stop at the first comma and discard the rest of the string. I know OnyxBlack already handled that and his method is probably fine as well, but I always like things to be the "one size fits all" type (however discredited this approach is nowadays), to eliminate redundancy and doing the same work again for a different scenario.

Anyway, that's just me, and by no means I try to convince anyone to do as I do - just wanted to clarify the reasoning behind it. As long as the stuff is functional, any method is valid - it's just that personally I don't like to work twice when I can work once and be done with it.
Drat... You know, you are right. This isn't going to handle commas in the two possible "Narratives" per day. I'll have to tweak it again.
nikko
Posts: 23
Joined: December 5th, 2017, 5:58 pm

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by nikko »

thanx 4 answers. i got something not all. how to make this working that way in one regexp:
"precipChance":[null,100,10,60,60,80,80,80,100,80,80,60,40,50,50,50],"precipType":[null,"rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain"],
User avatar
jsmorley
Developer
Posts: 21387
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Edit June 20, 2020 - Tweaked the regular expression to account for the fact that there can be commas in several of the fields, like the "narrative" entries. Also added a missing WindPhrase field to the measures.

Get the .rmskin in the first post of this thread.
User avatar
jsmorley
Developer
Posts: 21387
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

nikko wrote: June 20th, 2020, 2:29 pm thanx 4 answers. i got something not all. how to make this working that way in one regexp:
"precipChance":[null,100,10,60,60,80,80,80,100,80,80,60,40,50,50,50],"precipType":[null,"rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain"],
I think the best thing is for you to take a look at the file WeatherComJSONMeasures.inc that is included with the .rmskin.