It is currently August 5th, 2020, 1:49 pm

⭐ Weather.com - Parsing the V3 JSON - NEW!

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
Yincognito
Posts: 2144
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

nikko wrote:
June 20th, 2020, 2:29 pm
thanx 4 answers. i got something not all. how to make this working that way in one regexp:
"precipChance":[null,100,10,60,60,80,80,80,100,80,80,60,40,50,50,50],"precipType":[null,"rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain","rain"],
But jsmorley's skin and his .inc files already get that. Why would you need to bother with the regexes yourself? :confused:
Take a look here and use the corresponding measures from the long quoted list in that post in your meters - that's all. For example, use [@ForecastDay2DayPrecipitationPercent] for the precipitation chance from the daytime part of the 2nd forecast day, and [@ForecastDay2DayPrecipitationType] for the precipitation type from the same period. Change the number from 2 to whatever day you want (max. 15) and if you want the values for the nighttime of that day, just replace the 2nd occurence of 'Day' in the measure name(s) with 'Night'.
User avatar
jsmorley
Developer
Posts: 21212
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Yes, the current RegExp to get all the different types of "forecast" information, for one day (Day 1 or "today" in this case) for both "day" and "night" looks like this:
RegExp=(?siU)"getSunV3DailyForecastUrlConfig":.*"duration:15day;.*"dayOfWeek":\[(?:.*[,|\]]){0}(.*)[,|\]].*"narrative":\[(?:null,|".*"[,|\]]){0}(null|".*")[,|\]].*"qpf":\[(?:.*[,|\]]){0}(.*)[,|\]].*"qpfSnow":\[(?:.*[,|\]]){0}(.*)[,|\]].*"sunriseTimeLocal":\[(?:.*[,|\]]){0}(.*)[,|\]].*"sunsetTimeLocal":\[(?:.*[,|\]]){0}(.*)[,|\]].*"temperatureMax":\[(?:.*[,|\]]){0}(.*)[,|\]].*"temperatureMin":\[(?:.*[,|\]]){0}(.*)[,|\]].*"cloudCover":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"dayOrNight":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"daypartName":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"iconCode":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"narrative":\[(?:null,|".*"[,|\]]){0}(null|".*")[,|\]](null|".*")[,|\]].*"precipChance":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"precipType":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"qpf":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"qpfSnow":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"relativeHumidity":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"snowRange":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperature":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperatureHeatIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"temperatureWindChill":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"thunderCategory":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"thunderIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"uvDescription":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"uvIndex":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windDirection":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windDirectionCardinal":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"windPhrase":\[(?:null,|".*"[,|\]]){0}(null|".*")[,|\]](null|".*")[,|\]].*"windSpeed":\[(?:.*[,|\]]){0}(.*),(.*)[,|\]].*"wxPhraseLong":\[(?:null,|".*"[,|\]]){0}(null|".*")[,|\]](null|".*")[,|\]].*"wxPhraseShort":\[(?:null,|".*"[,|\]]){0}(null|".*")[,|\]](null|".*")[,|\]]
This is why you probably DON'T really want to parse the JSON yourself, but just use the .inc @Include files I provide. This is not a wheel most of you are likely to enjoy re-inventing...
User avatar
jsmorley
Developer
Posts: 21212
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

I made a couple more small tweaks, be sure to get the latest .rmskin.
nikko
Posts: 23
Joined: December 5th, 2017, 5:58 pm

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by nikko »

tnanx for explanation y and js. one more question can you explain this most complex thing;
wxPhraseLong":\[(?:null,|".*"[,|\]]){2}(null|".*")[,|\]](null|".*")[,|\]].*
User avatar
jsmorley
Developer
Posts: 21212
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

nikko wrote:
June 22nd, 2020, 10:13 am
tnanx for explanation y and js. one more question can you explain this most complex thing;
wxPhraseLong":\[(?:null,|".*"[,|\]]){2}(null|".*")[,|\]](null|".*")[,|\]].*
I've gone over what most of those characters do here:

https://forum.rainmeter.net/viewtopic.php?f=118&t=34628&start=510#p178928

The difference is that I needed to account for the fact that there could be commas in several of the fields, and just capturing (.*),, or any number of any character followed by a comma, risks having the value be truncated if a comma, like "rain, followed by sun" is in the data.

So for several of the string fields, I search for the null static value, which is not surrounded by quotes, OR ".*", which gets everything between quotes. That is then ended by either a comma or a square bracket.

Regular expression is not particularly for the faint of heart, but it is actually pretty perfect for what it's intent is. It is as "terse" as it can possibly be, to keep the expressions short, while being incredibly powerful, with the tools and logic needed to parse just about anything. I certainly wouldn't want to make a career out of writing it, but it has stood the test of time admirably, with nothing that has ever challenged it.

The only way to understand what a particular expression does is to mentally "walk through it", putting what it is saying into logical phrases in your head. "search for this, followed by this or that, ending with this". Regular expression is probably the most purely "logical" exercise you will ever do in programming.

The first key to is is knowing what the reserved characters, the "vocabulary" of the thing, mean:

https://www.debuggex.com/cheatsheet/regex/pcre
That site loads kinda slow, but be patient, it's one of the best "cheatsheets".
User avatar
Yincognito
Posts: 2144
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

nikko wrote:
June 22nd, 2020, 10:13 am
tnanx for explanation y and js. one more question can you explain this most complex thing;
wxPhraseLong":\[(?:null,|".*"[,|\]]){2}(null|".*")[,|\]](null|".*")[,|\]].*
To add to what jsmorley said, besides the all known RegEx101 (which, despite being the most featured, has some things that I don't like regarding the small size of the "window docks" - having them adjustable doesn't compensate for that drawback, in my view), another online regex tester is RegExr, which I use almost exclusively to build and test my regular expressions. Once you set the regex "flavor" (PCRE instead of Javascript in our case, since this is used in Rainmeter) from the top right corner, you're good to go. For example, in the case of the regex you asked about, this is how it would look like (pay attention to what buttons / options / left-side sidebar menus are selected and the text I've written in the middle, as I intentionally left them that way):
RegExr.jpg
You do not have the required permissions to view the files attached to this post.
User avatar
jsmorley
Developer
Posts: 21212
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

Right, and the last note I would make is to remember that with the non-capture group (?: ...), when you specify an "instance" of the group to "skip to", using {n} as a "quantifier", the number in the {} is "zero-based", in that what you are saying is "skip this many". To get the first instance, you don't want to skip any, so the number is {0}. To get the second instance, you want to skip one, so the number is {1}.
User avatar
jsmorley
Developer
Posts: 21212
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by jsmorley »

In any case, it has been stable for 2-3 weeks now, and I'm cautiously hopeful.

I'm just keeping an eye on things, since the "sections" that are supported in the JSON are:

getSunV3LocationPointUrlConfig
getSunWeatherAlertHeadlinesUrlConfig
getSunV3HourlyForecastUrlConfig
getSunV3CurrentObservationsUrlConfig
getSunV3CurrentDateTimeByGeocodeUrlConfig
getSunV3DailyForecastUrlConfig
getSunV3HistoricalOneDayHourlyConditionsUrlConfig
etSunV3GlobalAirQualityUrlConfig
getSunIndexPollenDaypartUrlConfig
getSunV3SnowSkiConditionsUrlConfig
getSunIndexRunWeatherDaypartUrlConfig
getSunV3TidalPredictionsUrlConfig

And I'm curious to see if there is a change to

getSunIndexPollenDaypartUrlConfig

Which isn't titled with the "V3" yet, so I'm holding off on that for a bit to see if that gets an update as well.

Alerts are an entirely different animal, and honestly, I just couldn't care less about their opinion on "running weather", there is no "good" weather for running... but Pollen is of some interest to a lot of folks.
User avatar
Yincognito
Posts: 2144
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Yincognito »

jsmorley wrote:
June 22nd, 2020, 1:01 pm
And I'm curious to see if there is a change to

getSunIndexPollenDaypartUrlConfig

Which isn't titled with the "V3" yet, so I'm holding off on that for a bit to see if that gets an update as well.
I don't think you need to worry about that (besides the title name change, which is trivial to adapt to), since that section is already structured like all V3 ones (i.e. only one field name occurence and its values in an [array]). You could have worried if the structure was the "old type" (i.e. as much occurences of the field name as there are days / day parts, stuff present in {sections} instead of [arrays], etc).

As I said, these structures are basically matrices, where the V2 enumerated things "column by column", so to speak, and V3 is enumerating them "row by row":

-------------------------Day 1------------------Day 2-----------------Day 3-------
Field 1---------ValueDay1Field1-----ValueDay2Field1-----ValueDay3Field1
Field 2---------ValueDay1Field2-----ValueDay2Field2-----ValueDay3Field2
Field 3---------ValueDay1Field3-----ValueDay2Field3-----ValueDay3Field3
User avatar
Defrilitus
Posts: 1
Joined: February 1st, 2020, 4:17 pm

Re: ⭐ Weather.com - Parsing the V3 JSON

Post by Defrilitus »

I have been a Rainmeter user for nearly fifteen years, but must admit that the last weather.com update left me completely nonplussed. I would therefore like, first of all, to thank JS Morley & Co for all their (successful) efforts to make the new JSON weather version accessible to all. I freely admit that, without JS Morley, I would have been completely lost ;-)

Considering the above, I would like to give something back to the community and I remember that, some months ago, there was a question about time zones.

I like to see the interval between the last update and the current local time (pure laziness, of course!). This is, of course, simple within the same time-zone; but becomes more difficult when different time-zones are involved (every so often I like to check the weather in another part of the world).

The alphabetic time-zone abbreviations provided by weather.com (e.g., “EDT”) are absolutely useless, as they can mean anything and everything. I therefore extract the numeric time-zone (e.g., +0200) from each weather.com forecast page that I consult, and use this to calculate the time elapsed between the last update and my (local) system time.

My solution may be somewhat clunky (regular expressions are not my cup of tea), but it works; and I think that showing a location's time-zone is important. By the way, I would be happy to give any explanations needed; in the meantime, please feel free to use/adapt/improve as you wish.

One caveat: I have no idea how the measures/meters below would work with American formatting (e.g., "am").

Code: Select all

;WeatherComJSONMeasures

[@LocationTimeZone]
Measure=WebParser
URL=[@CurrentConditionsParent]
StringIndex=17
RegExpSubstitute=1
Substitute="+0000":"±0000","^.*(......)$":"\1",'"':""

; Measures

[@LocationTimeAdjusted]
; Location time adjusted for calculations
Measure=Time
TimeStamp=[@CurrentObservationTimeStamp]
TimeStampFormat=%Y-%m-%dT%H:%M
DynamicVariables=1

[@LocationZoneCalc]
; Location time-zone formatted for calculations
Measure=Calc
Formula=([@LocationTimeZone] * 0.01)
DynamicVariables=1
Substitute=".15":".25",".3":".5",".45":".75","±0000":"0"

[@LocalTimeAdjusted]
; Local (system) time adjusted for calculations using location time-zone
Measure=Time
TimeZone=[@LocationZoneCalc]
Format=%Y-%m-%dT%H:%M:%S
DynamicVariables=1

[@TimeDifference]
; Difference between location time and local (system) time, calculated using location time-zone for both
Measure=Calc
Formula=([@LocalTimeAdjusted:TimeStamp]-[@LocationTimeAdjusted:TimeStamp])
; Change MeterTimeElapsed text if time difference > 24h
IfAboveValue=86399
IfAboveAction=[!SetOption MeterTimeElapsed Text "> 24h"]
; Otherwise keep things as they are
IfBelowValue=86400
IfBelowAction=[!SetOption MeterTimeElapsed Text "%1"]
DynamicVariables=1

[@TimeElapsed]
; @TimeDifference formatted for display
Measure=Time
TimeStamp=[@TimeDifference]
Format=%#Hh %#Mm
DynamicVariables=1
RegExpSubstitute=1
Substitute="^0h 0m":"< 1m","^0h ":""

; Meters

[SeparatorUpdate]
Meter=IMAGE
MeterStyle=sSeparator

[MeterUpdateString]
Meter=STRING
Text="Last update"

[MeterUpdate]
MeasureName=@CurrentObservationTime
Meter=String
Text=%1

[SeparatorTimeElapsed]
Meter=IMAGE
MeterStyle=sSeparator

[MeterTimeElapsedString]
Meter=STRING
Text="Time elapsed"

[MeterTimeElapsed]
MeasureName=@TimeElapsed
Meter=String

[SeparatorTimeZone]
Meter=IMAGE
MeterStyle=sSeparator

[MeterTimeZoneString]
Meter=STRING
Text="Time zone"

[MeterTimeZone]
MeasureName=@LocationTimeZone
Meter=String
Thanks again for all your help.
Last edited by Defrilitus on July 1st, 2020, 8:22 pm, edited 2 times in total.