It is currently March 29th, 2024, 8:18 am

⭐ Weather.com - Parsing the V3 JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Seems to me that I need to stop using those "pairs" entirely, and just individually capture each value in the "array" as a separate entity, and then just pass the day/night indicator, "D" or "N" as a part of the values.

If the very first indicator is "N" instead of "D", then I just disable the final measure so I don't get an error, or use a lookahead on it.
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

jsmorley wrote: August 3rd, 2020, 10:28 pmI'd prefer to keep the approach the same from the standpoint of the end user, where "today" just returns null values that can be reacted to with IfMatch. This is going to require some thought on how I force in null values for "today" and then push the others forward by one, and deal with the missing data at the end of the parsing.
As long as this doesn't hang Rainmeter due to empty capture (shouldn't happen for WebParser though, only for String measures), it should be a starting point:

Code: Select all

RegExp=(?siU)(?|.*"dayInd":\["D".*"grassPollenIndex":\[(?:.*[,|\]]){0}(.*),|())
Tested with a string similar to what pbutler6 posted.

EDIT: Just read your last posts, and yes, I concur. Treating the field values individually is best. I already do that, but well, our approaches are a bit different, so it isn't relevant.
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Yeah, I'm starting to think that the best approach, although it's a bit different for the skin author than the other data is to:

capture each individual element of each "array" as a separate entity. Add captures for the date, day/night indicator (D/N), and the day name value (today/tonight/tomorrow/tomorrow night/thursday/thursday night) and create measures for those.

I also think I will change this to a 14day version of the .inc, so it will always return 0-26 values and won't ever "run out of" data. It might start with "day" and end with "night" or start with "night" and end with "day", but it will be a consistent number of values.

I wish there was a V3 version of this, although I don't see one. The old V2 version of this stuff was I think tailored to the "cards" that used to be on the actual website, that worked like that. During the day it started with "today/tonight" and at night it started with "tonight/tomorrow". I'm betting a V3 version this data would be indexed the same as the other V3 stuff.
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

jsmorley wrote: August 3rd, 2020, 11:40 pm Yeah, I'm starting to think that the best approach, although it's a bit different for the skin author than the other data is to:

capture each individual element of each "array" as a separate entity. Add captures for the date, day/night indicator (D/N), and the day name value (today/tonight/tomorrow/tomorrow night/thursday/thursday night) and create measures for those.

Have at least the 30th capture be a (?(?= lookahead) so if they decide to use the full 15day .inc file, they don't get an error.

I wish there was a V3 version of this, although I don't see one. The old V2 version of this stuff was I think tailored to the "cards" that used to be on the actual website, that worked like that. During the day it started with "today/tonight" and at night it started with "tonight/tomorrow". I'm betting a V3 version this data would be indexed the same as the other V3 stuff.
I don't use lookaheads for this, I just take how many groups are available through the (?<group>){0,N}+ approach:

Code: Select all

(?siU)"grassPollenIndex":\[(?:.*[,|\]]){0,29}+([^"\{\[\]\}]*)[,|\]]
which automatically makes non existing captures blank, but then I use it on String measures and have empty capture failsafe mechanisms, like replacing stuff with _\1_ or <\1> instead of simply \1 (so that the result isn't empty, even if the capture is), clearing the garbage afterwards. You'd probably be more comfortable with lookaheads though - I just shared this as an alternative, that's all.

Yeah, I tried the V3 approach on pollen, but just couldn't nail the right section name (couldn't use V1 so I settled for V2). As I said, it takes some attempts and permutations (if the thing is available in V3 for this API Key, that is) to find the correct URL syntax, as the documentation, besides being "confidential", is quite sparse on these things...

I'm not sure it would change things if a V3 was available and known by us. Maybe it's a pollen thing, after all. :confused:
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Capturing the values that have distinct "day/night" components was mostly done in "pairs" like that for the purpose of organizing the measures into "days" in the .inc file, so they were displayed that way in About / Skins. It just made things a bit easier to follow when creating a skin.

There is no other particular value in pairing up the values like that.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

As to any lookaheads, I'm now leaning toward this:
I also think I will change this to a 14day version of the .inc, so it will always return 0-26 values and won't ever "run out of" data. It might start with "day" and end with "night" or start with "night" and end with "day", but it will be a consistent number of values.
User avatar
Yincognito
Rainmeter Sage
Posts: 7029
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by Yincognito »

jsmorley wrote: August 4th, 2020, 12:09 am As to any lookaheads, I'm now leaning toward this:
As you wish. :D Personally, I don't mind non-existing values at all, I just substitute ^$ with a default no data string in each WebParser child, e.g. "(?i)(?:^$|^N\/A$|^null$)":"Not Available", and be done with it.

In fact, I intentionally set my "forecast days" variable to 16 (i.e. greater than 15) just to make sure making stuff blank / default works in my implementation. But then, as you no doubt noticed, my methods aren't suited for anyone, so in the end it's each skin designer's choice how to handle such scenarios.
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

Yincognito wrote: August 4th, 2020, 12:18 am As you wish. :D Personally, I don't mind non-existing values at all, I just substitute ^$ with a default no data string in each WebParser child, e.g. "(?i)(?:^$|^N\/A$|^null$)":"Not Available", and be done with it.

In fact, I intentionally set my "forecast days" variable to 16 (i.e. greater than 15) just to make sure making stuff blank / default works in my implementation. But then, as you no doubt noticed, my methods aren't suited for anyone, so in the end it's each skin designer's choice how to handle such scenarios.
Yes, but it's not about the final result, it's about the error in the log when you attempt to parse a value that doesn't exist. If I try to parse that 30th value from the array, which will be there 1/2 of each day, that will be a problem. A lookahead solves that, but I don't see that it helps the user to have an "empty" 30th value, when in fact it is the expected 1st value that is really missing.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

In case anyone here hasn't tumbled to this yet, if you are writing a skin based on these Pollen .inc files, STOP. I should have something new by tomorrow sometime, but it won't in any way be the same.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Weather.com - Parsing the V3 JSON - NEW!

Post by jsmorley »

So I'm looking at keeping it simple.

Code: Select all

; =================================================================================================
; WeatherComJSONPollen.inc by JSMorley
; "Weather.com - Parsing the JSON" (https://forum.rainmeter.net/viewtopic.php?f=118&t=34628)
;
; Requires that WeatherComJSONVariables.inc be @Included in the skin before this file.
;
; Updated August 4, 2020
; ==================================================================================================

[@PollenSuperParent]
Measure=WebParser
URL=https://api.weather.com/v2/indices/pollen/daypart/15day?geocode=#Latitude#,#Longitude#&language=#Language#&format=json&apiKey=#APIKey#
UpdateRate=#UpdateRate#
Flags=Resync | NoCookies
UserAgent=#UserAgent#
LogSubstringErrors=0
StringIndex=1
RegExp=(?siU)^(.*)$
FinishAction=[!EnableMeasureGroup Parents]

; ====================================================
; Create Parent measures for the various data elements
; ====================================================

[@PollenValidDateStampsParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"fcstValidLocal":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)
FinishAction=[!EnableMeasureGroup Times]

[@PollenDayPartIndicatorParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"dayInd":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)
FinishAction=[!EnableMeasureGroup Times]

[@PollenDayPartNameParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"daypartName":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)

[@PollenGrassIndexParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"grassPollenIndex":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)

[@PollenGrassCategoryParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"grassPollenCategory":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)

[@PollenTreeIndexParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"treePollenIndex":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)

[@PollenTreeCategoryParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"treePollenCategory":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)

[@PollenRagweedIndexParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"ragweedPollenIndex":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)

[@PollenRagweedCategoryParent]
Measure=WebParser
Group=Parents
Disabled=1
URL=[@PollenSuperParent]
RegExp=(?siU)"ragweedPollenCategory":\[(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*),(.*)

; ====================================================
; Part 1
; ====================================================

[@PollenPart1ValidDateStamp]
Measure=WebParser
URL=[@PollenValidDateStampsParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1ValidWeekDay]
Measure=Time
Group=Times
Disabled=1
TimeStamp=[@PollenPart1ValidDateStamp]
DynamicVariables=1
TimeStampFormat=%Y-%m-%dT%H:%M:%S
FormatLocale=#Language#
Format=%A

[@PollenPart1DayPartIndicator]
Measure=WebParser
URL=[@PollenDayPartIndicatorParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1DayPartName]
Measure=WebParser
URL=[@PollenDayPartNameParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1GrassIndex]
Measure=WebParser
URL=[@PollenGrassIndexParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1GrassCategory]
Measure=WebParser
URL=[@PollenGrassCategoryParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1TreeIndex]
Measure=WebParser
URL=[@PollenTreeIndexParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1TreeCategory]
Measure=WebParser
URL=[@PollenTreeCategoryParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1RagweedIndex]
Measure=WebParser
URL=[@PollenRagweedIndexParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart1RagweedCategory]
Measure=WebParser
URL=[@PollenRagweedCategoryParent]
StringIndex=1
RegExpSubstitute=1
Substitute=#CommonSubstitute#

; ====================================================
; Part 2
; ====================================================

[@PollenPart2ValidDateStamp]
Measure=WebParser
URL=[@PollenValidDateStampsParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2ValidWeekDay]
Measure=Time
Group=Times
Disabled=1
TimeStamp=[@PollenPart2ValidDateStamp]
DynamicVariables=1
TimeStampFormat=%Y-%m-%dT%H:%M:%S
FormatLocale=#Language#
Format=%A

[@PollenPart2DayPartIndicator]
Measure=WebParser
URL=[@PollenDayPartIndicatorParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2DayPartName]
Measure=WebParser
URL=[@PollenDayPartNameParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2GrassIndex]
Measure=WebParser
URL=[@PollenGrassIndexParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2GrassCategory]
Measure=WebParser
URL=[@PollenGrassCategoryParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2TreeIndex]
Measure=WebParser
URL=[@PollenTreeIndexParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2TreeCategory]
Measure=WebParser
URL=[@PollenTreeCategoryParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2RagweedIndex]
Measure=WebParser
URL=[@PollenRagweedIndexParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#

[@PollenPart2RagweedCategory]
Measure=WebParser
URL=[@PollenRagweedCategoryParent]
StringIndex=2
RegExpSubstitute=1
Substitute=#CommonSubstitute#
All those "child" measures will be repeated 29 times. That way the missing 30th index at night won't factor in.


1.jpg
You do not have the required permissions to view the files attached to this post.