⭐ Weather.com - Parsing the V3 JSON

[jsmorley]

I was looking at the new Pollen files and noticed that the UserAgent was for Firefox 78 while my browser was on 79 (I assume jsmorley 's is also updated) and this started me thinking the a variable #userAgent# in the WeatherComJSONVariables.inc file might be a good idea.

Between the expanding number of .inc files, the possibility of a security problem in a specific version of the browser, and just wanting to keep the userAgent up to date with the latest and greatest version being used by the user I think that this might be a way to set up for an easy fix in the future if this problem arose.

in any case I am simply agog and appreciative of al the work and problem solving involved in getting the weather feed up and running. This makes my transition from DarkSky to weather.com much easier and I thank you.

There is literally no reason to keep the UserAgent string up to date or any particular version. The entire point of it, in this case, is just to ensure that what the site returns to WebParser is the same thing the site returns to a reasonably modern browser. It just makes it easier to be sure everyone is on the same page (pun intended) when there are problems we are trying to solve.

I actually think this was more relevant when we were parsing the embedded JSON from the actual website. Now that we are getting the JSON directly, well, JSON is JSON isn't it... I'm not entirely sure that that UserAgent option still serves any useful purpose, although it certainly can't hurt.

However, fair enough. I'm not opposed in principle to centralizing where that value is stored, and will likely take your advice the next time I update things. That way, if you do want to change it for some reason, you will only have to do it one place.

[thohan]

I actually think this was more relevant when we were parsing the embedded JSON from the actual website. Now that we are getting the JSON directly, well, JSON is JSON isn't it... I'm not entirely sure that that UserAgent option still serves any useful purpose, although it certainly can't hurt.

However, fair enough. I'm not opposed in principle to centralizing where that value is stored, and will likely take your advice the next time I update things. That way, if you do want to change it for some reason, you will only have to do it one place.

For some reason I had momentarily forgotten that we were getting the JSON more directly from the source so my thought about having to keep the useragent up to date is less meaningful.

Nevertheless, Thank you !!!

[jsmorley]

Moved the UserAgent value into WeatherComJSONVariables.inc as a variable.

This is a minor change that doesn't effect functionality, and there is no real reason to rush to update all your skins with these new .inc files.

I honestly don't think it is going to matter what you set this to, although I set it to, and recommend, some modern desktop browser:

Code: Select all

UserAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0

[SCR]

I have added .inc files for up to 15 days of "Pollen forecasts"

WeatherComJSONPollen.inc
WeatherComJSONPollen10Day.inc
WeatherComJSONPollen7Day.inc
WeatherComJSONPollen5Day.inc
WeatherComJSONPollen3Day.inc

This forecasts values for Grass / Tree / Ragweed pollen levels.

I also corrected an issue with WeatherComJSONAlerts.inc that would produce an error in the log if there were no current alerts on the site.

Get the new .rmskin in the first post of this tread.

Terrific, I have allergies to all three. Anything above 2 tells me to stay inside if at all possible.

Thank you.

[pbutler6]

I'm looking at your new pollen inc files. It looks like you are coding the first data field of the index as "Today", the second field as "Tonight", etc. The JSON data seems to change about 3 PM to make the first field "Tonight", the second field "Tomorrow", etc.

Am I missing something?

{"metadata":{"language":"en-US","transactionId":"1596488108238:1369678634","version":"1","latitude":38.71,"longitude":-77.07,"expireTimeGmt":1596489768,"statusCode":200},"pollenForecast12hour":{"fcstValid":[1596495600,1596538800,1596582000,1596625200,1596668400,1596711600,1596754800,1596798000,1596841200,1596884400,1596927600,1596970800,1597014000,1597057200,1597100400,1597143600,1597186800,1597230000,1597273200,1597316400,1597359600,1597402800,1597446000,1597489200,1597532400,1597575600,1597618800,1597662000,1597705200],"fcstValidLocal":["2020-08-03T19:00:00-0400","2020-08-04T07:00:00-0400","2020-08-04T19:00:00-0400","2020-08-05T07:00:00-0400","2020-08-05T19:00:00-0400","2020-08-06T07:00:00-0400","2020-08-06T19:00:00-0400","2020-08-07T07:00:00-0400","2020-08-07T19:00:00-0400","2020-08-08T07:00:00-0400","2020-08-08T19:00:00-0400","2020-08-09T07:00:00-0400","2020-08-09T19:00:00-0400","2020-08-10T07:00:00-0400","2020-08-10T19:00:00-0400","2020-08-11T07:00:00-0400","2020-08-11T19:00:00-0400","2020-08-12T07:00:00-0400","2020-08-12T19:00:00-0400","2020-08-13T07:00:00-0400","2020-08-13T19:00:00-0400","2020-08-14T07:00:00-0400","2020-08-14T19:00:00-0400","2020-08-15T07:00:00-0400","2020-08-15T19:00:00-0400","2020-08-16T07:00:00-0400","2020-08-16T19:00:00-0400","2020-08-17T07:00:00-0400","2020-08-17T19:00:00-0400"],"dayInd":["N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N"],"num":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29],"daypartName":["Tonight","Tomorrow","Tomorrow night","Wednesday","Wednesday night","Thursday","Thursday night","Friday","Friday night","Saturday","Saturday night","Sunday","Sunday night","Monday","Monday night","Tuesday","Tuesday night","Wednesday","Wednesday night","Thursday","Thursday night","Friday","Friday night","Saturday","Saturday night","Sunday","Sunday night","Monday","Monday night"],"grassPollenIndex":[1,1,0,1,1,1,1,1,1,1,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"grassPollenCategory":["Low","Low","None","Low","Low","Low","Low","Low","Low","Low","Moderate","Moderate","Moderate","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low"],"treePollenIndex":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"treePollenCategory":["None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None"],"ragweedPollenIndex":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"ragweedPollenCategory":["None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None"]}

[Yincognito]

I'm looking at your new pollen inc files. It looks like you are coding the first data field of the index as "Today", the second field as "Tonight", etc. The JSON data seems to change about 3 PM to make the first field "Tonight", the second field "Tomorrow", etc.

Am I missing something?

{"metadata":{"language":"en-US","transactionId":"1596488108238:1369678634","version":"1","latitude":38.71,"longitude":-77.07,"expireTimeGmt":1596489768,"statusCode":200},"pollenForecast12hour":{"fcstValid":[1596495600,1596538800,1596582000,1596625200,1596668400,1596711600,1596754800,1596798000,1596841200,1596884400,1596927600,1596970800,1597014000,1597057200,1597100400,1597143600,1597186800,1597230000,1597273200,1597316400,1597359600,1597402800,1597446000,1597489200,1597532400,1597575600,1597618800,1597662000,1597705200],"fcstValidLocal":["2020-08-03T19:00:00-0400","2020-08-04T07:00:00-0400","2020-08-04T19:00:00-0400","2020-08-05T07:00:00-0400","2020-08-05T19:00:00-0400","2020-08-06T07:00:00-0400","2020-08-06T19:00:00-0400","2020-08-07T07:00:00-0400","2020-08-07T19:00:00-0400","2020-08-08T07:00:00-0400","2020-08-08T19:00:00-0400","2020-08-09T07:00:00-0400","2020-08-09T19:00:00-0400","2020-08-10T07:00:00-0400","2020-08-10T19:00:00-0400","2020-08-11T07:00:00-0400","2020-08-11T19:00:00-0400","2020-08-12T07:00:00-0400","2020-08-12T19:00:00-0400","2020-08-13T07:00:00-0400","2020-08-13T19:00:00-0400","2020-08-14T07:00:00-0400","2020-08-14T19:00:00-0400","2020-08-15T07:00:00-0400","2020-08-15T19:00:00-0400","2020-08-16T07:00:00-0400","2020-08-16T19:00:00-0400","2020-08-17T07:00:00-0400","2020-08-17T19:00:00-0400"],"dayInd":["N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N","D","N"],"num":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29],"daypartName":["Tonight","Tomorrow","Tomorrow night","Wednesday","Wednesday night","Thursday","Thursday night","Friday","Friday night","Saturday","Saturday night","Sunday","Sunday night","Monday","Monday night","Tuesday","Tuesday night","Wednesday","Wednesday night","Thursday","Thursday night","Friday","Friday night","Saturday","Saturday night","Sunday","Sunday night","Monday","Monday night"],"grassPollenIndex":[1,1,0,1,1,1,1,1,1,1,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"grassPollenCategory":["Low","Low","None","Low","Low","Low","Low","Low","Low","Low","Moderate","Moderate","Moderate","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low","Low"],"treePollenIndex":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"treePollenCategory":["None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None"],"ragweedPollenIndex":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"ragweedPollenCategory":["None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None","None"]}

Good catch. For some reason, pollen data seems to exhibit a slightly different behavior compared to the "regular weather" one. While the latter was ok with making the first field value null after 3 PM, the former just removes that value entirely - notice that there are only 29 values, not 30 (see the num field). Normally it shouldn't be difficult to manage this if one avoids messing again with the measures - changing the referenced measures in the meters themselves would suffice. Unfortunately the used naming convention doesn't make this an easy task to automate, since an index based naming system (e.g. day0part0, day0part1, day1part0, day1part1, etc. instead of todays, tomorrows, tonights and such) would have made this a question of getting a DIV (e.g. trunc(index/2), for the day number) and a MOD (e.g. index%2 for the day part number) from a "moment index" value incremented (or decremented) by an offset (in this case 1, i.e. a single day part to "skip"). With the current naming convention, doing it manually for every referenced measure involved is probably the uncomfortable, but possible solution.

That being said, maybe jsmorley has a different idea on how to handle this - I would be curious to see what that is.

[jsmorley]

Well bummer...

I'll have to chew on this.

I'd prefer to keep the approach the same from the standpoint of the end user, where "today" just returns null values that can be reacted to with IfMatch. This is going to require some thought on how I force in null values for "today" and then push the others forward by one, and deal with the missing data at the end of the parsing.

[jsmorley]

That might just be too difficult. I guess I will have to get and return the day part "name" like "today" and "tonight", and the skin will have to be written so that those are a part of the display.

So the measures will just be something like [@PollenDay0Part1GrassIndex]

[Yincognito]

I'd prefer to keep the approach the same from the standpoint of the end user, where "today" just returns null values that can be reacted to with IfMatch. This is going to require some thought on how I force in null values for "today" and then push the others forward by one, and deal with the missing data at the end of the parsing.

Well, maybe, just maybe, the missing data at the end could remain missing data and never be displayed in a meter if the "dayInd":["N" exists in the string, then:
- separate the RegExp option for today and tonight in your .inc
- use a regex conditional to test for the existence of "dayInd":["N" and capture nothing if it does or capture normally if it doesn't, e.g. (?(?=.*"dayInd":\["N")|<capture the 0th index(es) normally here>)

That would only require changes on the today and tonight measures (well, tonight would just have to be index 1 and be separated from today), the rest would be unaffected (unless you insist on handling the missing data at the end, that is). Basically, the today measures will be "fake" if today's data doesn't exist, and behave normally if it does.

EDIT: You could even use the branch reset group, i.e. (?|(a)|(b)) in regex for that, so that different captures will share the same group number.

[jsmorley]

Well, maybe, just maybe, the missing data at the end could remain missing data and never be displayed in a meter if the "dayInd":["N" exists in the string, then:
- separate the RegExp option for today and tonight in your .inc
- use a regex conditional to test for the existence of "dayInd":["N" and capture nothing if it does or capture normally if it doesn't, e.g. (?(?=.*"dayInd":\["N")|<capture the 0th index(es) normally here>)

That would only require changes on the today and tonight measures (well, tonight would just have to be index 1 and be separated from today), the rest would be unaffected (unless you insist on handling the missing data at the end, that is). Basically, the today measures will be "fake" if today's data doesn't exist, and behave normally if it does.

EDIT: You could even use the branch reset group, i.e. (?|(a)|(b)) in regex for that, so that different captures will share the same group number.

I don't really follow. The problem right now is that there are "groups" {0}, {2}, etc. that are capturing the "pairs" for [1,2,3,4 and so on. When the data starts with "N" instead of "D", those "pairs" become mismatched as instead of the group capturing "day1day", "day1night", it becomes "day1night", "day2"day".