⭐ Weather.com - Parsing the V3 JSON

[Yincognito]

Something I discovered recently, given the fact that I helped on some "radar" skins here on the forum. One can access weather.com radar as well, using something like (the supported formats are JPG and PNG, no GIF apparently):

Code: Select all

https://api.weather.com/v2/maps/dynamic?geocode=47.5,6.0&h=320&w=568&lod=5&product=twcRadarHcMosaic&map=light&format=png&language=en&apiKey=<ApiKey>&a=0

dynamic.png

Apparently, changing the a= parameter is doing something, for bigger differences.

The idea was to have something like:

Code: Select all

https://weather.com/weather/radar/interactive/l/Fowler+IN?canonicalCityId=7eb5f0887b7c498cb1a1b03f43da73b0e7336ef0d0e55150c0cf5e7634e260c7

(play with the Layer menu on the left or else you won't see much) that can be grabbed with WebParser and displayed. Sure, from the notes on the page it seems it's working only for the western hemisphere and only during daylight hours (whose daylight, I wonder?), but it's still something.

Also, I used my own image scraping Chrome extension to further investigate the matter and it appears one can get those images, albeit piece by piece (i.e. one for each map section):

Weather Radar.jpg

aka...

Code: Select all

https://api.mapbox.com/styles/v1/weather/cjza0fpcf54z71cnty86ulfhj/tiles/5/7/11?access_token=pk.eyJ1Ijoid2VhdGhlciIsImEiOiJjamxncnYwMmYxYXgxM3BzNWEyNHUzbmhhIn0.UqrZwn6qTa6WJwdHjPVUfA

Code: Select all

https://api.mapbox.com/styles/v1/weather/cjza0fpcf54z71cnty86ulfhj/tiles/5/8/11?access_token=pk.eyJ1Ijoid2VhdGhlciIsImEiOiJjamxncnYwMmYxYXgxM3BzNWEyNHUzbmhhIn0.UqrZwn6qTa6WJwdHjPVUfA

Code: Select all

https://api.weather.com/v3/TileServer/tile?product=satradFcst&ts=1598046300&fts=1598047200&xyz=16:24:6&apiKey=<ApiKey>

Code: Select all

https://api.weather.com/v3/TileServer/tile?product=satradFcst&ts=1598046300&fts=1598047200&xyz=16:23:6&apiKey=<ApiKey>

and so on.

Maybe others will find more about it, if they are interested.

[MikeG621]

I don't think this is a problem with the JSON (which I finally rolled into my skins, just don't have the time to maintain my scraper anymore), but has anyone else seen the alert SeverityCode values be non-zero while the Severity/Text/Descriptions are still blank?

[SilverAzide]

I don't think this is a problem with the JSON (which I finally rolled into my skins, just don't have the time to maintain my scraper anymore), but has anyone else seen the alert SeverityCode values be non-zero while the Severity/Text/Descriptions are still blank?

Yes, I have seen this as well. I just noticed it recently. If you have a location that has at least one alert, and then on the next update all alerts are gone, the severity codes will retain their previous numeric values. I guess it is because none of the measures can be refreshed since the whole alert section is missing, but I'm not sure why the values don't go to zero. Perhaps @JSMorley can chime in on why this would be. It is rather tedious to find a location with an expiring alert to try this out, but it is doable.

[jsmorley]

Yes, I have seen this as well. I just noticed it recently. If you have a location that has at least one alert, and then on the next update all alerts are gone, the severity codes will retain their previous numeric values. I guess it is because none of the measures can be refreshed since the whole alert section is missing, but I'm not sure why the values don't go to zero. Perhaps @JSMorley can chime in on why this would be. It is rather tedious to find a location with an expiring alert to try this out, but it is doable.

Not quite sure why this would be honestly. The [@AlertsParent] parent measure will always return some value, and although if there are no alerts present, no child measures will be populated, the RegExp won't "fail", as they are all lookaheads.

I wonder if both the "string" and "number" values are being kept in this case?

[MikeG621]

I wonder if both the "string" and "number" values are being kept in this case?

It's keeping just the number value, the string is blank.

json.PNG

[jsmorley]

It's keeping just the number value, the string is blank.
json.PNG

Hm... So it looks like the propensity of WebParser to "keep" existing values when no values are returned on a subsequent update might be getting in the middle of this, since when there are no alerts, literally nothing is returned. I think "number" values are more susceptible to this in WebParser.

I think we might need to send a [!CommandMeasure MeasureName "Reset"] just before we go out to look for alerts each time, but I need to chew on where to put that to make sense. You can't !CommandMeasure a measure that doesn't exist, or you get log blight. So for the many folks who don't use the Alerts .inc file, that becomes a factor.

I will explore a bit, but this tends to lead me to think that the .inc for Alerts might need to actually go out to the site itself, rather than just depending on the "super parent" contained in WeatherComJSONMeasures.inc. Then I can safely have a "super parent" measure that first does a "reset" on the "not as super parent" alert measure, followed by an "update" of that parent. In effect starting fresh and clean on each iteration of UpdateRate.

[SilverAzide]

Hm... So it looks like the propensity of WebParser to "keep" existing values when no values are returned on a subsequent update might be getting in the middle of this, since when there are no alerts, literally nothing is returned. I think "number" values are more susceptible to this in WebParser.

I think we might need to send a [!CommandMeasure MeasureName "Reset"] just before we go out to look for alerts each time, but I need to chew on where to put that to make sense. You can't !CommandMeasure a measure that doesn't exist, or you get log blight. So for the many folks who don't use the Alerts .inc file, that becomes a factor.

I will explore a bit, but this tends to lead me to think that the .inc for Alerts might need to actually go out to the site itself, rather than just depending on the "super parent" contained in WeatherComJSONMeasures.inc. Then I can safely have a "super parent" measure that first does a "reset" on the "not as super parent" alert measure, followed by an "update" of that parent. In effect starting fresh and clean on each iteration of UpdateRate.

Could something like a new bang like [!CommandMeasureGroup MeasureGroupName "Reset"] help here (i.e., where you could group all the alert child measures)?

I kind of like the fact that the super parent grabs everything; with my crappy internet connection, I'm liking how fast the new JSON API is.

[Yincognito]

Hm... So it looks like the propensity of WebParser to "keep" existing values when no values are returned on a subsequent update might be getting in the middle of this, since when there are no alerts, literally nothing is returned. I think "number" values are more susceptible to this in WebParser.

I think we might need to send a [!CommandMeasure MeasureName "Reset"] just before we go out to look for alerts each time, but I need to chew on where to put that to make sense. You can't !CommandMeasure a measure that doesn't exist, or you get log blight. So for the many folks who don't use the Alerts .inc file, that becomes a factor.

I will explore a bit, but this tends to lead me to think that the .inc for Alerts might need to actually go out to the site itself, rather than just depending on the "super parent" contained in WeatherComJSONMeasures.inc. Then I can safely have a "super parent" measure that first does a "reset" on the "not as super parent" alert measure, followed by an "update" of that parent. In effect starting fresh and clean on each iteration of UpdateRate.

Could something like a new bang like [!CommandMeasureGroup MeasureGroupName "Reset"] help here (i.e., where you could group all the alert child measures)?

I'm not quite familiar with the topic and all, but if it's about number values that are kept in the WebParser children, wouldn't disabling that child measure and updating it (optionally enabling it as well) afterwards clear the number in the child to 0? I just tried in my own WebParser child measures and it does the job (doesn't clear the strings though, as per the manual, but as far as I read they're not a problem here). My WebParser children use StringIndex and StringIndex2 - hopefully it isn't any different from your own measures in terms of the desired effect...

[jsmorley]

I think disable / enable of the child measures would probably work, although I'll have to test that in this context. The trick is when and how to execute that. The child measures only get one bite at the apple on each actual WebParser update of the parent measure, the parent measure "pushes" the data to the children, and if we aren't careful about when we disable / enable the measures, they can end up with no value at all. It simply cannot be on a FinishAction on the parent measure. That will just yank the rug out from under them. It can't be at some arbitrary time, as you don't want the values to change to zero when the parent hasn't even updated yet, as that will not work. It needs to happen JUST BEFORE the parent measure updates the values. I'm not sure yet how to do that that makes the most sense.

My gut reaction is a leading WebParser measure that does something, something really simple and certain, and then put a FinishAction on that which disables / enables all the alert measures, then forces an update of the actual parent measure. Need to be careful that a second, redundant update of the parent alert measure is not caused, as while it doesn't go out to the web, it does cart around a lot of data from the super-parent, and I'd like to avoid having it even just "read" that amount of data out of memory if I don't need to.

The easiest thing is to put a FinishAction on the overall super-parent that does the enable / disable, but that can't work, since many, if not most folks won't be using the alerts .inc file, and that would just cause errors in the log.

In addition, I think this has no advantage over the "Reset" command, which would have to be implemented with the same timing considerations. I could argue that "Reset" is in fact more certain and reliable in this case.

[Yincognito]

It can't be at some arbitrary time, as you don't want the values to change to zero when the parent hasn't even updated yet, as that will not work. It needs to happen JUST BEFORE the parent measure updates the values. I'm not sure yet how to do that that makes the most sense.

I don't follow. To me "when the parent hasn't even updated yet" is pretty much the same as "JUST BEFORE the parent measure updates the values", since children are a function of the parent. I understand the subtle difference in what you said, but from a practical point of view, it shouldn't make any difference: once that child measure is 0, it can't be set back to the "past" value of 4 or whatever if we're talking about a leftover from the previous WebParser update, can it? I mean, unless that non zero value is set (or kept) because it remains there in the weather.com source itself (which would be unfortunate), the following parent update will just skip both the number and the string value of the child measure since "if there are no alerts present, no child measures will be populated", like you said earlier.

Anyway, maybe I don't know all the details here, so I might be very well mistaken. The way I approach these things (although it never happened to me to need a "clearance" of past values of a WebParser, I only use "Reset" to hopefully save some memory in my Feeds skin) is to make a measure just before the WebParser parent, updating at the same rate, and do all I need there before the parent updates. Another solution that I think it could work (though I'm not sure) is to do what you need in the OnUpdateAction of the WebParser parent. This should happen close to the moment you want to, but the update of the WebParser parent must in this case be controlled from the UpdateDivider instead of the UpdateRate (the latter will just be set to 1) - and yes, I've done that already, it works, but I'm not sure if it will be the exact moment you want to (let's say I'm just 75% sure).

P.S. Maybe the fact that I didn't have such problems (can't speak about this case though) is because I always set my WebParser children with an UpdateDivider of -1, so, apart from the first "default" update at skin refresh which is a bit tricky to avoid normally (unless the parent measure is intially disabled), the children are only updated from the FinishAction of the parent.