⭐ Weather.com - Parsing the V3 JSON

[jsmorley]

LOL, although I doubt air quality in urban China is "excellent", that ("excellent" in China vs just "good" in the US) might be the case for other things besides the air quality nowadays...

Hehe... And in North Korea, all values come back as "Excellent".

[Yincognito]

Hehe... And in North Korea, all values come back as "Excellent".

Yep. They claim to have 0 Covid fellows, after all, and besides the political environment there, it might just well be, considering how tight are their points of entry in the country (or if by any chance they had a hand in the pandemic, since they promised some - until now unknown - "gift" to some folks at the end of last year, if I recall correctly)...

[jsmorley]

Ok, I have entirely reworked the "Pollen" .inc files.

WeatherComJSONPollenCurrent.inc
This will create measures for the Grass / Tree / Ragweed Pollen levels for the current day
WeatherComJSONPollen29Parts.inc
This will create measures for the Grass / Tree / Ragweed Pollen levels for 29 parts of day and night values. More or less 15 days
WeatherComJSONPollen15Parts.inc
This will create measures for the Grass / Tree / Ragweed Pollen levels for 15 parts of day and night values. More or less 7 days
WeatherComJSONPollen7Parts.inc
This will create measures for the Grass / Tree / Ragweed Pollen levels for 7 parts of day and night values. More or less 3 days

It is optional to @Include one and only one of these files.


The way the Pollen data is structured in the JSON, it is in arrays based on the part of the day. During the "day", there will be 30 values, one each for alternating "day" and "night", for 15 days.

However, after some point in the afternoon, there won't be a "forecast" for "day" anymore, and the array will just start with "night", one each for alternating "night" and "day" and contain only 29 values, or 14.5 days.

With all of the other "day" and "night" forecast data in the API, when it is "night", the "day" values are just empty or "null". With this data, the "night" is just moved up to take the place of "day", and there is just one less value in the array. This is why we can't have nice things...

The measures I have created will provide the following measures for each "part":

[@PollenPart1ValidWeekDay] : The day of the week, i.e. "Monday"
[@PollenPart1DayPartIndicator] : The part of day indicator, i.e. "D" or "N"
[@PollenPart1DayPartName] : The name of the part, i.e. "Today", "Tonight", "Tomorrow", "Tomorrow Night", "Wednesday", "Wednesday night"...
[@PollenPart1GrassIndex] : The amount of Grass Pollen, in a range from 0 (none) to 4 (very high)
[@PollenPart1GrassCategory] : A string reflecting the amount using "None", "Low", "Moderate", "High", "Very High"
[@PollenPart1TreeIndex] : The amount of Tree Pollen, in a range from 0 (none) to 4 (very high)
[@PollenPart1TreeCategory] : A string reflecting the amount using "None", "Low", "Moderate", "High", "Very High"
[@PollenPart1RagweedIndex] : The amount of Ragweed Pollen, in a range from 0 (none) to 4 (very high)
[@PollenPart1RagweedCategory] : A string reflecting the amount using "None", "Low", "Moderate", "High", "Very High"

Please get the new .rmskin in the first post of this thread.

So the right way to use this is to decide on how many "parts" you want to display, then just display them with the "PartName" value like "Tonight", or if you prefer, the WeekDay like "Monday" and the PartIndicator like "D" or "N". There is no sense in trying to do this by "day" as such, as you really can't access the measures that way. "Part2" might be "tonight", or it might be "tomorrow". The "parts" are basically 12-hour blocks of time, starting "now".

[jsmorley]

Minor tweak to the Pollen .inc files, to reduce the number of them and to provide one that returns the values for "current".

[SilverAzide]

OK, I need a little assistance with an issue illustrated by this skin but not directly weather.com related.

Can someone try using the location search feature in this skin and try to search for the text Wrocław? Take note of the "accented-L" symbol, it's not a latin letter L. The WebParser is returning nothing, and I can't figure out why. If you open your browser and use the exact string that WebParser is using (https://nominatim.openstreetmap.org/search.php?q=Wrocław&format=json), it is working perfectly. Searching for https://nominatim.openstreetmap.org/search.php?q=Wroc%C5%82aw&format=json also works, of course, but that's not the URL that's going out... I tried a custom header of "Accepts-Language: pl-PL", but that didn't help (the header is mentioned in the OSM API docs).

Using the latin "L" works, but that's not what normal users would enter. Is there some behind the scenes Unicode handling going on in the browsers but not in WebParser?

[jsmorley]

OK, I need a little assistance with an issue illustrated by this skin but not directly weather.com related.

Can someone try using the location search feature in this skin and try to search for the text Wrocław? Take note of the "accented-L" symbol, it's not a latin letter L. The WebParser is returning nothing, and I can't figure out why. If you open your browser and use the exact string that WebParser is using (https://nominatim.openstreetmap.org/search.php?q=Wrocław&format=json), it is working perfectly. Searching for https://nominatim.openstreetmap.org/search.php?q=Wroc%C5%82aw&format=json also works, of course, but that's not the URL that's going out... I tried a custom header of "Accepts-Language: pl-PL", but that didn't help (the header is mentioned in the OSM API docs).

Using the latin "L" works, but that's not what normal users would enter. Is there some behind the scenes Unicode handling going on in the browsers but not in WebParser?

Yeah, doesn't work for me either. I'm a bit stumped by this. The problem isn't the InputText plugin, it is properly setting Wrocław with the accented-L as the variable. I really have no idea why WebParser would behave any differently than your browser in this instance.

As you say, it works fine if the variable is Wroc%C5%82aw instead of Wrocław, but short of a boatload of Substitutes, I can't see a good way to get that from what the user types in.

I'm less and less enamored of automating this with https://nominatim.openstreetmap.org. I'm not in love with making all users go out there manually or alternatively going to:

https://support.google.com/maps/answer/18539
https://www.latlong.net/
https://www.mapcoordinates.net

But I have found the results from openstreetmap to be a bit hinky in general. I get really puzzling results a lot, and if you search for a location in a particular country, like "Moscow, Russia", you get results in Russian / Cyrillic, which is a bit annoying.

[SilverAzide]

Yeah, doesn't work for me either. I'm a bit stumped by this. The problem isn't the InputText plugin, it is properly setting Wrocław with the accented-L as the variable. I really have no idea why WebParser would behave any differently than your browser in this instance.

As you say, it work fine if the variable is Wroc%C5%82aw instead of Wrocław, but short of a boatload of Substitutes, I can't see a good way to get that from what the user types in.

So I just tried 東京都, 日本 too (which is the Toyko default present in the skin) and it choked there too. Can WebParser support a "URL encoding" option, or is that just for things like spaces, ampersands and whatnot?

[jsmorley]

So I just tried 東京都, 日本 too (which is the Toyko default present in the skin) and it choked there too. Can WebParser support a "URL encoding" option, or is that just for things like spaces, ampersands and whatnot?

Yeah, the :EncodeURL section variable parameter only encodes "reserved characters" in a URL.

[jsmorley]

If I just type in Wroclaw, with a normal "l", I do get the same result, and it works in the skin. It might be that we just have to tell them that you need to use English spelling of place names, like Munich instead of München. I get no results with München either, but do with Munich.

So the long and the short of it is that the "input" must be in English, but the "output" will be in whatever language the country result is.

[SilverAzide]

If I just type in Wroclaw, with a normal "l", I do get the same result, and it works in the skin. It might be that we just have to tell them that you need to use English spelling of place names, like Munich instead of München.

Yeah, that's my only advice as well. So, looks like there's a need for a "percent encoding" thing for strings. Can Lua turn Unicode text (char array?) into a percent-encoded string?