It is currently March 30th, 2020, 2:05 pm

⭐ weather.com - Parsing the JSON

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

SCR wrote:
February 8th, 2020, 5:18 pm
Late last night I was congratulating myself on completing the conversion of my weather skin. Feeling proud of myself that it worked, I went to sleep happy that the weather was once again on my screen and the job was done.

This morning as I settled in with my cup of coffee to read the latest and greatest from Rainmeter I was greeted with your post of your latest miracle.

Now I have to get back to another conversion project. However I don't mind a bit as I am now and forever addicted to Rainmeter. I have never had so much fun with any other software.

Besides, now that I'm in my late 70's Rainmeter keeps the gray matter in my head churning so you can add a health benefit to all the things Rainmeter can do. I don't understand a lot of it but it sure is fun to try and figure out.

Once again I offer my sincerest thanks for your hard work. Maybe not hard for you but near impossible for me. I appreciate all you do.
8-) I think it's worth converting to the JSON, for all the reasons I have talked about, and in addition, there is a bunch of stuff, which I have not yet explored, that can be added to the data. Stuff like "amount of snowfall" and "severity of thunderstorms" and "air quality" and all kinds of other cool stuff. As I figure them out, I will add them to the @Include file, but in a way that will be backwards compatible and won't "break" your existing skins. That's the beauty of this approach. You don't ever have to worry about the regular expression or what StringIndex things are.
SCR
Posts: 44
Joined: April 15th, 2015, 11:13 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SCR »

jsmorley wrote:
February 8th, 2020, 5:23 pm
8-) I think it's worth converting to the JSON, for all the reasons I have talked about, and in addition, there is a bunch of stuff, which I have not yet explored, that can be added to the data. Stuff like "amount of snowfall" and "severity of thunderstorms" and "air quality" and all kinds of other cool stuff. As I figure them out, I will add them to the @Include file, but in a way that will be backwards compatible and won't "break" your existing skins. That's the beauty of this approach. You don't ever have to worry about the regular expression or what StringIndex things are.
I most certainly intend to convert my weather skin but I'm also considering the new stuff already present. I may have to go back and resurrect and old skin I was developing for Weather Underground until they pulled the plug. I had created Day Night forecast meters and can now once again use them along with a few other goodies like my compass. RegEx has always been a big challenge for me and I've been slow to learn.

I haven't had a chance to take a good look at the new include files as life is once again getting in the way of having fun.

Thanks for telling me about coming additions. I will have to consider them in my design.

Be Well..
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

SCR wrote:
February 8th, 2020, 9:00 pm
I most certainly intend to convert my weather skin but I'm also considering the new stuff already present. I may have to go back and resurrect and old skin I was developing for Weather Underground until they pulled the plug. I had created Day Night forecast meters and can now once again use them along with a few other goodies like my compass. RegEx has always been a big challenge for me and I've been slow to learn.

I haven't had a chance to take a good look at the new include files as life is once again getting in the way of having fun.

Thanks for telling me about coming additions. I will have to consider them in my design.

Be Well..
Regular Expression is like golf. A minute to learn, a lifetime to master.
SCR
Posts: 44
Joined: April 15th, 2015, 11:13 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SCR »

jsmorley wrote:
February 8th, 2020, 9:39 pm
Regular Expression is like golf. A minute to learn, a lifetime to master.
Well dang, I better hurry up and kick it up a notch. My lifetime is getting shorter. :???:

I never had much luck at golf either. They wouldn't let my dog fetch the balls.
User avatar
SilverAzide
Posts: 716
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ weather.com - Parsing the JSON

Post by SilverAzide »

Hello!
I've been poking around with the JSON measures and @includes and I have some feedback. Hopefully it is the constructive type.

First, can you adjust all the wind degrees measures ([@CurrentWindDirectionDegrees], et al) to add min/max values so they can be referenced by Rotator measures? This will more easily allow creating "wind vanes" (or "wind barbs" if you prefer) without needing a Calc wrapper.

Code: Select all

[@CurrentWindDirectionDegrees]
...
MinValue=0
MaxValue=360
Secondly, the Degrees=° variable is in the "don't touch this stuff" section of the JSON variables file, but it is never referenced anywhere but in the main example skin. Can this be moved out of the "template" include files?

Lastly, the language include file(s) contain the DateFormat and TimeFormat variables. The language file is suggested as being optional, but these two variables are required for all the Time measures in the main Measures include file to work properly. I don't have a better solution for where these should go. Just mentioning that they aren't optional. In my iteration of this, I moved them to the JSON variables file and set them equal to my own skin variables (e.g., DateFormat=#MyDateFormatVar#), which has been my approach for all the variables in the include files.

And just one final whine to no one in particular: I'm very bummed to note that the JSON data is missing all the severe weather alert stuff, unless I somehow overlooked it. Seems that scraping some HTML is inevitable if this is something you want.
:(
DeviantArt Gadgets More...
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

SilverAzide wrote:
February 8th, 2020, 10:33 pm
Hello!
I've been poking around with the JSON measures and @includes and I have some feedback. Hopefully it is the constructive type.

First, can you adjust all the wind degrees measures ([@CurrentWindDirectionDegrees], et al) to add min/max values so they can be referenced by Rotator measures? This will more easily allow creating "wind vanes" (or "wind barbs" if you prefer) without needing a Calc wrapper.

Code: Select all

[@CurrentWindDirectionDegrees]
...
MinValue=0
MaxValue=360
Secondly, the Degrees=° variable is in the "don't touch this stuff" section of the JSON variables file, but it is never referenced anywhere but in the main example skin. Can this be moved out of the "template" include files?

Lastly, the language include file(s) contain the DateFormat and TimeFormat variables. The language file is suggested as being optional, but these two variables are required for all the Time measures in the main Measures include file to work properly. I don't have a better solution for where these should go. Just mentioning that they aren't optional. In my iteration of this, I moved them to the JSON variables file and set them equal to my own skin variables (e.g., DateFormat=#MyDateFormatVar#), which has been my approach for all the variables in the include files.

And just one final whine to no one in particular: I'm very bummed to note that the JSON data is missing all the severe weather alert stuff, unless I somehow overlooked it. Seems that scraping some HTML is inevitable if this is something you want.
:(

Done. I agree with all those assessments. I moved stuff around as you suggested. The Date and Time format variables seemed to make sense to me in the "language" file, cause they are sorta related to that. But on reflection, you are right. Those need to be certain to be @Included, and that "translation" stuff is very optional. I moved them to WeatherComJSONVariables.inc.

The ° variable I added simply because it's a bit of a pain for a user to know how to get it into their code. You can copy it from somewhere on the internet, hold down ALT and type 0176 on the number pad, or use [\0176] as a Character Variable. None of those are all that intuitive. You are right though, it's something I do in my skin, and shouldn't be part of the overall templates. I moved it to my GeneralVariables.inc.

I added MinValue and MaxValue to Wind degrees heading, % Precipitation, % Humidity, and UVIndex. It makes some sense in all those cases for things like Bar/Roundline/Rotator meters.

I'm not sure if weather.com measures those degrees from 1-360 or from 0-359. It has to be one of those to be accurate. From 0-360 is close enough for government work, but not correct. 0 and 360 are the same thing on a compass. I'll figure it out one of these days. For now, since I don't know for sure, I left it as 0-360. I don't think one "tick" either way on a 360 degree compass heading is going to have you miss the moon and sail off into deep space, screaming and clawing at the windows.

I'd be happy to implement Alerts, I do think they are in the JSON as well, but I have not yet found a way that I like. The biggest problem is that there can be anywhere from none to unlimited alerts, and WebParser is spectacularly bad at that. I don't want some Rube Goldberg or dozens of loookaheads. If I can get comfortable with an approach that is simple'ish and reliable, I'll put it in. In the meantime, if you look at my skin, I do detect the Alert with the highest "severity" and warn to you check the website. It's really easy, and good enough for now. I'm just not going to scrape the HTML at all if I can help it.

As an aside, here is a very cool chart for compass degrees...


1.png
You do not have the required permissions to view the files attached to this post.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

As suggested by Yincognito in the other thread I modified my #CommonSubstitute# variable to be:

CommonSubstitute='"':"","^null$":"","\\u002F":"/","\\u005C":"\","\\u003C":"<","\\u003E":">"

So this removes "quotes" from around string values, changes the literal null to an empty string, and decodes those UNICODE character references if they are found.

This RegExpSubstitute is on every "child" WebParser measure in the template.
User avatar
Yincognito
Posts: 1072
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

Finally, JSON weather parsing, yay. Now this is the thread for me! Although I made my own comprehensive JSON regexes, this is great for users to have as a reference! :thumbup:

In case you didn't know, there is a free key (not "hacked" or anything like that) that one can use to get only the desired part of the JSON (pretty much the same as you used to get the wxdata's XML), without messing with the rest of the gibberish on the main TWC page. You can even get the data by your geo coordinates, without ever having to deal with a "location code"! Of course, you'd have to know your latitude and longitude, but then, we're not in stone age anymore and this should be known by regular folks. And also, you can have different language and units, say en-US and metric system.

Let me know if you want me to post the key and some additional info regarding the format of the URL address.

On the regexes, these are mine, if anyone is interested (I use them with the StringIndex and StringIndex2 options of the WebParser child measures):

Code: Select all

WeatherRegex="(?siU)<script .*>.*window\.__data=\{.*"dal":\{.*"Location":\{.*,".*":\{.*"data":\{.*"location":\{(.*\{.*\}.*)\}.*\}.*\}.*\}.*"Observation":\{.*"geocode:.*":\{.*"data":\{.*"vt1observation":\{(.*)\}.*\}.*\}.*\},"DateTime":\{.*"ianaTimeZone:.*":\{.*"data":\{(.*)\}.*\}.*\},"DailyForecast":\{.*"geocode:.*":\{.*"data":\{.*"vt1dailyForecast":\[\{(.*)\},\{(.*)\},\{(.*)\},\{(.*)\},\{(.*)\},\{(.*)\},\{.*\}\].*\}.*\}.*\}.*\}.*"unitOfMeasurement":\{(.*)\}.*\}.*<\/script>"
LocationRegex="(?siU)^.*"latitude":(.*),"longitude":(.*),"city":(.*),.*,"adminDistrict":(.*),"adminDistrictCode":(.*),.*,"country":(.*),"countryCode":(.*),.*$"
ObservationRegex="(?siU)^.*"altimeter":(.*),"barometerTrend":(.*),.*,"barometerChange":(.*),"dewPoint":(.*),"feelsLike":(.*),"gust":(.*),"humidity":(.*),"icon":(.*),"observationTime":(.*),.*,"phrase":(.*),"precip24Hour":(.*),"snowDepth":(.*),"temperature":(.*),.*,"uvIndex":(.*),"uvDescription":(.*),"visibility":(.*),"windSpeed":(.*),"windDirCompass":(.*),.*$"
DateTimeRegex="(?siU)^.*,"timeZoneAbbreviation":(.*)$"
DayRegex="(?siU)^.*"validDate":(.*),"sunrise":(.*),"sunset":(.*),"moonIcon":(.*),"moonPhrase":(.*),"moonrise":(.*),"moonset":(.*),"dayOfWeek":(.*),.*,"day":\{.*"dayPartName":(.*),"precipPct":(.*),.*,"temperature":(.*),.*,"icon":(.*),.*,"phrase":(.*),"narrative":(.*),.*,"windDirCompass":(.*),.*,"windSpeed":(.*),"humidityPct":(.*),.*\},"night":\{"dayPartName":(.*),"precipPct":(.*),.*,"temperature":(.*),.*,"icon":(.*),.*,"phrase":(.*),"narrative":(.*),.*,"windDirCompass":(.*),.*,"windSpeed":(.*),"humidityPct":(.*),.*\}.*$"
UnitOfMeasurementRegex="(?siU)^.*,"temp":(.*),"speed":(.*),"distance":(.*),"accumulation":(.*),"precip":(.*),"pressure":(.*)$"
Unlike jsmorley's, they deal with almost all the parathesis (a pain in the a..., if you ask me), but that led me to discover something tricky about the "Location" section. It contains two subsections, but those are different in most cases!! The first subsection is usually "language:...:placeid:...": {...}, but can be other names as well, depending on the language and such. The second subsection is sometimes called "canonicalCityId:...:language:...{...}": {, but can also be named language:...placeid:...{...}, so again a variable name, which makes it a bit difficult to identify. The issue with these subsections and why they're important is that often only one of them will have its relevant fields populated, while the other will contain lots of null values. For example, one of them might have most of the info present, while the other will contain null city name, administrative district name and such. In my tests I've noticed that it's usually the 2nd of those subsections that contains the needed data, and the 1st is reserved for different languages or deal with the locid and not with the ("canonical") cityid. So be careful which one you get, otherwise you may be parsing some empty fields (although they exist in nearby locations in the JSON). Jsmorley probably got the displayName instead of "city" to workaround this issue, but that leaves the "adminDistrict" (and who knows, maybe others that we don't know of) prone to missing content.

P.S. Great job, jsmorley! You're the man. :thumbup:
Last edited by Yincognito on February 9th, 2020, 12:30 am, edited 1 time in total.
User avatar
jsmorley
Developer
Posts: 20425
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Parsing the JSON

Post by jsmorley »

I don't think we should publish or encourage the use of that API key. On the surface, that goes against the very purpose of an API key, and they are bound to lock it down sooner or later.

And yes, I use displayName, then adminDistrict (which may or may not be empty) then country to get the Location information. That seems pretty reliable in any language in any location in the testing I have done so far.

Code: Select all

(?siU)"Location":{.*"latitude":(.*),.*"longitude":(.*),.*"adminDistrict":(.*),.*"adminDistrictCode":(.*),.*"country":(.*),.*"countryCode":(.*),.*"displayName":(.*),.*"Observation":.*language:(.*):.*"timeZoneAbbreviation":(.*)}
I don't find "city" to be of much use at all. It is going to trot over from the specific location indicated in displayName to the nearest actual "city" associated with the location code, and that is exactly what I don't want. I want "Fort Hunt", not "Alexandria". displayName is what is shown at the top of the web page, and that is what I want.

If at ALL possible, and so far it has been, I want to avoid any "differences" that I have to deal with if the user uses the old-style location code, the new-style long location code or a US zip code. I don't want to know or care about v1 or v2 of the website. Except for some miss-mash in the "location" stuff, everything else seems to be consistent in the JSON.
User avatar
Yincognito
Posts: 1072
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: ⭐ weather.com - Parsing the JSON

Post by Yincognito »

jsmorley wrote:
February 9th, 2020, 12:29 am
And yes, I use displayName, then adminDistrict (which may or may not be empty) then country to get the Location information.
Again, you might be right this time as well. Because you take adminDistrict after the displayName (although it's technically ahead of the latter in the subsection's structure) you are in effect taking the adminDistrict from the 2nd subsection (the one usually having complete data), which is why it doesn't fail. I don't know if this is intentional, but if it is, your trick is genius work.
jsmorley wrote:
February 9th, 2020, 12:44 am
It's really not how we roll here in RainmeterLand.
Yep. Totally agree. Oh, where are you, wxdata XML, with that small and beautiful footprint? :-(



P.S. Oops, I edited this previous message instead of posting a new one. Well, nevermind...
Last edited by Yincognito on February 9th, 2020, 1:06 am, edited 2 times in total.