Weather Skins Not Working

[jsmorley]

I think the best way forward with this is to drop using the embedded/formatted JSON in the site HTML, and just use the API to get the raw JSON and parse that.

I think trying to turn those Unicode code point values like \u0034 into characters with Substitute is going to be problematic. I can't see having each end-user build a Substitute option that is specific to the character set for their local language, and if you tried to do all of them, it would be a Substitute with something like 65,000 pairs. Substitute won't even support that.

I'm increasingly of the opinion that we shouldn't try to keep "chasing" changes made in the HTML site, or the embedded JSON that is formatted in such a way that it is easy to read with their Javascript program under the covers, but something that we are going to be constantly struggling with as they mess with how they do the site.

Parsing the JSON from the API is no more difficult than it was to parse the embedded JSON before, and that can be done with a little work on my end, while keeping the design and output of the various @Include .inc files the same. So from that standpoint, you just drop in the new .inc files and it's fixed.

The only real rub here is the URL.

As far as I can tell so far, the URL for the API requires that you send it the latitude / longitude for the location you want. I can't see any way to use any of the long or short location codes, like bbd510e73cb5870f973cea9253bb1f9bfe239a129dfc2b89589cbb6d2d46b4bc or USVA0944, or postal code like 22308. It requires latitude and longitude in the format geocode=38.733,-77.058.

This will add a little extra work for the end-user, unless we can come up with some way to take one or more existing codes the user is currently using, and turn that into latitude and longitude.

Maybe the way to do this is to just use the site itself as the "tool" for a setup skin. If you go to the site using any of the location code, lat / long, postal code approaches in the URL, one of the things that is returned by the site is the latitude and longitude for the location. I can parse that, and write the latitude / longitude to the variables... In theory, can means I can keep the separate skin that allows you to search for location code by some text like "Fort Hunt, Virginia", and it will use our local database that turns that into USVA0944, then have the same skin hit weather.com to get the latitude / longitude for that location code.

[SCR]

So nobody has any idea how to use the old style Location Code (USVA0944) in the URL for the API?

I hope we can figure that out, as that Substitute just causes Rainmeter to eat its own face. It's just way too many substitute pairs on that huge amount of data.

The only other alternative that makes sense to me is to force users to find their Lat / Long and use that, then parse the JSON through the API. I suppose some skin, sorta like the one I have now to turn "Fort Hunt, VA" into "USVA0944" could be written to perhaps turn "Fort Hunt, VA" into "38.733,-77.058" for use in the URL. I'll have to find it, but there is some geocode lookup site that I think we can use.

I really think we might have to go with the API approach, as I just can't see trying to turn the thousands of possible HTML Unicode values into characters on a site that is just gigantic. This was working fine when the JSON was just "embedded" as vanilla JSON code in the middle of the HTML, but with them changing it to somehow "mix" the HTML and the JSON formatting, it's really getting unreasonable to parse since we don't have their Javascript code.

In the meantime, this set of @Include .inc files should get it working for the English language.

Includes.zip

The new WeatherComJSONMeasures5Day.inc worked for me, no errors logged. I haven't tried the WeatherComJSONMoon7Day.inc yet as I will have to change my skin back to these measures. I'll give it a try over the weekend.

Thank you just doesn't seem to say enough. I really appreciate what you do.

[jsmorley]

The new WeatherComJSONMeasures5Day.inc worked for me, no errors logged. I haven't tried the WeatherComJSONMoon7Day.inc yet as I will have to change my skin back to these measures. I'll give it a try over the weekend.

Thank you just doesn't seem to say enough. I really appreciate what you do.

Yeah, the temporary fix I did to deal with the escaped \" quote characters is ok, as long as you are using a language that uses the ASCII / Extended ASCII character sets. This probably fine for English, and European languages like French and German and such, but is going to be a problem with other languages like Hungarian, and probably Russian and almost certainly Japanese.

I'm working on something more robust. Probably be a day or two.

[SCR]

Yeah, the temporary fix I did to deal with the escaped \" quote characters is ok, as long as you are using a language that uses the ASCII / Extended ASCII character sets. This probably fine for English, and European languages like French and German and such, but is going to be a problem with other languages like Hungarian, and probably Russian and almost certainly Japanese.

I'm working on something more robust. Probably be a day or two.

My wife and I have medical tests scheduled over the next two days, they are going to be miserable. So I'll wait to make any further changes until at least the weekend.

[Yincognito]

I think the best way forward with this is to drop using the embedded/formatted JSON in the site HTML, and just use the API to get the raw JSON and parse that.

I think trying to turn those Unicode code point values like \u0034 into characters with Substitute is going to be problematic. I can't see having each end-user build a Substitute option that is specific to the character set for their local language, and if you tried to do all of them, it would be a Substitute with something like 65,000 pairs. Substitute won't even support that.

I'm increasingly of the opinion that we shouldn't try to keep "chasing" changes made in the HTML site, or the embedded JSON that is formatted in such a way that it is easy to read with their Javascript program under the covers, but something that we are going to be constantly struggling with as they mess with how they do the site.

Parsing the JSON from the API is no more difficult than it was to parse the embedded JSON before, and that can be done with a little work on my end, while keeping the design and output of the various @Include .inc files the same. So from that standpoint, you just drop in the new .inc files and it's fixed.

The only real rub here is the URL.

As far as I can tell so far, the URL for the API requires that you send it the latitude / longitude for the location you want. I can't see any way to use any of the long or short location codes, like bbd510e73cb5870f973cea9253bb1f9bfe239a129dfc2b89589cbb6d2d46b4bc or USVA0944, or postal code like 22308. It requires latitude and longitude in the format geocode=38.733,-77.058.

This will add a little extra work for the end-user, unless we can come up with some way to take one or more existing codes the user is currently using, and turn that into latitude and longitude.

Maybe the way to do this is to just use the site itself as the "tool" for a setup skin. If you go to the site using any of the location code, lat / long, postal code approaches in the URL, one of the things that is returned by the site is the latitude and longitude for the location. I can parse that, and write the latitude / longitude to the variables... In theory, can means I can keep the separate skin that allows you to search for location code by some text like "Fort Hunt, Virginia", and it will use our local database that turns that into USVA0944, then have the same skin hit weather.com to get the latitude / longitude for that location code.

Yep. I see the real rub here being the public API key consistency and availability. It's true that I found quite old posts on other sites that mentioned the same key, so it seems that it's stable, but then, you never know. The latitude and longitude requirements for the code are not at all a problem in my view - I mean, literally everybody can google their location and they'll find the location's latitude and longitude coordinates, at least on wiki. If that is hard for the user to do, then I'm out of words, really...

[jsmorley]

Yep. I see the real rub here being the public API key consistency and availability. It's true that I found quite old posts on other sites that mentioned the same key, so it seems that it's stable, but then, you never know. The latitude and longitude requirements for the code are not at all a problem in my view - I mean, literally everybody can google their location and they'll find the location's latitude and longitude coordinates, at least on wiki. If that is hard for the user to do, then I'm out of words, really...

Well, assuming they continue to embed the API key in the HTML the site returns, I guess I can grab that at the same time I get the latitude / longitude based on the location code. If they obfuscate or remove that key from the HTML on the site, we are done...

Finding a desired latitude / longitude certainly isn't hard... Just a bit hard to automate.

https://www.latlong.net/

You may be right, maybe I just need to tell people how to find their latitude / longitude and have them copy and paste those values into the variables and be done with it. Latitude and longitude have a universal, unambiguous meaning, while all those location codes and postal codes and all that are either specific to weather.com, or may or may not be available for a given location. I'm trying to make this transition as easy and seamless as possible, but that might be a fools errand.

[jsmorley]

Parsing the JSON from the API won't be hard at all, and all data is returned "as is", without any concerns about Unicode characters.

https://api.weather.com/v3/aggcommon/v3-wx-observations-current;v3-wx-forecast-daily-15day;v3alertsHeadlines;v3-location-point?format=json&geocode=38.723269,-77.0698821&units=m&language=hu-HU&apiKey=d522aa97197fd864d36b418f39ebb323

[Yincognito]

Well, assuming they continue to embed the API key in the HTML the site returns, I guess I can grab that at the same time I get the latitude / longitude based on the location code. If they obfuscate or remove that key from the HTML on the site, we are done...

Finding a desired latitude / longitude certainly isn't hard... Just a bit hard to automate.

https://www.latlong.net/

You may be right, maybe I just need to tell people how to find their latitude / longitude and have them copy and paste those values into the variables and be done with it. Latitude and longitude have a universal, unambiguous meaning, while all those location codes and postal codes and all that are either specific to weather.com, or may or may not be available for a given location. I'm trying to make this transition as easy and seamless as possible, but that might be a fools errand.

Yeah. That's what my custom tooltip shows when hovering over the "location code" in my Settings skin:

Code: Select all

Weather Location Code. The comma (',') separated latitude and longitude for your location.
Find them at 'https://nominatim.openstreetmap.org/search?q=[Location]&limit=1&format=xml'.

(courtesy of SilverAzide)

I believe it's simple enough for every user to understand. If one wants to automate the process as much as possible, then the link can be of course used to automatically find the latitude and longitude based on the location user input, before going for the weather.com URL, but personally I want to minimize the number of polled sites as much as possible, preferably to be just one (i.e. the weather one, in this case). That being said, this doesn't mean that you or any other skin designer can't implement it.

[Ballantyne]

On the off chance another was linked to other than Weather.com, tried
Everon (UPDATED 30-JUL-2020)
BY xxenium
Don't know the answer but linked to weather, using long code of city page it works
https://www.deviantart.com/download/830781803/ddqmjcb-e24a46c2-64bc-45bd-abed-8ab3efee1a49.rmskin?token=ac0908b0b046ec4f6c24664add5fc81bc562fa10&ts=1596117995
Don't know if this helps.

[jsmorley]

On the off chance another was linked to other than Weather.com, tried
Everon (UPDATED 30-JUL-2020)
BY xxenium
Don't know the answer but linked to weather, using long code of city page it works
https://www.deviantart.com/download/830781803/ddqmjcb-e24a46c2-64bc-45bd-abed-8ab3efee1a49.rmskin?token=ac0908b0b046ec4f6c24664add5fc81bc562fa10&ts=1596117995
Don't know if this helps.

Any author who gets the new .inc files I have posted can get their skin "working" in a sense, although only properly for English and some European languages. I personally wouldn't just yet. It is my intention to make some fundamental changes to how the .inc files get the data, and while at the end of it an author should more or less be able to drop in the new .inc files, I'm still sorting out how the "location" is going to be handled to make it as easy for the final end-user as possible.

I have some time constraints today, but should be able to get this working sometime late tomorrow.