It is currently February 26th, 2020, 9:30 am

Weather.com WXDATA feed failing again

General topics related to Rainmeter.
User avatar
jsmorley
Developer
Posts: 20282
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Weather.com WXDATA feed failing again

Post by jsmorley »

Cariboudjan wrote:
January 31st, 2020, 2:25 am
I mean, an XML file is pretty small and would only need to be accessed/updated on Rainmeter load. Rainmeter could conceivably have this information baked into Rainmeter and change with an update, or it could access the information from an XML file from Rainmeter.net and only do this when Rainmeter loads. It doesn't need to ping this data on each skin update/refresh/load. It only needs to do so when Rainmeter starts. If the information changes, a simple Rainmeter reboot would correct the error.
It's not a bad idea at all, but not one that I feel comfortable with to be honest. While I'm hopeful that it won't be, it might well be the case that changes to weather.com would require changes to more than just the regular expression, with changes needed to the other .inc files that include all the measures supporting the data in the skin. Changes to the StringIndex numbers associated with one or more child measures would need to be done there.

I just feel like this has a high risk of being more trouble than it is worth. On first blush, I just don't feel good about tying people's skins to the availability and accuracy of stuff on rainmeter.net. I don't feel like this is a role we want to play.
User avatar
Cariboudjan
Posts: 52
Joined: May 12th, 2019, 8:55 am

Re: Weather.com WXDATA feed failing again

Post by Cariboudjan »

That's why I mentioned a weather plugin instead. Instead of using webparser for weather data and assigning each specific piece of information to a StringIndex number (which would change), the specific points of data, regardless of their arrangement, are assigned to a specific variable.

So if Tomorrow's Wind Direction was previously a StringIndex 7 and it's been changed to a StringIndex 9 in the new XML, the new XML or the new Rainmeter update would reflect that. Calling something like...

Code: Select all

[WeatherPlugin]
Measure=Plugin
Plugin=WeatherPlugin

[ChildMeasure1]
Measure=WeatherPlugin
Day=2
Info=WindDirection
would reflect the information from StringIndex 7 or StringIndex 9 depending on whichever version of the XML you currently have. If the version number of the XML file is out of date, and 7 is wrong, it will correct itself when Rainmeter restarts. Meanwhile ALL existing skins continue humming along as if nothing even happened. And now all existing skins using the WeatherPlugin would continue to have the correct data, even if the source of the weather data needs to change, the authors are calling by measure rather than regexp/stringindex, meaning that the weather data will always filter down to the skins and always be compatible with any changes, regardless of how the author has written their skin.

Right now I plan to host this xml file for my own use. I will use a webparser to access the xml content, then use another webparser to use the other webparser's string as its regexp, then use ambiguous [childmeasure1], [childmeasure2], [childmeasure3], etc. for each specific StringIndex child measure, and assign each StringIndex= a variable #TodayHighTemp#, #TomorrowHighTemp#, #TomorrowWindDirection#, etc.

Then I can assign strings like so...

Code: Select all

[TomorrowWindDirection]
Meter=String
MeterStyle=WeatherStyle
Text=[ChildMeasure#TomorrowWindDirection#]
Using another webparser, I gather the values of these variables that !WriteKeyValue if the version number has changed. So today it might look like...

TodayHighTemp=1
TomorrowHighTemp=4
TomorrowWindDirection=7

And tomorrow it could be changed to...

TodayHighTemp=2
TomorrowHighTemp=3
TomorrowWindDirection=9

By doing all of this I can guarantee that my skin will work with all current users even if the WebParser, Source, or arrangement of the StringIndex changes. Any changes made to the XML file will filter down to all existing users without needing to update their current version of the suite. When they connect to the internet, it will automatically !WriteKeyValue all necessary changes necessary to keep the weather skin working, even if the source of the data itself needs to be totally changed. This is the plan anyway.

I don't see why it wouldn't work. However, I'd prefer not to take this route. I really think implementing a solid solution for all users, new and old, is a beneficial pursuit for Rainmeter. Please please consider it. It's 1 man's work on a mothership XML that could prevent the same amount of work collectively for thousands of people, not to mention preventing hundreds of skins from spontaneously breaking on tens of thousands of computers.

All of the time you would spend occasionally updating an XML file would be mitigated by all of the time saved not needing to respond to forum threads from people wondering why their weather skin stopped working.
User avatar
jsmorley
Developer
Posts: 20282
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Weather.com WXDATA feed failing again

Post by jsmorley »

I will continue to chew on this, but here are my thoughts at this time.

1) It's more complicated than just hosting some XML that has the regular expression for parsing the site. The approach uses several @Include .inc files, that can't be just "read" or "parsed", but must be download and put in a spot on the end-user's hard drive that is appropriate for the skin they are being used in. I just don't see how we can ever accomplish this in a way that is at once easy to maintain and easy to use. Not without a completely different approach at least.

2) We are just NOT going to bake this into Rainmeter code. We are just NOT going to have updates to Rainmeter be driven by the whims of some web designer on a web site we have absolutely no control over. We are not going to create a plugin, and we are not going to bake it in. If someone else wants to create a plugin for this, more power to them. I promise that they will tire of the effort, and the plugin will be orphaned and useless sooner or later.

3) Even if we did jump through hoops to get this working, I'm not thrilled with the implied "constraint" on how authors design and develop their skins. I know no author "has" to sign up for this approach, but except for the most ambitious author, all the chatter on the site will be about using this approach, and how you are a knucklehead if you don't. We provide WebParser, you decide how to use it. Use weather.com, use Dark Sky, use Forcea, it's up to you. There is no single "right way" to approach this.

4) And probably the deciding factor for me is that I don't feel like the Rainmeter project wants or needs to take "ownership" of this issue. Things can change with any skin, for any purpose, that uses WebParser to access information from the internet. The Wild Wild Web is a big and unruly place. While I understand the charm of having us take responsibility for keeping those skins working, I personally won't sign up the project team for this responsibility. Over my dead body really...

At least for now, I will continue to provide the help I can with the regular expression and @Include files, and will do my best to keep them current, but at the end of the day it's up to the skin authors to manage the skins they distribute, and the end-users they distribute them to.

Just stay tuned to this thread:

https://forum.rainmeter.net/viewtopic.php?f=118&t=34470
User avatar
SilverAzide
Posts: 689
Joined: March 23rd, 2015, 5:26 pm

Re: Weather.com WXDATA feed failing again

Post by SilverAzide »

+1, FWIW.... I don't want to see Rainmeter turned into a de facto, pseudo-"weather API", based on page-scraping logic at the mercy of a disinterested third-party (or, in fact, a hostile third party). Rainmeter needs to stay what it is.
DeviantArt Gadgets More...
User avatar
balala
Rainmeter Sage
Posts: 9781
Joined: October 11th, 2010, 6:27 pm
Location: Gheorgheni, Romania

Re: Weather.com WXDATA feed failing again

Post by balala »

SilverAzide wrote:
January 31st, 2020, 5:51 pm
I don't want to see Rainmeter turned into a de facto, pseudo-"weather API", based on page-scraping logic at the mercy of a disinterested third-party (or, in fact, a hostile third party). Rainmeter needs to stay what it is.
Completely agree:
User avatar
kyriakos876
Posts: 921
Joined: January 30th, 2017, 2:01 am
Location: Greece

Re: Weather.com WXDATA feed failing again

Post by kyriakos876 »

Hello guys, I found this (this is the weather for Greece,Athens for example)

https://www.weatherzone.com.au/world/europe/greece/athens

which has an RSS feed which goes like (tap the icon circled in the image below for the RSS feed) :

https://rss.weatherzone.com.au/?u=12994-1285&lt=twcid&lc=160252&obs=1&fc=1

and it seems to be in par with what google says for some cities I searched, tho I I'm not sure of an elegant way to get the code of the link for the city other than scraping the Icon of the RSS feed in that first link above...
I also noticed it may not have all the information some people want but for a simple weather skin should be okay.
As far as Icons go, one could try different weather conditions in this link: http://www.weatherzone.com.au/images/icons/fcast_30/mostly_sunny.gif
not an elegant way but yeah... I'm sure there's documentation somewhere in their site but I can't find it rn. I'll update this if I find something.
rss.png
You will also probably have to convert for Fahrenheit in your code but that's pretty simple. I haven't notice a setting, or a change in the link that converts to Fahrenheit so yeah.... feel free to explore. I'll also try to make a python script that can be hosted in one of many online python hosting platforms, that spits an RSS in a google doc or something, that can then be parsed easily. That's it for now. Peace :great:
You do not have the required permissions to view the files attached to this post.
djamman
Posts: 4
Joined: January 3rd, 2020, 12:17 pm

Weather.com location

Post by djamman »

Weather.com is now giving me bad (old?) data using the URL for the current weather with the US zipcode (85375) as the location. It works better using the long hexadecimal string. But the current temperature is still incorrect.

Location= 85375 - temp = 66
Location= 8d0d136bd6e8938b299a137d4548c333b74466bcb6ab7869bb0b338912697b29 - temp = 57
Accuweather or local thermometer - temp is 46 or 47

Doug

Edit: This is probably a local issue - other zip codes seem to be ok. Moderator should probably delete this post.
xenium
Posts: 474
Joined: January 4th, 2018, 9:52 pm

Re: Weather.com location

Post by xenium »

djamman wrote:
February 1st, 2020, 11:18 am
Weather.com is now giving me bad (old?) data using the URL for the current weather with the US zipcode (85375) as the location. It works better using the long hexadecimal string. But the current temperature is still incorrect.

Location= 85375 - temp = 66
Location= 8d0d136bd6e8938b299a137d4548c333b74466bcb6ab7869bb0b338912697b29 - temp = 57
Accuweather or local thermometer - temp is 46 or 47

Doug

Edit: This is probably a local issue - other zip codes seem to be ok. Moderator should probably delete this post.
First of all, use Us zipcode in URL
When you are on the page with your city click on the Today tab.
The long code to use will appear in the link
It seems that TWC associates a long code with a US zipcode
User avatar
jsmorley
Developer
Posts: 20282
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Weather.com location

Post by jsmorley »

xenium wrote:
February 1st, 2020, 12:25 pm
First of all, use Us zipcode in URL
When you are on the page with your city click on the Today tab.
The long code to use will appear in the link
It seems that TWC associates a long code with a US zipcode
If I use the old style code for my location in the skin:

USVA0944

Found by carefully drilling down in https://weather.codes/
https://weather.codes/united-states-of-america/virginia/

Don't bother using their search box, it doesn't work... Scroll to the bottom of the page, click on your "country", then click on whatever "region or state" is correct, then use CTRL-F in your browser to search for your exact location.

That returns a location name of "Fort Hunt, VA", which is correct.

And if I look in the HTML/json that is returned, the latitude and longitude being used is within a mile of my actual location. Very, very close and accurate indeed.

It's exactly the same latitude and longitude referenced in https://en.wikipedia.org/wiki/Fort_Hunt,_Virginia

While it doesn't say in the code, I think the weather observation for this code is done at a small airfield at a military base "Ft. Belvoir" that is just down the road a bit from me. The old wxdata return used to tell me that.


If I use my zipcode of 22308, or even search on the site for "Fort Hunt, VA", it returns the new long code of:

b48efc57780b9849d26b01c8abf4cc4e268471cb02e8054dfd9801206d2f9add

Which when used in a skin will return a location name of "Alexandria, VA" and a latitude and longitude that is 10 miles or more distant from my actual location. While the differences in weather are going to be slight, there are differences.

I'm betting that with this code the weather observation is done at "Reagan National Airport" near Alexandria, which is many miles from me.

Fort Hunt, Virginia is just a local "area". Alexandria, Virginia is an actual "city", and my zipcode, and in fact my mailing address both resolve to "Alexandria, VA". However, I don't care what temperature it is in the city of Alexandria, 10-15 miles away, I care what temperature it is in Fort Hunt, walking distance from my house. That's why I continue to use the old code in my skin.


Having said that, at the end of the day, from the standpoint of accuracy, I'm not sure it makes a huge amount of difference if you use the old code, a US zipcode or the new code. I'm just hoping that it doesn't, and won't ever, make any difference in how the site is parsed. That could get ugly indeed.
Licoeurne
Posts: 1
Joined: February 4th, 2020, 7:35 am

Re: Weather.com WXDATA feed failing again

Post by Licoeurne »

Thank you very much! :great: