It is currently March 28th, 2024, 4:43 pm

Scraping Weather.com

General topics related to Rainmeter.
mak_kawa
Posts: 908
Joined: December 30th, 2015, 9:47 am

Re: Scraping Weather.com

Post by mak_kawa »

Hi qwak

You have resolved the issue with Weather.com? Could you provide some information for the HTML source? That is...does the string "nowcard-location" exist before your location name in the source? If so, it is the legacy source format *before the disaster*. If not, it is the newly formatted damned :-) source that broke my RegExp parsing(s). And probably your WebParser uses RegExp(s) that was already modified to support the new html structure.

BTW, I didn't know about the BBC Weather until now. At a glance, it seems to support many global locations, such as East Asia... maybe another promising site for me?
qwak
Posts: 3
Joined: June 5th, 2020, 3:02 am

Re: Scraping Weather.com

Post by qwak »

mak_kawa wrote: June 5th, 2020, 5:27 am Hi qwak

You have resolved the issue with Weather.com? Could you provide some information for the HTML source? That is...does the string "nowcard-location" exist before your location name in the source? If so, it is the legacy source format *before the disaster*. If not, it is the newly formatted damned :-) source that broke my RegExp parsing(s). And probably your WebParser uses RegExp(s) that was already modified to support the new html structure.

BTW, I didn't know about the BBC Weather until now. At a glance, it seems to support many global locations, such as East Asia... maybe another promising site for me?
As I mentioned, I resolved it by using an updated skin:
https://forum.rainmeter.net/viewtopic.php?t=31607

This involves doing a search for the city/location in weather.com and using the end bit in the resulting url as code for the skin.
mak_kawa
Posts: 908
Joined: December 30th, 2015, 9:47 am

Re: Scraping Weather.com

Post by mak_kawa »

Hi qwak

You are talking about xenium's weather skin using JSON data of the Weather.com. Sorry, I was talking about old-style "non-JSON" information in the source from the site.

BTW, I have now changed my mind about the divorce from Weather.com. :-)
From now, I will rebuild RegExp in my Weather.com skin to extract data from the JSON part of the source. At a glance, the JSON data format is rather more simple than I presumed, and quite informative. But maybe a major modification for my skin is needed.
Of course, I know jsmorley(and others)'s concentrated work on the JSON data of the Weather.com source. But I want to make it by myself as "a Rainmeter hobbyist". It might be a fun...
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Scraping Weather.com

Post by jsmorley »

The biggest reason why I would try hard to stick with weather.com is that as far as I can tell, it is the single most "current" observation you are likely to find. The current conditions are updated pretty much every 10 minutes, which is far better than most sites, that update every hour or even longer. I really don't care what the weather was an hour ago, I'm already getting a wedgie from my soaked underwear by that point... I'm looking at you, Yahoo.

Time will tell, but I'm hopeful that they will only do a radical change to the JSON rarely. No doubt changing the JSON causes some amount of downstream work on their end as well, so they probably won't do it casually or terribly often. Let's hope anyway.

I've said this before, and I will say it again, I DON'T recommend parsing the HTML in any way. That is entirely different. That can and will change often, based on the whim of some propeller-hat webmaster or soulless marketing weasel.
User avatar
pul53dr1v3r
Posts: 442
Joined: July 30th, 2014, 10:30 am

Re: Scraping Weather.com

Post by pul53dr1v3r »

Has anyone noticed mistakes where on the site with metric units choosen some imperial units appear such as visibility and wind speed?
Seems that weather.com is not finished the job yet with their site.

In my weather skin i corrected it converting the values in measures.

Noticed it in Silver's skin too, but it obviously has nothing to with the code:

Image
Margrave
Posts: 2
Joined: June 6th, 2020, 8:35 pm

Re: Scraping Weather.com

Post by Margrave »

I wish the Win10 Widgets weather could get fixed/updated.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Scraping Weather.com

Post by jsmorley »

pul53dr1v3r wrote: June 6th, 2020, 8:19 pm Has anyone noticed mistakes where on the site with metric units choosen some imperial units appear such as visibility and wind speed?
Seems that weather.com is not finished the job yet with their site.

In my weather skin i corrected it converting the values in measures.

Noticed it in Silver's skin too, but it obviously has nothing to with the code:

Image
If you select en-GB (Great Britain) as the language you get "hybrid" units. Most stuff is metric, but wind speed is in mph and distance is in miles. If you want English, but pure metric, I suggest en-CA (Canada).

The Brits are fine with all that Celsius and Millibars nonsense, but have not given up Miles in favor of Kilometers in common practice. Check their speed-limit signs...

This "hybrid" business is new to the JSON, it used to be that en-GB was purely metric. Just need to keep that in mind.


us.jpg
en-US (United States)

gb.jpg
en-GB (Great Britain)

ca.jpg
en-CA (Canada)
mak_kawa
Posts: 908
Joined: December 30th, 2015, 9:47 am

Re: Scraping Weather.com

Post by mak_kawa »

Hi jsmorley
I DON'T recommend parsing the HTML in any way.
Yes I understand bitterly. :-) But it is a sort of "unavoidables" at least for me... I want to get rid of any scraping HTMLs in Rainmeter, but impossible so far.

BTW, I have built a weather skin that extracts current weather state from the Weather.com JSON. But for the daily forecasts and hourly forecasts, I am still struggling. For the weather warnings, I don't get started yet. They might be easy for Rainmeter wizards, but not me.
And on this occasion, I could switch source for my air quality and pollen info skin to Weather.com JSON. It is really informative surprisingly.
User avatar
eclectic-tech
Rainmeter Sage
Posts: 5382
Joined: April 12th, 2012, 9:40 pm
Location: Cedar Point, Ohio, USA
Contact:

Re: Scraping Weather.com

Post by eclectic-tech »

Margrave wrote: June 6th, 2020, 8:39 pm I wish the Win10 Widgets weather could get fixed/updated.
It will be in the next week or so... I only have 2 hands and 1 WIFE (happy wife=happy life!)
User avatar
pul53dr1v3r
Posts: 442
Joined: July 30th, 2014, 10:30 am

Re: Scraping Weather.com

Post by pul53dr1v3r »

jsmorley wrote: June 6th, 2020, 8:47 pm If you select en-GB (Great Britain) as the language you get "hybrid" units. Most stuff is metric, but wind speed is in mph and distance is in miles. If you want English, but pure metric, I suggest en-CA (Canada).

The Brits are fine with all that Celsius and Millibars nonsense, but have not given up Miles in favor of Kilometers in common practice. Check their speed-limit signs...

This "hybrid" business is new to the JSON, it used to be that en-GB was purely metric. Just need to keep that in mind.



us.jpg



gb.jpg



ca.jpg
ah so. As that wasn't case with old JSON, i thought it's their mistake, but now i'm on the one hand happy as the site seems finally settled down, but on the other hand i'm not as my primary language when work with the weather skin is en-GB, and now there are inaccurate values for wind speed and visibility in other languages, but easy to fix. :Whistle i know that their cars (with steering wheel on the right side) uses mph as unit of speed but was used to the old weather.com.
Thanks for the explanation. :great:
Post Reply