It is currently July 7th, 2020, 3:17 am

Scraping Weather.com

General topics related to Rainmeter.
mak_kawa
Posts: 717
Joined: December 30th, 2015, 9:47 am

Scraping Weather.com

Post by mak_kawa »

I am really swayed by the recent behavior of weather.com html feed. Maybe it is(was) discussed in this thread and also other several threads, that is "known issue". But what I have surprised is that the issue is solved by itself *without* any "clear IE cookies"...

Used URL and RegExp are as follows, and they work well in RainRegExp testing/parsing with no problem.

Code: Select all

URL=https://weather.com/forecast/allergy/l/(Location Code)
RegExp=(?siU).*Pollen Breakdown.*Tree Pollen.*</circle></svg></div>(.*): <strong>(.*)</strong>.*</circle></svg></div>(.*): <strong>(.*)</strong>.*</circle></svg></div>(.*): <strong>(.*)</strong>.*Grass Pollen.*</circle></svg></div>(.*): <strong>(.*)</strong>.*</circle></svg></div>(.*): <strong>(.*)</strong>.*</circle></svg></div>(.*): <strong>(.*)</strong>.*Ragweed Pollen.*</circle></svg></div>(.*): <strong>(.*)</strong>.*</circle></svg></div>(.*): <strong>(.*)</strong>.*</circle></svg></div>(.*): <strong>(.*)</strong>.*
When I load/refresh the skin with the WebParser measure using above options, RegExp error has occurred. The reason is that the contents of obtained html source is "for mobile device". It is natural that the error occurs. And I have deleted IE11 cookies and set UserAgent= in the WebParser measure referring the discussion at the above-mentioned thread, but no luck at all.

However, oh however... when I leave the failed skin working as it is, after one or two updates of the WebParser measure, the html feed suddenly becomes "for desktop device" as expected, so the measure returns right results. I did actually nothing at all, no deletion of IE cookies.
At the next load/refresh, obtained html source is "for mobile" again, and after update, it becomes "for desktop" again and everything is OK again... Really weird mystery possibly caused by the discretion of Weather.com's web engineer.

So, to avoid this glitch (for me), I temporarily set the mouse action as [!CommandMeasure (WebParser measure name) Update] to a button. This solves the issue manually and even instantly.

Anyone encountered such odd behavior of weather.com and WebParser measure?!
Sorry if this is a "well-known and needless-to-say" issue.
User avatar
balala
Rainmeter Sage
Posts: 10968
Joined: October 11th, 2010, 6:27 pm
Location: Gheorgheni, Romania

Re: Scraping Weather.com

Post by balala »

mak_kawa wrote:
June 2nd, 2020, 11:31 pm
So, to avoid this glitch (for me), I temporarily set the mouse action as [!CommandMeasure (WebParser measure name) Update] to a button. This solves the issue manually and even instantly.
Just to not have to manually click (or do whatever), just to get the skin working, you should try to add the following OnRefreshAction option to the [Rainmeter] section of your skin: OnRefreshAction=[!Delay "5000"][!CommandMeasure "(WebParser measure name)" "Update"]. If you do this, when you load the skin, you'll get it updated after five seconds (as you probably know, due to the 5000 parameter of the !Delay bang), without having to click, or do anything. Maybe you'll have to make a few tries to figure out the best value, but I suppose this method should work.
mak_kawa
Posts: 717
Joined: December 30th, 2015, 9:47 am

Re: Scraping Weather.com

Post by mak_kawa »

Hi balala

Yes exactly, but I hesitate a little to scrape weather.com again after first scraping...:-)

By the way, since this morning, my Weather.com scraping skin is completely broken because of its "new-style" and "new-formatted" html source. IE cookie removal and continuous skin update don't work anymore. I must fully rebuild my RegExp(s)...

Probably the Weather.com is in its transitional situation. But I am not sure about its future and stability. Honestly, I am being buffeted by Weather.com for a long time... So, I am considering that I have to get rid of Weather.com, seriously. Some weather APIs as Weatherbit, OpenWeatherMap, AccuWeather, and DarkSky (unfortunately being dead) kindly welcome me . :-)
User avatar
balala
Rainmeter Sage
Posts: 10968
Joined: October 11th, 2010, 6:27 pm
Location: Gheorgheni, Romania

Re: Scraping Weather.com

Post by balala »

mak_kawa wrote:
June 4th, 2020, 12:14 am
Yes exactly, but I hesitate a little to scrape weather.com again after first scraping...:-)
Don't think it would be a problem.
mak_kawa wrote:
June 4th, 2020, 12:14 am
By the way, since this morning, my Weather.com scraping skin is completely broken because of its "new-style" and "new-formatted" html source. IE cookie removal and continuous skin update don't work anymore. I must fully rebuild my RegExp(s)...
Until yesterday I had no problems with my weather skins, having more, all of them worked. But yesterday I got them all broken. Now have to try the described steps in order to get them again working. I'm curious how can I...
mak_kawa wrote:
June 4th, 2020, 12:14 am
Probably the Weather.com is in its transitional situation. But I am not sure about its future and stability. Honestly, I am being buffeted by Weather.com for a long time... So, I am considering that I have to get rid of Weather.com, seriously. Some weather APIs as Weatherbit, OpenWeatherMap, AccuWeather, and DarkSky (unfortunately being dead) kindly welcome me . :-)
Agree...
mak_kawa
Posts: 717
Joined: December 30th, 2015, 9:47 am

Re: Scraping Weather.com

Post by mak_kawa »

The Weather.com has introduced "Local Pollen Alert" probably since last winter. This broke the "weather alert procedure" in my skin for the Weather.com. Why/how broke? Details are omitted, sorry.
And...I just finally supported the Local Pollen Alert together with the weather alert in my skin only few days ago, spending a great deal of time with RegExp. *But & but*... it has been instantly broken now by newly formatted HTML source of the site...alas, my...:-)
Sadly I have no energy any longer to reanalyze the source and rebuild my RegExp. What happens twice will happen three times.

So, I almost have decided to "divorce" Weather.com reluctantly. Goodbye my beloved fancy Weather.com!

Ps. The Local Pollen Alert information in Weather.com is not affected by recent format change strangely and fortunately. So I incorporate it independently into my Air Quality skin. I am partly happy...
User avatar
xenium
Posts: 546
Joined: January 4th, 2018, 9:52 pm

Re: Scraping Weather.com

Post by xenium »

mak_kawa wrote:
June 4th, 2020, 1:42 pm
The Weather.com has introduced "Local Pollen Alert" probably since last winter. This broke the "weather alert procedure" in my skin for the Weather.com. Why/how broke? Details are omitted, sorry.
And...I just finally supported the Local Pollen Alert together with the weather alert in my skin only few days ago, spending a great deal of time with RegExp. *But & but*... it has been instantly broken now by newly formatted HTML source of the site...alas, my...:-)
Sadly I have no energy any longer to reanalyze the source and rebuild my RegExp. What happens twice will happen three times.

So, I almost have decided to "divorce" Weather.com reluctantly. Goodbye my beloved fancy Weather.com!

Ps. The Local Pollen Alert information in Weather.com is not affected by recent format change strangely and fortunately. So I incorporate it independently into my Air Quality skin. I am partly happy...
My Windows 10 Weather Enterprise skin( HTML source) displays both alerts: severe weather alert and pollen alert.
The website has been displaying pollen alerts for a long time.
But a month ago the design of the Allergies page was completely changed.
Capture.PNG
You do not have the required permissions to view the files attached to this post.
User avatar
fonpaolo
Moderator
Posts: 1397
Joined: April 11th, 2013, 8:08 pm
Location: Italy

Re: Scraping Weather.com

Post by fonpaolo »

I can understand you mak_kawa, I feel almost the same, but for other reasons, however I'm still very doubtful that all of these problems are generated by cookies, IE or something else...
I'm still using my weather skin and until now, it's working.

Other than that, I'm interested in the alerts, I'm not asking you for the code, but, is there a source, documentation, something that you can suggest?
mak_kawa
Posts: 717
Joined: December 30th, 2015, 9:47 am

Re: Scraping Weather.com

Post by mak_kawa »

Hi xenium and fonpaolo

Really weird... for me, the returned html source from Weather.com is completely different from previous one that I know well. Confirmation has been done with MS-Edge, IE11, Chrome, Firefox, RainRegExp, and of course WebParser. As already said, for me, IE cookie removal worked only once at the first time, but doesn't work at next time anymore.
Considering that both of you don't encounter such behavior, possibly it is due to my OS/browser/internet environments or something else, I don't know.

As for the weather and pollen alerts, probably there is a smart and effective way that "RegExp wizard" can figure out... but not me. :-)
I am really tired and bored with this issue...:-) and in any case I will quit Weather.com except the pollen info.
Untitled-1.png
Maybe it will be broken someday. Anyway, I am 50% happy with this.
You do not have the required permissions to view the files attached to this post.
Last edited by mak_kawa on June 4th, 2020, 11:57 pm, edited 2 times in total.
mak_kawa
Posts: 717
Joined: December 30th, 2015, 9:47 am

Re: Scraping Weather.com

Post by mak_kawa »

Hi fonpaolo

As far as I know, there is no documents for the weather/pollen alerts of Weather.com. I independently analyzed the source.
When I notice the string "icon-severe-alert", "SevereAlertBar" , and "warning-text" in the HTML source, I judge that the alerts are present, and change RegExp according the number of the alerts. This worked well until the "source format change"... sad.
qwak
Posts: 3
Joined: June 5th, 2020, 3:02 am

Re: Scraping Weather.com

Post by qwak »

Just curious, although I resolved my issue with weather.com (downloaded new updated skin "Windows10WeatherEnterprise"), why not use alternate weather source? Yes I am sure many of these have been tried with similar issues, but in Linux, I am using BBC Weather for the weather applet and it has world-wide codes as well.

https://www.bbc.com/weather/2643743