It is currently February 29th, 2020, 4:06 am

weather.com - Parsing the HTML

Our most popular Tips and Tricks from the Rainmeter Team and others
User avatar
jsmorley
Developer
Posts: 20297
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Some Help With Parsing weather.com

Post by jsmorley »

SilverAzide wrote:
January 28th, 2020, 2:16 am
Okie dokie! No worries...
Mind you, I'm all for displaying some or all of the alerts in a tooltip on the icon, but I haven't yet seen an approach that allows that to happen in a consistent or reliable way.

I'm also gun shy about the fact that xenium says that the "alerts" stuff has changed like three times in the last year, while the rest of site has been consistent. I DO NOT want to be back in this shitstorm in a couple of months.
User avatar
SilverAzide
Posts: 700
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Some Help With Parsing weather.com

Post by SilverAzide »

jsmorley wrote:
January 28th, 2020, 2:02 am
It may be that there is an inconsistency with the long location id it puts in the link, and the original location code, for me it is USVA0944, and something goes haywire.
Oh yes, I forgot to mention that. If you use the old-style codes, the URL does go haywire. It only works properly with the new-style codes.
DeviantArt Gadgets More...
User avatar
jsmorley
Developer
Posts: 20297
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Some Help With Parsing weather.com

Post by jsmorley »

SilverAzide wrote:
January 28th, 2020, 2:20 am
Oh yes, I forgot to mention that. If you use the old-style codes, the URL does go haywire. It only works properly with the new-style codes.
There is no new-style code I have found that gets me to my exact location, which is Fort Hunt, VA. Only USVA0944 seems to work. The new-style code gets "close", but that only counts in horseshoes and hand-grenades.

I don't give a tinker's damn what the weather is up the road in Alexandria, they are on their own.
User avatar
SilverAzide
Posts: 700
Joined: March 23rd, 2015, 5:26 pm

Re: ⭐ Some Help With Parsing weather.com

Post by SilverAzide »

jsmorley wrote:
January 28th, 2020, 2:22 am
There is no new-style code I have found that gets me to my exact location, which is Fort Hunt, VA. Only USVA0944 seems to work. The new-style code gets "close", but that only counts in horseshoes and hand-grenades.
Agreed. In fact, I have (had???) at least 4 of the old-style codes all around my area (zip code, city, nearby airport, nearby military base). I can still search for every one but they all are called the same thing (by city name). The actual new-style code is different in every case, which is super confusing.
DeviantArt Gadgets More...
Nabu
Posts: 12
Joined: May 19th, 2016, 7:44 am

Re: ⭐ Some Help With Parsing weather.com

Post by Nabu »

jsmorley wrote:
January 27th, 2020, 2:04 pm
I can't swear I didn't use any features of Rainmeter implemented after 3.3.2. I might well have.

The only thing I can suggest is to download the .rmskin, rename it to .zip, extract it, and install it in the skins folder manually. Then load it up and see what happens.
Thanks. I did what you said. It didn't work. Not sure if I miss something. Tried it with the old WXDataTutorial skin as well, by changing WxDataTutorial.ini file. Same behaviour:

Image

Running portable installation of Rainmeter if that does matter. Any further help, please?
xenium
Posts: 474
Joined: January 4th, 2018, 9:52 pm

Re: ⭐ Some Help With Parsing weather.com

Post by xenium »

SilverAzide wrote:
January 28th, 2020, 1:37 am


One annoyance with the weather data is that the "severityCode" style of identifying the type of alert that you are using in your Weather.com skin does not seem to have a 1:1 correspondence to the codes being captured here. The codes in xenium's regexp match the color of the icon they use in the UI, but these are somewhat inconsistent with the type of alert. For example, a "moderate" alert (severityCode 3) could show up as a 2 or 3 in the UI. The "severityCode" also references all sorts of annoying stuff like marine alerts ("Small Craft Advisory", etc.). This data does not seem to ever appear in the data xenium is capturing (fortunately); it also never appeared in wxdata either, IIRC.
Image1.png
The alerts section of my code displays ALL weather alerts on the website. :D
They are sorted according to the alert level (1 to 4) exactly as on the website.
When there are multiple alerts that have the same alert level, they are displayed based on the importance that TWC considers each to have.

If you notice that my code does not display all the alerts existing on the website, this is because there are 2 versions of the Weather.com website:
1.The "new" version that uses the new codes.
This version is accessed when you open your browser and go to TWC
2.And the "old" version that uses the old codes for locations.
This version is "hidden" and can be accessed by replacing, at the new version, the long code with the old code in the link.

Between the two versions there are small differences in data display
In case of alerts there may be bigger differences.
There are situations when one version displays more alerts than the other.

So.. when you compare the data of the skin with those of the website, make sure that both use the same type of code for the location.


The biggest weirdness is that an alert can be displayed several times on the website (same text, same validity period).
I don't understand why.
User avatar
jsmorley
Developer
Posts: 20297
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ Some Help With Parsing weather.com

Post by jsmorley »

Added "reference" guide information for the various @Include files to the bottom of the first post in this thread.
User avatar
jsmorley
Developer
Posts: 20297
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

⭐ weather.com - Some Tools for Parsing

Post by jsmorley »

General discussions about parsing weather.com have been moved to this thread:

https://forum.rainmeter.net/viewtopic.php?f=4&t=34579&p=171127#p171098
User avatar
kyriakos876
Posts: 921
Joined: January 30th, 2017, 2:01 am
Location: Greece

Re: ⭐ weather.com - Some Tools for Parsing

Post by kyriakos876 »

Hello, if I understand correctly the only thing that might change is the code in the Weather.com site which would break the RegExp file.
Can't we just host this file in GitHub like I did for testing here:

https://raw.githubusercontent.com/BananaMasterz/RainWeather/master/WeatherComRegExp.inc

and then if anything changes in the weathercom site, the mods of rainmeter could update the RegExp file.

(A second file in github will host the version of the file so the user doesnt have to parse the whole thing everytime, this second file is:
https://raw.githubusercontent.com/BananaMasterz/RainWeather/master/Version)

Finally, with lua it automatically opens the RegExp file and updates it.

Here's an example:

Code: Select all

[Rainmeter]
Update=1000
MouseScrollDownAction=[!Refresh]
DynamicWindowSize=1

[Variables]
Version=7.2.2020

[GetVersion]
Measure=WebParser
Plugin=WebParser
Url=https://raw.githubusercontent.com/BananaMasterz/RainWeather/master/Version
RegExp=(?siU)^(.*)$
FinishAction=[!EnableMeasure CheckVersion][!updateMeasure CheckVersion]

[CheckVersion]
Measure=String
String=[GetVersion]
IfMatch=#Version#
IfMatchAction=[!SetOption Meter Text "No update avaliable"][!Update]
IfNotMatchAction=[!SetOption Meter Text "New update available"][!Update]
Disabled=1
DynamicVariables=1

[GetRegExp]
Measure=WebParser
Plugin=WebParser
Url=https://raw.githubusercontent.com/BananaMasterz/RainWeather/master/WeatherComRegExp.inc
RegExp=(?siU)^(.*)$
UpdateRate=600

[Meter]
Meter=String
DynamicVariables=1
So in the IfNotMatchAction we can have the lua run and update the file. Just an idea...
User avatar
jsmorley
Developer
Posts: 20297
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: ⭐ weather.com - Some Tools for Parsing

Post by jsmorley »

No, sorry.

As I said, we are just not going to host anything or take ownership of what does or doesn't change on a website we have no control over.