It is currently August 20th, 2019, 1:57 am

WebParser value as URL for new WebParser

Help with creating, editing & fixing problems with skins
User avatar
qwerky
Posts: 181
Joined: April 10th, 2014, 12:31 am
Location: Canada

WebParser value as URL for new WebParser

qwerky » February 7th, 2019, 10:50 pm

Hi. I have a WebParser StringIndex which returns a URL (https://weather.gc.ca/warnings/report_e.html?on61). How can I use this returned value as the URL in a new WebParser measure?
User avatar
jsmorley
Developer
Posts: 19383
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: WebParser value as URL for new WebParser

jsmorley » February 7th, 2019, 11:05 pm

Code: Select all

[FakeSite]
Measure=String
String=https://weather.gc.ca/warnings/report_e.html?on61

[MeasureGetReferredURL]
Measure=WebParser
URL=[&FakeSite]
RegExp=(?siU)^(.*)$
DynamicVariables=1
The key is using [&MeasureName] and DynamicVariables=1 when you want to use the value of a measure as all, or part, of the URL option on a WebParser parent measure.

You are absolutely going to want a FinishAction on the very first parent measure, one that sends [!CommandMeasure SecondParentMeasure "Update"] to the second parent measure when it is done. If you don't, the second parent measure will initially fail as there is no value yest for the child measure it is asking for, and it won't try again for 10 minutes (UpdateRate) by default.
User avatar
jsmorley
Developer
Posts: 19383
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: WebParser value as URL for new WebParser

jsmorley » February 7th, 2019, 11:21 pm

In fact I'd probably use something like:

Code: Select all

[FakeSite]
Measure=String
String=https://weather.gc.ca/warnings/report_e.html?on61
FinishAction=[!SetOption MeasureGetReferredURL URL "[*&FakeSite*]"][!CommandMeasure MeasureGetReferredURL "Update"]

[MeasureGetReferredURL]
Measure=WebParser
RegExp=(?siU)^(.*)$
DynamicVariables=1
So I don't even set the value for URL on the second parent until the first parent is done. That will eliminate any bogus initial errors in the log as well.

Obviously you can't use FinishAction on a String measure, but this is just a non-working example, since I don't have enough of your code to really test properly.

Note that FinishAction goes on "parent" WebParser measures, not "child" WebParser measures.

So it's going to be:

[FirstParent]
FinishAction setting [*&FirstChild*] as the URL for [SecondParent]

[FirstChild]
URL=[FirstParent]
StringIndex=1

[SecondParent]
User avatar
qwerky
Posts: 181
Joined: April 10th, 2014, 12:31 am
Location: Canada

Re: WebParser value as URL for new WebParser

qwerky » February 7th, 2019, 11:57 pm

Thank you. So this is the main (first parent) measure, which reads and parses the main weather page:

Code: Select all

[msrSite]
Measure=WebParser
URL=#PageURL#
RegExp=#regexBase#
UpdateRate=#SiteUpdateRate#
FinishAction=[!SetOption msrHeaderAlertPage URL "[*&msrHeaderAlertURL*]"][!CommandMeasure msrHeaderAlertPage "Update"]
Next, this measure reads and parses the Header group from the first measure (and its results will be used by a subsequent WebParser measure, so is it a parent or child?):

Code: Select all

[msrHeaderGroup]
Measure=WebParser
URL=[msrSite]
RegExp=#regexHeader#
StringIndex=#IndexHeader#
Now this measure obtains from the above measure, the full URL, and strips off the trailing part:

Code: Select all

[msrHeaderAlertURL]
Measure=WebParser
URL=[msrHeaderGroup]
StringIndex=#HeaderIndexAlertURL#
RegExpSubstitute=1
Substitute="^(.*)#.*$":"\1"
Finally, this is the new parent measure which hopes to use the URL from the preceding measure:

Code: Select all

[msrHeaderAlertPage]
Measure=WebParser
;URL=[&msrHeaderAlertURL]
regexBase=(?siU).*<p>.*>(.*)<section
;StringIndex=1
DynamicVariables=1
I wasn't sure where to place the FinishAction (it's shown here on the very first parent measure), but no matter what variations I try, Rainmeter About always shows [msrHeaderAlertPage] to be empty.
User avatar
jsmorley
Developer
Posts: 19383
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: WebParser value as URL for new WebParser

jsmorley » February 8th, 2019, 12:04 am

There is no such option as regexBase.
User avatar
jsmorley
Developer
Posts: 19383
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: WebParser value as URL for new WebParser

jsmorley » February 8th, 2019, 12:09 am

I need the values set for all those #Variables#, so I can actually test and find the best solution.
User avatar
qwerky
Posts: 181
Joined: April 10th, 2014, 12:31 am
Location: Canada

Re: WebParser value as URL for new WebParser

qwerky » February 8th, 2019, 12:14 am

jsmorley wrote:
February 8th, 2019, 12:04 am
There is no such option as regexBase.
Oh, dear me! Stupid, :oops: stupid :? , stupid :? . Thanks for pointing that out.
User avatar
qwerky
Posts: 181
Joined: April 10th, 2014, 12:31 am
Location: Canada

Re: WebParser value as URL for new WebParser

qwerky » February 8th, 2019, 12:23 am

jsmorley wrote:
February 8th, 2019, 12:09 am
I need the values set for all those #Variables#, so I can actually test and find the best solution.
Okay, here we go:

Code: Select all

[msrSite]
Measure=WebParser
URL=https://weather.gc.ca/city/pages/on-143_metric_e.html
RegExp=(?siU)property="mainContentOfPage"(.*)</ul>(.*)<h2>Forecast.*Forecast(.*)>(.*)Detailed Forecast(.*)Normals:(.*)<h2>Averages(.*)<h2>Yesterday(.*)</details>
UpdateRate=600
FinishAction=[!SetOption msrHeaderAlertPage URL "[*&msrHeaderAlertURL*]"][!CommandMeasure msrHeaderAlertPage "Update"]

[msrHeaderGroup]
Measure=WebParser
URL=[msrSite]
RegExp=(?siU).*property="name">(.*), <abbr title="(.*)">(.*)</abbr>.*<a href="(.*)">(?(?=<).*</div>.*class.*>)(.*)</.*href="(.*)">Past
StringIndex=1

[msrHeaderAlertURL]
Measure=WebParser
URL=[msrHeaderGroup]
StringIndex=4
RegExpSubstitute=1
Substitute="^(.*)#.*$":"\1"
(This should resolve to:  "https://weather.gc.ca/warnings/report_e.html?on61")

[msrHeaderAlertPage]
Measure=WebParser
;URL=[&msrHeaderAlertURL]
UpdateRate=600
RegExp=(?siU).*<p>.*>(.*)<section
StringIndex=1
DynamicVariables=1
User avatar
jsmorley
Developer
Posts: 19383
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: WebParser value as URL for new WebParser

jsmorley » February 8th, 2019, 12:30 am

This:

Code: Select all

[msrHeaderAlertURL]
Measure=WebParser
URL=[msrHeaderGroup]
StringIndex=4
RegExpSubstitute=1
Substitute="^(.*)#.*$":"\1"
(This should resolve to:  "https://weather.gc.ca/warnings/report_e.html?on61")
Does NOT resolve to https://weather.gc.ca/warnings/report_e.html?on61, it resolves to /warnings/report_e.html?on61, so it is not a fully qualified URL. You can solve that though:

Code: Select all

FinishAction=[!SetOption msrHeaderAlertPage URL "https://weather.gc.ca[*&msrHeaderAlertURL*]"][!CommandMeasure msrHeaderAlertPage "Update"]
So as you can see, we append the first bit, https://weather.gc.ca before we set the URL on [msrHeaderAlertPage].

1.jpg
You do not have the required permissions to view the files attached to this post.
User avatar
qwerky
Posts: 181
Joined: April 10th, 2014, 12:31 am
Location: Canada

Re: WebParser value as URL for new WebParser

qwerky » February 8th, 2019, 12:51 am

Oh, I think you've done it! :D

But unfortunately, I won't be able to test until tomorrow. Silly me :oops: , I think I've been blocked from the weather site! I forgot to put a UpdateRate=600 on that second parent meter in my main skin, so it was likely hitting the site every second. :?

But I can still access the weather site in a browser, so it seems it is only blocking Rainmeter, not everything from my IP. If I don't access it any more today, hopefully by tomorrow the block will be removed.