Page 1 of 2
WebParser value as URL for new WebParser
Posted: February 7th, 2019, 10:50 pm
by qwerky
Hi. I have a WebParser StringIndex which returns a URL (
https://weather.gc.ca/warnings/report_e.html?on61). How can I use this returned value as the URL in a new WebParser measure?
Re: WebParser value as URL for new WebParser
Posted: February 7th, 2019, 11:05 pm
by jsmorley
Code: Select all
[FakeSite]
Measure=String
String=https://weather.gc.ca/warnings/report_e.html?on61
[MeasureGetReferredURL]
Measure=WebParser
URL=[&FakeSite]
RegExp=(?siU)^(.*)$
DynamicVariables=1
The key is using [
&MeasureName] and DynamicVariables=1 when you want to use the value of a measure as all, or part, of the URL option on a WebParser parent measure.
You are absolutely going to want a FinishAction on the very first parent measure, one that sends [!CommandMeasure SecondParentMeasure "Update"] to the second parent measure when it is done. If you don't, the second parent measure will initially fail as there is no value yest for the child measure it is asking for, and it won't try again for 10 minutes (UpdateRate) by default.
Re: WebParser value as URL for new WebParser
Posted: February 7th, 2019, 11:21 pm
by jsmorley
In fact I'd probably use something like:
Code: Select all
[FakeSite]
Measure=String
String=https://weather.gc.ca/warnings/report_e.html?on61
FinishAction=[!SetOption MeasureGetReferredURL URL "[*&FakeSite*]"][!CommandMeasure MeasureGetReferredURL "Update"]
[MeasureGetReferredURL]
Measure=WebParser
RegExp=(?siU)^(.*)$
DynamicVariables=1
So I don't even set the value for URL on the second parent until the first parent is done. That will eliminate any bogus initial errors in the log as well.
Obviously you can't use FinishAction on a String measure, but this is just a non-working example, since I don't have enough of your code to really test properly.
Note that FinishAction goes on "parent" WebParser measures, not "child" WebParser measures.
So it's going to be:
[FirstParent]
FinishAction setting [*&FirstChild*] as the URL for [SecondParent]
[FirstChild]
URL=[FirstParent]
StringIndex=1
[SecondParent]
Re: WebParser value as URL for new WebParser
Posted: February 7th, 2019, 11:57 pm
by qwerky
Thank you. So this is the main (first parent) measure, which reads and parses the main weather page:
Code: Select all
[msrSite]
Measure=WebParser
URL=#PageURL#
RegExp=#regexBase#
UpdateRate=#SiteUpdateRate#
FinishAction=[!SetOption msrHeaderAlertPage URL "[*&msrHeaderAlertURL*]"][!CommandMeasure msrHeaderAlertPage "Update"]
Next, this measure reads and parses the Header group from the first measure (and its results will be used by a subsequent WebParser measure, so is it a parent or child?):
Code: Select all
[msrHeaderGroup]
Measure=WebParser
URL=[msrSite]
RegExp=#regexHeader#
StringIndex=#IndexHeader#
Now this measure obtains from the above measure, the full URL, and strips off the trailing part:
Code: Select all
[msrHeaderAlertURL]
Measure=WebParser
URL=[msrHeaderGroup]
StringIndex=#HeaderIndexAlertURL#
RegExpSubstitute=1
Substitute="^(.*)#.*$":"\1"
Finally, this is the new parent measure which hopes to use the URL from the preceding measure:
Code: Select all
[msrHeaderAlertPage]
Measure=WebParser
;URL=[&msrHeaderAlertURL]
regexBase=(?siU).*<p>.*>(.*)<section
;StringIndex=1
DynamicVariables=1
I wasn't sure where to place the FinishAction (it's shown here on the very first parent measure), but no matter what variations I try, Rainmeter About always shows [msrHeaderAlertPage] to be empty.
Re: WebParser value as URL for new WebParser
Posted: February 8th, 2019, 12:04 am
by jsmorley
There is no such option as regexBase.
Re: WebParser value as URL for new WebParser
Posted: February 8th, 2019, 12:09 am
by jsmorley
I need the values set for all those #Variables#, so I can actually test and find the best solution.
Re: WebParser value as URL for new WebParser
Posted: February 8th, 2019, 12:14 am
by qwerky
jsmorley wrote: ↑February 8th, 2019, 12:04 am
There is no such option as
regexBase.
Oh, dear me! Stupid,
stupid
, stupid
. Thanks for pointing that out.
Re: WebParser value as URL for new WebParser
Posted: February 8th, 2019, 12:23 am
by qwerky
jsmorley wrote: ↑February 8th, 2019, 12:09 am
I need the values set for all those #Variables#, so I can actually test and find the best solution.
Okay, here we go:
Code: Select all
[msrSite]
Measure=WebParser
URL=https://weather.gc.ca/city/pages/on-143_metric_e.html
RegExp=(?siU)property="mainContentOfPage"(.*)</ul>(.*)<h2>Forecast.*Forecast(.*)>(.*)Detailed Forecast(.*)Normals:(.*)<h2>Averages(.*)<h2>Yesterday(.*)</details>
UpdateRate=600
FinishAction=[!SetOption msrHeaderAlertPage URL "[*&msrHeaderAlertURL*]"][!CommandMeasure msrHeaderAlertPage "Update"]
[msrHeaderGroup]
Measure=WebParser
URL=[msrSite]
RegExp=(?siU).*property="name">(.*), <abbr title="(.*)">(.*)</abbr>.*<a href="(.*)">(?(?=<).*</div>.*class.*>)(.*)</.*href="(.*)">Past
StringIndex=1
[msrHeaderAlertURL]
Measure=WebParser
URL=[msrHeaderGroup]
StringIndex=4
RegExpSubstitute=1
Substitute="^(.*)#.*$":"\1"
(This should resolve to: "https://weather.gc.ca/warnings/report_e.html?on61")
[msrHeaderAlertPage]
Measure=WebParser
;URL=[&msrHeaderAlertURL]
UpdateRate=600
RegExp=(?siU).*<p>.*>(.*)<section
StringIndex=1
DynamicVariables=1
Re: WebParser value as URL for new WebParser
Posted: February 8th, 2019, 12:30 am
by jsmorley
This:
Code: Select all
[msrHeaderAlertURL]
Measure=WebParser
URL=[msrHeaderGroup]
StringIndex=4
RegExpSubstitute=1
Substitute="^(.*)#.*$":"\1"
(This should resolve to: "https://weather.gc.ca/warnings/report_e.html?on61")
Does NOT resolve to
https://weather.gc.ca/warnings/report_e.html?on61, it resolves to
/warnings/report_e.html?on61, so it is not a fully qualified URL. You can solve that though:
Code: Select all
FinishAction=[!SetOption msrHeaderAlertPage URL "https://weather.gc.ca[*&msrHeaderAlertURL*]"][!CommandMeasure msrHeaderAlertPage "Update"]
So as you can see, we append the first bit,
https://weather.gc.ca before we set the URL on [msrHeaderAlertPage].
1.jpg
Re: WebParser value as URL for new WebParser
Posted: February 8th, 2019, 12:51 am
by qwerky
Oh, I think you've done it!
But unfortunately, I won't be able to test until tomorrow. Silly me
, I think I've been blocked from the weather site! I forgot to put a
UpdateRate=600 on that second parent meter in my main skin, so it was likely hitting the site every second.
But I can still access the weather site in a browser, so it seems it is only blocking Rainmeter, not everything from my IP. If I don't access it any more today, hopefully by tomorrow the block will be removed.