It is currently March 29th, 2024, 1:50 am

Yet Another Web Parser Request - I Think I'm Close

Get help with creating, editing & fixing problems with skins
User avatar
Mor3bane
Posts: 943
Joined: May 7th, 2016, 7:32 am

Yet Another Web Parser Request - I Think I'm Close

Post by Mor3bane »

I have decided to parse a web page that shows the world population increasing in real time.
I have slowed it down to once a second as the site updates really fast.

Here's what I have got so far - it doesn't work.

Code: Select all

[WorldPop]
Measure=Plugin
Plugin=Plugins\WebParser.dll
UpdateDivider=500
Url=http://www.theworldcounts.com/counters/shocking_environmental_facts_and_statistics/world_population_clock_live
RegExp="(?siU)<p class='counter'>id= .*counters_number_interval_5'>(.*)</p>.*"
Thanks for any time on this :great:
My DevArt Gallery

There are many ways to be different - there is only one way to be yourself - be amazing at it

The law of averages says what it means; even if you get everything right, you will get something wrong. Therefore; self managing error trapping initiates another set of averages - amongst the errors, some of them will not be errors, instead those instances will appear to be "luck". One cannot complain of the 'appearance' of 'infinite regress of causation', even if it does not have a predictable pattern, only that it requires luck to achieve.
User avatar
FreeRaider
Posts: 826
Joined: November 20th, 2012, 11:58 pm

Re: Yet Another Web Parser Request - I Think I'm Close

Post by FreeRaider »

Apart that RegExp is RegExp=(?siU)<p class='counter' id='counters_number_interval_5'>(.*)</p>.*, but it returns "loading..."
User avatar
Mor3bane
Posts: 943
Joined: May 7th, 2016, 7:32 am

Re: Yet Another Web Parser Request - I Think I'm Close

Post by Mor3bane »

FreeRaider wrote:Apart that RegExp is RegExp=(?siU)<p class='counter' id='counters_number_interval_5'>(.*)</p>.*, but it returns "loading..."
Bah - I didn't even get that with out debug :P
watamidoing wrong?
My DevArt Gallery

There are many ways to be different - there is only one way to be yourself - be amazing at it

The law of averages says what it means; even if you get everything right, you will get something wrong. Therefore; self managing error trapping initiates another set of averages - amongst the errors, some of them will not be errors, instead those instances will appear to be "luck". One cannot complain of the 'appearance' of 'infinite regress of causation', even if it does not have a predictable pattern, only that it requires luck to achieve.
User avatar
eclectic-tech
Rainmeter Sage
Posts: 5384
Joined: April 12th, 2012, 9:40 pm
Location: Cedar Point, Ohio, USA

Re: Yet Another Web Parser Request - I Think I'm Close

Post by eclectic-tech »

Mor3bane wrote:Bah - I didn't even get that with out debug :P
watamidoing wrong?
If you access the site once every few seconds, you will probably be blocked, thinking it is an attack... :uhuh:

This should get you the current number RegExp="(?siU).*<p class='counter' id='counters_number_interval_5'>.*number_interval\((.*),.*"

Returns 7490337325.513295

I would try to find an alternate method that would average the numbers returned, only parsing the site every few minutes...
User avatar
balala
Rainmeter Sage
Posts: 16110
Joined: October 11th, 2010, 6:27 pm
Location: Gheorgheni, Romania

Re: Yet Another Web Parser Request - I Think I'm Close

Post by balala »

eclectic-tech beat me again, but I had my reply written when he posted his, so I post mine too (sorry eclectic-tech...)
FreeRaider wrote:Apart that RegExp is RegExp=(?siU)<p class='counter' id='counters_number_interval_5'>(.*)</p>.*, but it returns "loading..."
Normally. The piece of the html code containing the needed number is the following one:

Code: Select all

<div class='number'>
<p class='counter' id='counters_number_interval_5'>loading...</p>
<script>
  number_interval(7490335971.327857, 2.538563344286236, 0, 'counters_number_interval_5')
</script>
You'd need the first number from the number_interval function. But the posted RegExp (RegExp="(?siU)<p class='counter'>id= .*counters_number_interval_5'>(.*)</p>.*") should get the the string between the id='counters_number_interval_5'> and </p> parts of code. And this indeed is loading... (as FreeRaider said). Better said, it would be, but finally not even that can't be get, because in your RegExp you have some extra characters/expressions (RegExp="(?siU)<p class='counter'[color=#FF0000]>[/color]id=[color=#FF0000] .*[/color]counters_number_interval_5'>(.*)</p>.*"), which makes the plugin to not return anything.
Replace the RegExp with the following one: RegExp=(?siU)<script>.*number_interval\((.*), .*, .*, 'counters_number_interval_.*'\).*</script>. This will return the proper number, but this will be a static number, which doesn't grows accordingly to the site.
Showing the increasing number is very hard to be made accurately, because you should parse the site on each second. Beyond from the fact that the WebParser plugin can't get the data so quickly (it needs some time to parse the site), even if it could, probably sooner or later you would be blocked (read the Note here).
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Yet Another Web Parser Request - I Think I'm Close

Post by jsmorley »

I don't have time today to look at it, but I suspect there is some formula, maybe based on that second number in the "interval" stuff, that determines how fast the counter counts up. So I'd be tempted to get the "current" number every 10 minutes or so, and in between have some counter in Rainmeter that increments it to be as close as possible to the way the one on the site counts. It will "correct" for any error every 10 minutes.

At a minimum you could put debug=2 on the measure, and then run it a few times, getting the number from WebParserDump.txt between , and figure out the "average" increment per second yourself. Maybe using an OnChangeAction on a Time measure that is returning the seconds, or for faster updating, maybe something using the Counter function in Rainmeter and a low Update. I'm pretty sure there is some static formula they use, which they might adjust on some periodic basis, since I doubt they have a way to actually count babies...
User avatar
Mor3bane
Posts: 943
Joined: May 7th, 2016, 7:32 am

Re: Yet Another Web Parser Request - I Think I'm Close

Post by Mor3bane »

jsmorley wrote:I'm pretty sure there is some static formula they use, which they might adjust on some periodic basis, since I doubt they have a way to actually count babies...
I was thinking the same thing - that they couldnt be counting actual babies. :o That would take an unrealistic effort.

If you do find yourself with some time, jsmorely, perhaps the site page contains something that I didnt see.

Anyway, I will increase the UpdateDivider to minutes rather than seconds in the meantime.

Cheers folks :thumbup:

Edit: Also I don't think the page will let a parser ping it faster than some filter allows, or at least the meter does not update unless I Refresh All.
My DevArt Gallery

There are many ways to be different - there is only one way to be yourself - be amazing at it

The law of averages says what it means; even if you get everything right, you will get something wrong. Therefore; self managing error trapping initiates another set of averages - amongst the errors, some of them will not be errors, instead those instances will appear to be "luck". One cannot complain of the 'appearance' of 'infinite regress of causation', even if it does not have a predictable pattern, only that it requires luck to achieve.