It is currently June 13th, 2024, 8:47 am

Backwards WebParsing?

Get help with creating, editing & fixing problems with skins
Bishop149
Posts: 25
Joined: March 23rd, 2016, 3:07 pm

Re: Backwards WebParsing?

Post by Bishop149 »

@jsmorley

Thanks for that very informative explanation, i think I mostly understand but do have a few questions about the .lua

1) When you are defining the patterns, destinationPattern, aimedPattern & expectedPattern are all subfields to be searched for within trainPattern. How is this defined? I note that for trainPattern .- is used to return the info whereas in the other 3 (.-) is used. Is this it?

Edit: Actually no, what is searched for within what is defined in the loop, so is this a mistake? Should that first .- be (.-)

If I also wanted to return the time at which the information was last refreshed (which would have no subfields) would the following do it if inserted above trainPattern:

Code: Select all

refreshedPattern = 'T.-</request_time>'
2) function Update() is blank, what is that? Its obviously not required but does it need to be there for the lua to be valid?

3) If I wanted to change the website to be parsed I assume the best approach would be to use !CommandMeasure to set the URL of [MeasureSite] and the refresh the whole skin?

Your having basically done this one for me (Many thanks!!) I'm actually trying to write another skin that's going to do something very similar on a different feed. I'll see how I get on adapting this code to the new purpose, in light of this is there a way of testing .lua code, seeing what the variables are returning etc?

I will also no doubt have questions on my new code, where is best to post them? Shall I just continue this thread?
User avatar
jsmorley
Developer
Posts: 22637
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Backwards WebParsing?

Post by jsmorley »

1) When you are defining the patterns, destinationPattern, aimedPattern & expectedPattern are all subfields to be searched for within trainPattern. How is this defined? I note that for trainPattern .- is used to return the info whereas in the other 3 (.-) is used. Is this it?

Edit: Actually no, what is searched for within what is defined in the loop, so is this a mistake? Should that first .- be (.-)
trainPattern is only used in the string.find function, and the only point there is to return the numeric positions where the string starts and ends for each "train" in the entire feed. So there is no need to (capture) anything, and no parentheses are needed. In fact, they would break it...

trainStart, trainEnd = string.find(entireFeed, trainPattern, startPos)

Note: We move from "train" to "train" by changing startPos.

After we know where in the string the first / next "train" is, we then extract just the sub-string of that train's information into a separate string with:

oneTrain = string.sub(entireFeed, trainStart, trainEnd)

We then use that oneTrain variable to do our actual matching / capturing for the various bits of information we want.
If I also wanted to return the time at which the information was last refreshed (which would have no subfields) would the following do it if inserted above trainPattern:

Code: Select all

refreshedPattern = 'T.-</request_time>'

Since that is something you are only getting once on each call to ParseSite(), and there is only one in the feed, there really isn't any need to set a variable to define the pattern to get it. In the ParseSite() function, pretty much anywhere after

entireFeed = inputMeasure:GetStringValue()

You can use something like:

feedTime = string.match(entirefeed, '<request_time>(.-)</request_time>')

Then you will have a variable feedTime that you can use SKIN:Bang to !SetOption the Text option of some meter as you like, much the same way we set the other bits of information in the skin.

Note, don't put this inside the loop, you only need to do it once per call to ParseSite().
2) function Update() is blank, what is that? Its obviously not required but does it need to be there for the lua to be valid?
Update() is not used or required in this case. I left it since in many or most cases, you will run the Lua on each update of the skin, or more to the point of the Script measure, and I just consider it part of the normal "template" for .lua files I use. I just always include Initialize() and Update(), even if I don't use them. Feel free to remove it in this case if you want.

If they exist, the function Initialize() is automatically run by Rainmeter when the skin is loaded or refreshed, and Update() is automatically run on each update of the Script measure in the skin. They are not however required.
3) If I wanted to change the website to be parsed I assume the best approach would be to use !CommandMeasure to set the URL of [MeasureSite] and the refresh the whole skin?
Two approaches...

If you want the change to be temporary, perhaps driven by something you set using the InputText plugin or by clicking on a list of alternative URL's to use, then I would just use !SetOption to change the URL of the WebParser measure, followed by:

[!CommandMeasure MeasureSite "Update"]

I think that should work to have the measure go out and get the data from the new URL, without needing to refresh the skin. Don't forget that if you refresh the skin, any change you "dynamically" make to any options or variables will be reset to their staring value. Refreshing the skin is analogous to unloading and reloading it.

To make the change permanent / persistent, you would need to use !WriteKeyValue to physically write the new URL option on the parent WebParser measure, and then refresh the whole skin.
Your having basically done this one for me (Many thanks!!) I'm actually trying to write another skin that's going to do something very similar on a different feed. I'll see how I get on adapting this code to the new purpose, in light of this is there a way of testing .lua code, seeing what the variables are returning etc?
Glad to help. Let me know if anything else isn't clear. As to testing, about the best I can offer is that you can use a:

print(any value)

Statement anywhere in the Lua code, and it will output the value to the Rainmeter Log in About / Log. So you could for instance do:

destinationName = string.match(oneTrain, destinationPattern)
print(destinationName)

or

print('The value of the variable destinationName is: '..destinationName)

In order to check if you getting what you expect from that variable. I use that approach all the time. My example skin had print() statements all over the place at one time or another while I was writing / testing it, then I just remove them when I'm sure it all works.

The About / Log and About / Skins panels are your very best friend in Rainmeter in general, and Lua in particular.

And keep this handy...

https://docs.rainmeter.net/manual/lua-scripting/
Bishop149
Posts: 25
Joined: March 23rd, 2016, 3:07 pm

Re: Backwards WebParsing?

Post by Bishop149 »

Ok thanks, I think I get it.

Wow!
I just tested my new skin and it pretty much worked first time using a modified version of your code!
I think that's a first.
Its a skin to grab torrent info from qBittorrent's WebUI, which I figure might be handy for others so I'll post it up when I'm done with it (crediting jsmorley of course!)

The modifications I made were:
- A few more patterns to search for.
- Changed most of the varible names.
- I no longer cared about filtering by an equivalent of the "operator" so removed that IF term

I have discovered you can do mathematical transformations in the .lua for example one of the returns is in Bytes/s, so I convert to kB/s as follows:

Code: Select all

downloadSpeed = string.match(oneTorrent, speedPattern)/1000
How might I format that so it only returns up to 1 decimal place?
I have discovered that NumOfDecimals=1 on the meter doesn't work, I assume because it is just reading text not a numerical measure.
I could instead shoot the .lua output to the formula of a calc measure and then use rainmeter to do further calculations but I figured it would be more elegant if I could do it in the .lua?
User avatar
jsmorley
Developer
Posts: 22637
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Backwards WebParsing?

Post by jsmorley »

Add this function to your .lua file and call it with the value you want to "round" as the parameter.

https://docs.rainmeter.net/snippets/round/

I'm glad this came together for you. That makes any work I did to explain it all worthwhile.
User avatar
jsmorley
Developer
Posts: 22637
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Backwards WebParsing?

Post by jsmorley »

By the way, I think you might find this snippet useful for scaling bytes:

https://docs.rainmeter.net/snippets/autoscale/
Bishop149
Posts: 25
Joined: March 23rd, 2016, 3:07 pm

Re: Backwards WebParsing?

Post by Bishop149 »

Hopefully my final question on this.

Lets say that the following returns a number representing the download speed, is there anyway in the lua to append whatever that number is with a text string for example " Kb/s"

Code: Select all

downloadSpeed = string.match(oneTorrent, speedPattern)/1000
I've been hunting around a few wikis but haven't found much!
I though the following might do it but no:

Code: Select all

downloadSpeed = string.match(oneTorrent, speedPattern)/1000
downloadSpeedkbs = .. downloadSpeed" kb/s"
Edit: Ah ha! I had the syntax slightly wrong, the below works!

Code: Select all

downloadSpeed = string.match(oneTorrent, speedPattern)/1000
downloadSpeedkbs = downloadSpeed.." kb/s"
Bishop149
Posts: 25
Joined: March 23rd, 2016, 3:07 pm

Re: Backwards WebParsing?

Post by Bishop149 »

Well I though I had this, but it appears not.

I was doing all my testing on a local .html file designed to replicate exactly the format of the output from the webUI (my not actually having the real thing available to the test computer), it was working perfectly.

I got home and substituted the local file for the real url and nowt is working.
I'm pretty sure the original webparser term (below) just isn't grabbing anything.

Code: Select all

[MeasureFeed]
Measure=Plugin
Plugin=WebParser
URL=http://IPv4:Port/query/torrents?filter=downloading&sort=eta&limit=4
RegExp=(?siU)^(.*)$
StringIndex=1
UpdateRate=10
FinishAction=[!CommandMeasure MeasureScript "ParseFeed()"]
What exactly does RegExp mean?
I assumed it just meant "grab the lot"

Edit: Oh I dunno, I think the bloody thing wants a password, despite the fact I've specifically disabled authentication the WebUI options. I tried giving it the password via HTTP authentication and still doesn't work.
This may have been a waste of effort.
qBittorrent uses a cookie based authentication.
Oh crap, still doesn't explain why "Bypass authentication for local host" is doing sod all
User avatar
jsmorley
Developer
Posts: 22637
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Backwards WebParsing?

Post by jsmorley »

I'm quite sure it will want a password. In uTorrent (not the same thing I know) you can "enable" http authentication in the settings, but I don't believe you can disable the password entirely.

uTorrentURL=http://#UserName#:#Password#@127.0.0.1:#uTorrentPort#/gui/?list=1

Try literally using localhost in the URL, which is 127.0.0.1
User avatar
jsmorley
Developer
Posts: 22637
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Backwards WebParsing?

Post by jsmorley »

Something like:

URL=http://127.0.0.1:8080/query/torrents?filter=downloading&sort=eta&limit=4

Dunno what port you have it using, I just used 8080 as an example.
User avatar
jsmorley
Developer
Posts: 22637
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Backwards WebParsing?

Post by jsmorley »