It is currently March 28th, 2024, 7:36 pm

Use Lua to extract info from webpage

Discuss the use of Lua in Script measures.
Krishneel Chand
Posts: 16
Joined: March 30th, 2018, 12:33 am

Use Lua to extract info from webpage

Post by Krishneel Chand »

hi, I have a rainmeter skin that uses RegExp to get info from a website. How can I load the page once and then extract multiple info from it using lua?
the website is below
https://www.spaceflightinsider.com/launch-schedule/

From this page, I interested in "Next Launch" data

Code: Select all

<span>Next Launch&colon; <a href="https://www.spaceflightinsider.com/launch-schedule#launch-374">CRS-17</a></span>
				</div>
				<div class="cd_info">
					<table>
						<tr>
							<th>Mission</th>
							<td class="mission"><a href="https://www.spaceflightinsider.com/launch-schedule#launch-374">CRS-17</a></td>
						</tr><tr>
							<th>Vehicle</th>
							<td class="vehicle">Falcon 9 v1.2</td>
						</tr>
						<noscript>
						<tr>
							<th>Time</th>
							<td>26 Apr 05:55:00 EDT</td>
						</tr>
Some of the info I'm interested in;
Mission: CRS-17
Vehicle: Falcon 9 v1.2
Time: 26 Apr 05:55:00 EDT


For that, I'm currently using multiple RegExp (loads page many times) but I think we can Lua to load the page once then extract info from that as many times as we want.

Can someone please provide an example?
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Use Lua to extract info from webpage

Post by jsmorley »

I'm not sure why you would want or need multiple WebParser parent measures for this, and I don't see any real need to leap to Lua either.

Code: Select all

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1

[MeasureSite]
Measure=WebParser
URL=https://www.spaceflightinsider.com/launch-schedule/
RegExp=(?siU)Next Launch&colon;.*<td class="mission"><a href.*>(.*)</a.*<td class="vehicle">(.*)</td>.*<th>Time</th>.*<td>(.*)</td>

[MeasureNextMissionName]
Measure=WebParser
URL=[MeasureSite]
StringIndex=1

[MeasureNextMissionVehicle]
Measure=WebParser
URL=[MeasureSite]
StringIndex=2

[MeasureNextMissionDateTime]
Measure=WebParser
URL=[MeasureSite]
StringIndex=3

[MeterNextMissionName]
Meter=String
MeasureName=MeasureNextMissionName
FontSize=11
FontWeight=400
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
AntiAlias=1

[MeterNextMissionVehicle]
Meter=String
MeasureName=MeasureNextMissionVehicle
Y=R
FontSize=11
FontWeight=400
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
AntiAlias=1

[MeterNextMissionDateTime]
Meter=String
MeasureName=MeasureNextMissionDateTime
Y=R
FontSize=11
FontWeight=400
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
AntiAlias=1

1.png

I think that depending on the ability to clearly delineate in the HTML the various bits you want to retrieve, you should be able to use one parent measure, with an appropriately designed RegExp, to get as many child measures as you need.
You do not have the required permissions to view the files attached to this post.
Krishneel Chand
Posts: 16
Joined: March 30th, 2018, 12:33 am

Re: Use Lua to extract info from webpage

Post by Krishneel Chand »

The thing is that their code keeps changing so if one thing changes it leads to an error for everyhting. So I didn't have an option other than to create a RegExp for each item so that one error doesn't affect the others

I have 7 RegExp measures which work in all situations. If one item is missing it just shows "N/A"
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Use Lua to extract info from webpage

Post by jsmorley »

Then you might look at this:

Code: Select all

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1

[MeasureSite]
Measure=WebParser
URL=https://www.spaceflightinsider.com/launch-schedule/
RegExp=(?siU)^(.*)$
FinishAction=[!EnableMeasureGroup NextMission]

[MeasureNextMission]
Measure=WebParser
Group=NextMission
URL=[MeasureSite]
RegExp=(?siU)Next Launch&colon;.*<table>(.*)</table>
Disabled=1

[MeasureNextMissionName]
Measure=WebParser
Group=NextMission
URL=[MeasureNextMission]
RegExp=(?siU)<td class="mission"><a href.*>(.*)</a>
StringIndex=1
StringIndex2=1

[MeasureNextMissionVehicle]
Measure=WebParser
Group=NextMission
URL=[MeasureNextMission]
RegExp=(?siU)<td class="vehicle">(.*)</td>
StringIndex=1
StringIndex2=1
Disabled=1

[MeasureNextMissionDateTime]
Measure=WebParser
Group=NextMission
URL=[MeasureNextMission]
RegExp=(?siU)<th>Time</th>.*<td>(.*)</td>
StringIndex=1
StringIndex2=1
Disabled=1

[MeterNextMissionName]
Meter=String
MeasureName=MeasureNextMissionName
FontSize=11
FontWeight=400
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
AntiAlias=1

[MeterNextMissionVehicle]
Meter=String
MeasureName=MeasureNextMissionVehicle
Y=R
FontSize=11
FontWeight=400
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
AntiAlias=1

[MeterNextMissionDateTime]
Meter=String
MeasureName=MeasureNextMissionDateTime
Y=R
FontSize=11
FontWeight=400
FontColor=255,255,255,255
SolidColor=47,47,47,255
Padding=5,5,5,5
AntiAlias=1
So the goal is to get the entire site into one "super parent" , that you then create a "parent" for the individual "chunk" of the code that contains the information you want. Then you create parent/child measures, using StringIndex=1 to get the result of parent, (the "child bit") and using a RegExp and StringIndex2=1 to extract the specific bit you want (the "parent" bit).

So in a sense [MeasureNextMissionName] is a "child / parent / child". StringIndex makes is a "child", RegExp makes it a "parent" and StringIndex2 makes it a "child" (of itself).

The long and the short of this is that if the "chunk" for "Next launch" fails, other "chunks" you might get with this same approach will not.

There are several ways to approach this, but the overarching goal should be to only hit the site itself one time. The idea here is much the same as your 7 separate "parent" measures, but having only one of the 7 actually trudge out to the site and download the information. All the others just use the information that first, "super parent" measures gets.
Krishneel Chand
Posts: 16
Joined: March 30th, 2018, 12:33 am

Re: Use Lua to extract info from webpage

Post by Krishneel Chand »

awesome! thanks ;-) :welcome: