Run WebParser only once?

Nookz · Post by **Nookz** » November 25th, 2023, 2:28 am

Heya!, I come to you with yet another issue ^^'.

I've read the documentation but I am still confused as to what can and can't be done with WebParser..
I'm not sure I know how to explain this but I'll do my best (.rmskin at the bottom)
My setup:
I have "ID"s variables inside the Variables.inc file (1 to 5)
I have "OldID"s variables inside the ini file (1 to 5)
I have "Name" variables inside the ini file (1 to 5)
one "IDA" variable inside the ini file set to 1 by defaut.

What I'm trying to do:
When the skin is loaded or refreshed:
-a Loop measure start at 1 with UpdateDivider=-1, Increment at 1 and OnChangeAction set to update the following measure:
-a other measure set to UpdateDivider=-1, checks to see if [#OldID[#IDA]] and [#ID[#IDA]] are identical. If yes, Increase #IDA# by one and update the Loop. If not, enable all the WebParser measures and set [#OldID[#IDA]] to [#ID[#IDA]].
On the same measure I do a second check, If #IDA#=5, Disable the loop measure.

-In the case that the IDs were different, I fetch the name and the Image Banner

you probably notice already how wrong this is looking..

-Then I do a final check to update meters and update the loop or disable everything if #IDA# is above 4.

The goal would be to have the parsing happen only once for each ID that differ from the last time and never after that.
I'm aware that It would probably be a lot easier to just have a measure for each, but I would really like to make this work since It looks cleaner, it's more probably also efficient and I feel like this could help me understand the WebParser a bit better too.

I've tried a bunch of different things and nothing seems to work..
I'm also well aware that there probably is a ton of things wrong with my current code.. like I said.. I tried a bunch of things and I think it only got messier and messier over time, but:
I know the Measures are parsing the right things since:
-I can see that it download the images properly, but it doesn't update them in the skin.
-I can see the name updating with a debug meter, but it always only set the first Name. The best I managed is having the second name it fetche be written as the first one.

More things I don't really understand (with the current version), but suspect are related to the enabling/disabling of the WebParser either not working like I think or not working at all..:
-If the IDs are the same, the first name get removed and becomes blank.
-If the IDs are the same the loop stays at 0, but the #IDA# increases to 2..

Updated:

Steam_.rmskin

Nookz · Post by **Nookz** » November 25th, 2023, 5:02 am

K so I worked on it a bit more and I think I managed to 'stabilize' the mess I had created..

I added delays since it would change the IDA variable faster than the parsing was able to happen..
Now it properly finds and update all the image and it cycles through all the IDs like it should ( altho I suspect it still does not disable the webparser measures when it's done..)
BUT, It doesn't update the names anymore..

I'm still lost, but making progress x.x

Nookz · Post by **Nookz** » November 25th, 2023, 6:32 pm

Ok I definitely do not understand what's going on. I haven't change the code at all but the name fetching started working. I had to increase the delay again for the name to update in time..

So now I got everything working as I wanted (altho a bit bummed about the Delays, but I can live with that).. except the part that I initially asked about: is it possible to disable the webparser at all so that it stop everything until next refresh/load? Or do I just crank the UpdateRate to an absurd number?

new problem: some Names are giving a matching error (-1)
This doesn't:
STAR WARS Jedi: Survivor™
The Wild Eight

But this does:
Diablo® IV
??

Post by **Yincognito** » November 26th, 2023, 4:24 am

OK, so I've been busy the last couple of days with another project, but just by looking at your earlier codes, there are a bunch of improper things there (infinite loops, the loops not related to the WebParsers, etc). I'll probably be busy another couple of days, but just so you know, it would be a lot simpler if:
- you'd describe your practical / real goal, not the code itself, since we can read the code too
- you'd simplify the code parts you have trouble with and put them in a separate (and valid skin) sample that you'd post here; once they're fixed, they can be pasted back to your more complex environment, if any
- you'd try to simplify (at least initially) your WebParser process, e.g. make 5 of them, one for each id or such if you need concurrent access (all at once), or let us know if you really need sequential access (one by one)

There would be more to say, but it's late now so maybe tomorrow you can have a clear and compact description of what you'd like to achieve in reality, so we can help better.

Nookz · Post by **Nookz** » November 26th, 2023, 6:26 am

So in the meantime I managed to mostly accomplish what I was trying to do.
which was:
1.Based on a manually inputted Steam gameID (inside Variables.inc), parse the steam website for both the banner and the name of the game. (This now works, except for some*(*this actually only happens with this one game I tested so far) Game name that are giving back regex errors. I can't figure out why that is. the structure is the same on every page. At first I was thinking it was probably special characters in the name, but that doesn't seem to be the issue since other games with the same special character in their name parse just fine.. So I'm still at a loss on that front)
2.Be able to have this behaviour be repeated over again for each ID present in Variables.inc AND with only one set of measures. meaning I'm trying to avoid having a parsing measure for each and every game. (This also work now, as far as I know)
3.both #1 and #2 should only happen when the skin loads/refresh and also only happen if the IDs of Variables.inc differ from the OldID in the .ini file. Otherwise, every measure should be disabled.(Once again, this works)

what follows was not part of my original post and might actually be complicating everything else since you seem to be saying the entire code was horribly wrong.

(if you only looked at the first code I posted it makes sense tho.

) , but

What I've been trying to do since is:
-use a lua script and use that parsed name to find the closest matching folder name inside the steam games location. (This is also working)
From there I'm trying to once again find the closest match to the name I parsed from the steam website, but this time I'm looking inside said folder and subfolders and I'm looking for an .exe. (This sorta works. at first I was only looking inside the folder itself with lua and it worked great. But I realised the .exe is not always going to be at the root level.. so I tried to do the recursive search in lua but I failed miserably so I tried using a combination of lua and the FileView Plugin instead. Now I think I have something that should be working on the lua side of things, but I have yet again hit a wall since I don't know how to or if I can even use a nested variable for the Path of the FileView parent measure.) This behaviour should also loop for each ID, and also use as little measures as possible so that it can be scale more easily, and it should also only happen whenever the IDs and OldIDs do not match ( this part is simple enough and as previously mentioned, already works)

Then obviously, the ultimate goal is to have a layout of steam game banners that you can click on to launch and that can be switched with minimal efforts. (basically you change the ID and refresh the skin and it does the work for you. No need to find the executable path or find a proper banner or anything like that) I realise this is a bit niche of a use and that going the 'just add more measures' route would be a hundred times easier, but it is mostly an exercise to help me understand Rainmeter and lua better at this point.

I know.. I'm asking way too many questions x.x

Post by **Yincognito** » November 26th, 2023, 1:14 pm

Nookz wrote: ↑November 26th, 2023, 6:26 am So in the meantime I [...]
I know.. I'm asking way too many questions x.x

No no, you're not asking too many questions, and yes, I was referring mostly to your first code being all over the place (this is why I said "earlier codes"), but yeah, your 1. 2. 3. description above is much clearer now. The reason why this (the real goal) is preferable to be described when asking for help is because often there are many (and sometimes simpler or more straightforward) ways to achieve it, and when one describes his current code process, he's automatically inducing the same approach for the one trying to help, which might not always be the most effective or even easier to understand, that's all.

Can you post the regex you're having trouble with and either a link to the page or the literal page part that you're interrogating so one can test, reproduce and see what can be done to work for all game titles? Also, are you sure the circumstances are exactly the same for both the working titles and the not working ones? I'm talking about the page itself, the pattern and the process used in the skin to interrogate things...

Nookz · Post by **Nookz** » November 26th, 2023, 5:42 pm

Yincognito wrote: ↑November 26th, 2023, 1:14 pm Can you post the regex you're having trouble with and either a link to the page or the literal page part that you're interrogating so one can test, reproduce and see what can be done to work for all game titles?

Of course. This is the Regex I'm using:

Code: Select all

[GetGameName]
Measure=WebParser
URL=https://store.steampowered.com/app/[#ID[#IDA]]/
RegExp=(?siU)<span itemprop="name">(.*)</span>.*
DynamicVariables=1

and this is the title I've been having trouble with:
https://store.steampowered.com/app/2344520
The ID here being: 2344520

Using the same regex with any of those works perfectly fine:
632360
526160
421020
316010
Also works perfectly for this title who also use special characters in their name
1172380
And also for this one who uses the same exact special character
262060

Yincognito wrote: ↑November 26th, 2023, 1:14 pm Also, are you sure the circumstances are exactly the same for both the working titles and the not working ones? I'm talking about the page itself, the pattern and the process used in the skin to interrogate things...

-The pages themselves are always structured the same way and the Name is always located between <span itemprop="name"> and </span>. Even in the example above that is throwing an error.
-On the skin side, everything from retrieving the ID, parsing the page, retrieving and using/displaying the information is done in the exact same way with the exact same measures for all of the IDs
So I think I can confidently say that yes, I'm sure.

Post by **Yincognito** » November 26th, 2023, 8:32 pm

Nookz wrote: ↑November 26th, 2023, 5:42 pmSo I think I can confidently say that yes, I'm sure.

Well, you're wrong.

For ID=2344520, the site redirects the user (me at least, who doesn't have an account and such) to an age check step (probably before going to the actual page), which has a different address: https://store.steampowered.com/agecheck/app/2344520
On that version (?) of the page, the pattern you're looking for does not exist. So, this has nothing to do with regex or Rainmeter, the circumstances, as I suspected, are different. The solution is to take this possibility into account in your regex pattern. If you're curious, there are 7 instances of Diablo on that page (can't color them since I posted this into code tags, but you get the idea):

Code: Select all

...
<head>
	<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
			<meta name="viewport" content="width=device-width,initial-scale=1">
		<meta name="theme-color" content="#171a21">
		<title>Save 40% on Diablo® IV on Steam</title>
	<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
...
						<meta name="twitter:card" content="summary_large_image">
					<meta name="Description" content="Join the fight for Sanctuary in Diablo IV, the ultimate action RPG adventure. Experience the critically acclaimed campaign and new seasonal content.">
			
	<meta name="twitter:site" content="@steam" />

						<meta property="og:title" content="Save 40% on Diablo® IV on Steam">
					<meta property="twitter:title" content="Save 40% on Diablo® IV on Steam">
					<meta property="og:type" content="website">
					<meta property="fb:app_id" content="105386699540688">
					<meta property="og:site" content="Steam">
					<meta property="og:url" content="https://store.steampowered.com/app/2344520/">
					<meta property="og:description" content="Join the fight for Sanctuary in Diablo IV, the ultimate action RPG adventure. Experience the critically acclaimed campaign and new seasonal content.">
					<meta property="twitter:description" content="Join the fight for Sanctuary in Diablo IV, the ultimate action RPG adventure. Experience the critically acclaimed campaign and new seasonal content.">
			
			<link rel="canonical" href="https://store.steampowered.com/app/2344520/">
...
				<script type="text/javascript">

		function CheckAgeGateSubmit( callbackFunc )
		{
						if ( $J('#ageYear').val() == 2023 )
			{
				ShowAlertDialog( '', 'Please enter a valid date' );
				return false;
			}

			$J.post(
				'https://store.steampowered.com/agecheckset/' + "app" + '/2344520/',
				{
					sessionid: g_sessionID,
					ageDay: $J('#ageDay').val(),
					ageMonth: $J('#ageMonth').val(),
					ageYear: $J('#ageYear').val()
				}
			).done( function( response ) {
				switch ( response.success )
				{
					case 1:
						callbackFunc();
						break;

					case 24:
					case 15:
					{
						var dialog = ShowAlertDialog( 'Error', 'Sorry, but you\'re not permitted to view these materials at this time.' );
						dialog.done( function ()
						{
							top.location.href = 'https://store.steampowered.com/';
						} );
					}
					break;

					case 2:
						ShowAlertDialog( 'Error', 'There was a problem verifying your age.  Please try again later.' );
						break;
				}
			} );
					}

		function HideAgeGate( )
		{
			var bHideAll = false;
			console.log(bHideAll);
			var strCookiePath = bHideAll ? '/' : "\/app\/2344520";
			V_SetCookie( 'wants_mature_content', 1, 365, strCookiePath );
			
			document.location = "https:\/\/store.steampowered.com\/app\/2344520\/Diablo_IV\/?snr=1_direct-navigation__";
		}

		function ViewProductPage()
		{
			CheckAgeGateSubmit( HideAgeGate );
		}
...

And yeah, this sometimes happens on other sites too. You have to be aware of these things when designing a regex pattern, to make it as flexible as possible while not sacrificing precision.

P.S. Also, you should escape the slashes aka / or other reserved characters in regex by preceding them with a backslash aka \ so it would be something like: RegExp=(?siU)<span itemprop="name">(.*)<\/span>.*. You can even see how they do it themselves in the small Javascript code part I posted above...

Nookz · Post by **Nookz** » November 27th, 2023, 2:33 am

ahhhh! It makes total sense now.
I just have to figure out a way to account for this agecheck page now. (Managed to get everything else related to this skin to work as I wanted ^^)
Thks for the clarifications. extremely useful, as always!

Post by **Yincognito** » November 27th, 2023, 11:18 am

Nookz wrote: ↑November 27th, 2023, 2:33 am ahhhh! It makes total sense now.
I just have to figure out a way to account for this agecheck page now. (Managed to get everything else related to this skin to work as I wanted ^^)
Thks for the clarifications. extremely useful, as always!

Well, a good way to account for that would be to grab the title from a part of the page that's common to both the normal page and the age check one, if there is any. Of course, if you need more than just the game title, things complicate a bit, but then you can always use lookahead assertions in regex (check the WebParser related pages, tips and tricks in the manual to learn about them) to extract the data you're interested in only if it exists on the page. It won't be that easy if you need a ton of stuff from there and you want the lookaheads in the same regex pattern, but for just a few it isn't that hard. Or, you could only use a lookahead for the game title, and then either automatically let the other fields' measures silently fail with no return, or manually update the other fields' measures yourself only if the game title measure got something other than the empty string (IfMatch would help here).

EDIT: Just remembered that you only need the icon for the game. If I recall correctly, such an icon exists on the age check page too, it's just that it probably has a different location in that webpage source, compared to the normal one...

Run WebParser only once?

Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?

Re: Run WebParser only once?