Append to file when downloading with WebParser

[Yincognito]

Assuming that a file that you want to download from the internet already exists in the chosen download location, is there any way that you can append the new content on a subsequent download instead of recreating the file? The only way I can think of doing this in Rainmeter would be to have the file already made (and empty) in the INI format, then get the new downloaded content in one or two WebParser measures (parent + child), followed by a !WriteKeyValue in the WebParser's FinishAction option that writes the content in incrementally named keys in that INI formatted file. Then finally, to actually be able to use that file, you'd have to load it in yet another WebParser measure and design your RegExp option in order to strip off the undesired section and key "markers" (e.g. [SectionName] and KeyName=) from the content.

If another easier method doesn't exist, would the Rainmeter devs be willing to add an option in the WebParser measure (say, AppendFile=1 to do it "in plain sight", or even Debug=3 to do it in background and append the downloaded webpage to WebParserDump.txt) to allow this capability? I don't know about the amount of usage scenarios (obviously it would help some skin designers out there, albeit probably not that many), but certainly it wouldn't be that big of a deal to implement it, especially if doing it at the end of WebParserDump.txt...

[jsmorley]

Also unlikely to happen... The fundamental issue I have with this is that there is no way to distinguish between two separate WebParser parents, where you want to append the entire output of RegExp=(?siU)^(.*)$ to the output of a previous parent measure in the skin, and a second, third, and so on iteration of the same measure on subsequent UpdateRate updates. That will mean you will soon have hundreds and eventually thousands of copies of the same information in a gigantic and rapidly increasing WebParserDump.txt file.

I'd be tempted to just download the entire site(s) into separately named files, and deal with it from there.

Code: Select all

[MeasureSite1]
Measure=WebParser
URL=https://www.rainmeter.net
Download=1
DownloadFile=Site1.txt

[jsmorley]

If you really want them all appended into one file to parse, I suppose you might use a FinishAction on the "last" parent measure, one that uses RunCommand to do something like "copy Site1.txt+Site2.txt+Site3.txt AllSites.txt" to copy/append all the files into a new combined file. Then have a FinishAction on the RunCommand measure fire off another WebParser parent that parses that.

You could also have a Lua script that does the concatenation, and in my view, also does the parsing. Not sure there is any need to create a physical file of all the concatenated data, just read them all in with Lua and parse away using pattern matching. I'd put the entire result into some form of "table" in Lua, and use Inline Lua in the various meters to pluck off and use the desired bits of data.

[Yincognito]

That will mean you will soon have hundreds and eventually thousands of copies of the same information in a gigantic and rapidly increasing WebParserDump.txt file.

Oh, I see. So, if you have, say, 3 WebParser measures and you'd want only one to dump the content, you'd end up with all 3 doing the same, is that what you're saying? If so, does that mean that the Debug option is a global setting, and not one specific to a certain WebParser measure? This is a little bit confusing, because, in the end, the skin designer adds that option to the desired WebParser measures, so he would expect that only those measures dump their content, not all of them...

I'd be tempted to just download the entire site(s) into separately named files, and deal with it from there.

Yeah, it crossed my mind too, but since I'm "moving" between all the captures from the content of all the files, that would involve unnecessary disk activity when "jumping" from a file to another that could be avoided. This is partly the reason why I would favor an "all in one" approach in this. Anyway, it's a valid solution, so it was noted.

If you really want them all appended into one file to parse, I suppose you might use a FinishAction on the "last" parent measure, one that uses RunCommand to do something like "copy Site1.txt+Site2.txt+Site3.txt AllSites.txt" to copy/append all the files into a new combined file. Then have a FinishAction on the RunCommand measure fire off another WebParser parent that parses that.

Yup, now that's a very very good idea - thanks for this! I don't know how I could miss this option, as it pretty much solves the issue.

You could also have a Lua script that does the concatenation, and in my view, also does the parsing.

I like programming, but in the case of Rainmeter, I try to avoid Lua scripts at any cost, LOL. It kind of defeats the purpose of Rainmeter if you have to do things in Lua, because it is a bit redundant to have to code in 2 systems when you could just code the whole thing in one system right from the start.

[jsmorley]

Oh, I see. So, if you have, say, 3 WebParser measures and you'd want only one to dump the content, you'd end up with all 3 doing the same, is that what you're saying? If so, does that mean that the Debug option is a global setting, and not one specific to a certain WebParser measure? This is a little bit confusing, because, in the end, the skin designer adds that option to the desired WebParser measures, so he would expect that only those measures dump their content, not all of them...

It's not global, only for the measure where Debug=2 is set, but it is recreated each time the parent measure is updated, when the UpdateRate counter is reached, and if we changed it to append to it, it would append each and every time that parent measure is updated, and would soon be gigantic. We are just never going to support that.

[jsmorley]

I like programming, but in the case of Rainmeter, I try to avoid Lua scripts at any cost, LOL. It kind of defeats the purpose of Rainmeter if you have to do things in Lua, because it is a bit redundant to have to code in 2 systems when you could just code the whole thing in one system right from the start.

That's up to you, although I disagree of course. Lua is well integrated in Rainmeter, and has capabilities that are difficult if not impossible to do in skin code. But, to each his own...

[Yincognito]

It's not global, only for the measure where Debug=2 is set, but it is recreated each time the parent measure is updated, when the UpdateRate counter is reached, and if we changed it to append to it, it would append each and every time that parent measure is updated, and would soon be gigantic. We are just never going to support that.

Yeah, but it would be done willingly, it's not like it would happen behind the user's back - he would still have to set Debug=3 (which would apend to file), in my envisioned scenario, to differentiate it from Debug=2 (which recreates the file). Then, when the user wants to go back to the "normal" way of dumping content, he would only need to set things back to Debug=2 using a !SetOption or modifying the skin code, and puff, the gigantic stuff is there no more. Anyway, this was just an idea - an idea that would work, IMHO. The fact that you're reluctant to implement it, it's a different story, and I have nothing against that - after all, it's you who are the developers...

Since we're at it and we're having this conversation, I think that either the word </content> or the same word </content> in a snippet, or the same word

Code: Select all

</content>

in a code breaks the parsing of the Rainmeter feed when using the (?siU).*<content.*>(.*)</content>.* RegExp to parse the content of a feed entry (because CDATA doesn't encode certain < and > in the XML) - at least this was my impression last time I checked. I'll check back now after posting this and see how the XML source looks like, so disregard this if I'm mistaken.

[jsmorley]

Yeah, not sure I follow....

Code: Select all

		<content type="html" xml:base="https://forum.rainmeter.net/viewtopic.php?t=31611&amp;p=159810#p159810"><![CDATA[
<blockquote class="uncited"><div>It's not global, only for the measure where Debug=2 is set, but it is recreated each time the parent measure is updated, when the UpdateRate counter is reached, and if we changed it to append to it, it would append each and every time that parent measure is updated, and would soon be gigantic. We are just never going to support that.</div></blockquote>Yeah, but it would be done willingly, it's not like it would happen behind the user's back - he would still have to set <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">Debug=3</span> (which would apend to file), in my envisioned scenario, to differentiate it from <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">Debug=2</span> (which recreates the file). Then, when the user wants to go back to the "normal" way of dumping content, he would only need to set things back to <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">Debug=2</span> using a <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">!SetOption</span> or modifying the skin code, and puff, the gigantic stuff is there no more. Anyway, this was just an idea - an idea that would work, IMHO. The fact that you're reluctant to implement it, it's a different story, and I have nothing against that - after all, it's you who are the developers...  <img class="smilies" src="https://forum.rainmeter.net/images/smilies/innuendo.gif" width="20" height="20" alt=";-)" title="Wink"> <br><br>Since we're at it and we're having this conversation, I think that either the word &lt;/content&gt; or the same word <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">&lt;/content&gt;</span> in a snippet, or the same word <div class="codebox"><p>CODE: </p><pre><code>&lt;/content&gt;</code></pre></div> in a code breaks the parsing of the Rainmeter feed when using the <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">(?siU).*&lt;content.*&gt;(.*)&lt;/content&gt;.*</span> RegExp to parse the content of a feed entry (because CDATA doesn't encode certain <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">&lt;</span> and <span style="font-family:Consolas,'Courier New',Menlo,Monaco,monospace;font-size:1em;font-style:normal;line-height:1.3em;background:#fff;border:1px solid #869aa8;padding:0 3px;color:#003807;display:inline">&gt;</span> in the XML) - at least this was my impression last time I checked. I'll check back now after posting this and see how the XML source looks like, so disregard this if I'm mistaken.<p>Statistics: Posted by <a href="https://forum.rainmeter.net/memberlist.php?mode=viewprofile&amp;u=33362">Yincognito</a> — 3 minutes ago</p><hr />
]]></content>

[Yincognito]

Yeah, not sure I follow....(just for testing in a quote:<content></content>)

Yeah, I know, I've checked it out too and it's ok. It happened to me when I was getting the feed entry from this post, and my custom tag renamer (that was renaming <description> to <d> at that time) was getting the incomplete content. Unfortunately, I can't have that in the feed anymore, unless I post it again and spam you with such, and, well, maybe I was wrong, after all...

Anyway, if it happens again, I'll let you know.

EDIT: I figured out why that happened: I was parsing again the string after previously decoding it. I know, the decoding should be the last operation to be done and no parsing after that should happen, but I was "saving" the content in the aggregator after decoding it, in order to reduce the size of the string. Will have to rethink that approach...