It is currently March 28th, 2024, 9:25 pm

Maximum length of a string in a measure

Get help with creating, editing & fixing problems with skins
User avatar
Yincognito
Rainmeter Sage
Posts: 7027
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Maximum length of a string in a measure

Post by Yincognito »

What is the maximum length of a string in a Rainmeter measure? Is there any limit to it, or it is only limited by the memory available (or something like that)? Also, how much a very long String measure impacts a skin performance (CPU and memory usage)?

I'm asking this because I want to build some kind of "news aggregator" ticker skin (that I can "query" on, like the defunct Yahoo Query Language did with a variable number of feeds from a variable number of sites), and I would like to know what would be the best solution to "store" the aggregated XML data from the news sites I take the information from:
  • getting each site data in a WebParser measure, then add the contents of that measure to an "aggregator" String measure which I could interrogate further OR
  • getting each site data in a WebParser measure by downloading the data to files, then "load" the needed file in another WebParser measure to further "filter" its content and present a particular feed "item" from a particular site to the user
Obviously, the first option would be preferable from a skin designer point of view, since all feeds would be stored in the same place (the String measure mentioned above) and getting a particular item from a particular site would be just a matter of changing two indexes in a regex (the index of the site and the index of the item) - thus my question. I could work similarly with the second option as well, but I would have to replace the index of the site with the index of the file I take the needed data from, and this would require working with the slower (compared to RAM) hard disk.

NOTE: No, I don't want to hardcode N*M measures for M items from N sites. My skin has to be adaptable to whatever number of sites the user wants to get feeds from, and modifying the skin each time he adds or removes a site from his list is not what I want to achieve.
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Maximum length of a string in a measure

Post by jsmorley »

As far as I know, there is no limit to the size of a string in Rainmeter. If there is one, it in the range of 4,294,967,291 bytes, or about 4gig. It's just a question of memory, although strings are not huge memory hogs in any case. If you are trying to load 4gig of text into a single string, I just can't go there with you. I can't imagine that there is any particular impact on CPU either, other than the work required to load the string into memory. I guess that could be significant as it is loading, again at some ridiculously large amount. Presumably you are not displaying the entire string in a single meter at once.

I doubt there would be any appreciable impact on memory or CPU until you got to a fairly ridiculous size of a single string, something in the megabytes of text.
User avatar
Yincognito
Rainmeter Sage
Posts: 7027
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: Maximum length of a string in a measure

Post by Yincognito »

jsmorley wrote: February 25th, 2019, 5:19 pm As far as I know, there is no limit to the size of a string in Rainmeter. If there is one, it in the range of 4,294,967,291 bytes, or about 4gig. It's just a question of memory, although strings are not huge memory hogs in any case. If you are trying to load 4gig of text into a single string, I just can't go there with you. I can't imagine that there is any particular impact on CPU either, other than the work required to load the string into memory. I guess that could be significant as it is loading, again at some ridiculously large amount. Presumably you are not displaying the entire string in a single meter at once.

I doubt there would be any appreciable impact on memory or CPU until you got to a fairly ridiculous size of a single string, something in the megabytes of text.
Great info, jsmorley! No, the string would be "loaded" in a String measure, not meter. The meter will actually hold only one item at a time (since having a long text sliding in a skin increases the CPU usage proportionally with the length of the text in pixels). The item loaded in the meter will be replaced with the next one (by updating the site/item indexes in the regex) when its text finished sliding in the skin.
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Maximum length of a string in a measure

Post by jsmorley »

I would caution against pushing this too far, as although the limit of a string in C++ is pretty big, there is a point, somewhere in the low 10's of megabytes, where regular expression will fail. Understand however, that 10meg of text is a BOATLOAD of text. The entire US Constitution, which is 17,958 lines, is 18Kb of text. The entire King James Bible is only about 4MB.

If you are trying to store more than 1MB of text in a single string and parsing it, I think its time to rethink the approach.
User avatar
Yincognito
Rainmeter Sage
Posts: 7027
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: Maximum length of a string in a measure

Post by Yincognito »

jsmorley wrote: February 25th, 2019, 5:50 pm I would caution against pushing this too far, as although the limit of a string in C++ is pretty big, there is a point, somewhere in the low 10's of megabytes, where regular expression will fail. Understand however, that 10meg of text is a BOATLOAD of text. The entire US Constitution, which is 17,958 lines, is 18Kb of text.
Ouch! Good to know. Yes, the content passed to the String measure will have to be significantly reduced. Fortunately, that's easy to do, since a lot of that content will not be needed (only the items' title, description, link and date, along with the site's title, description, link, date and probably the image url as well) will be needed. The rest will have to be stripped off in the WebParser measure before being added to the String measure.

That would help to allow more sites and items to be available to the user, in the end. If I were to store the entire content of the XMLs, I would have reached 10 MB really fast, as a regular CNN feed is about 100 000 bytes without removing the bloatware from it. The character reference decoding would probably help as well in reducing the content.

Of course, in practice I'll have to see how all this impacts the performance, but in theory, this should be doable without a noticeable negative impact.
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
Yincognito
Rainmeter Sage
Posts: 7027
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: Maximum length of a string in a measure

Post by Yincognito »

jsmorley wrote: February 25th, 2019, 5:50 pmIf you are trying to store more than 1MB of text in a single string and parsing it, I think its time to rethink the approach.
...but you said the low 10's of MB would be ok. Or you were just referring to regex when writing that? This doesn't sound so good... :(
Fortunately there's the downloading alternative, although, like I said, I would prefer not to use that because of speed considerations between RAM and the disk.
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Maximum length of a string in a measure

Post by jsmorley »

Yincognito wrote: February 25th, 2019, 6:29 pm ... a regular CNN feed is about 100 000 bytes without removing the bloatware from it.
I sorta resist that. Do you have a link to the feed?
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Maximum length of a string in a measure

Post by jsmorley »

jsmorley wrote: February 25th, 2019, 6:35 pm I sorta resist that. Do you have a link to the feed?
Oh, sorry, I misread that. Well, 100,000 bytes is about 97.5Kb, so I don't see that as a big deal. You would need to stack 10 of those up to reach 1MB.
User avatar
Yincognito
Rainmeter Sage
Posts: 7027
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: Maximum length of a string in a measure

Post by Yincognito »

jsmorley wrote: February 25th, 2019, 6:39 pm Oh, sorry, I misread that. Well, 100,000 bytes is about 97.5Kb, so I don't see that as a big deal. You would need to stack 10 of those up to reach 1MB.
Yeah, 100,000 bytes without removing the bloatware. Once that is reduced to the fields needed for a normal feed skin, I think the size would be greatly diminished. I'm looking for at least 50 feed sites data to be available to work with in my skin (although personally I only use 5), but, as I said, once the unneeded content is removed (I could even replace the tags with shorter ones, say <D> for <description>, etc.), this may be doable without adverse effect.
Profiles: Rainmeter ProfileDeviantArt ProfileSuites: MYiniMeterSkins: Earth
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: Maximum length of a string in a measure

Post by jsmorley »

You are going to have a skin that goes out to 50 separate sites on the internet and downloads all of them? I can't even begin to tell you how much I hate this idea... ;-)