It is currently March 28th, 2024, 9:34 am

BUG: Excessive disk IO

Report bugs with the Rainmeter application and suggest features.
CollinChaffin
Posts: 2
Joined: May 7th, 2017, 6:34 am

BUG: Excessive disk IO

Post by CollinChaffin »

Hi, I have taken a short screencap to demonstrate what I would consider a bug. You very well may tell me it is not as I would be surprised if somehow you were not aware so unless you are unaware I would then request it (and won't beleve I'd be calling it this) as a feature enhancement.

Env:
Rainmeter v4.0 x64 on Windows 7 x64 Ultimate.

Issue:
Rainmeter continually re-reads from disk every second (or less) every skin resource file when an animation is active as part of that skin ini descriptor.

Demonstration of issue:
https://youtu.be/oSvAsNefIBc

Notes:
As I put in video comments, Procmon used and filtered on disk activity. As shown, Rainmeter continually in sub-second loops, returning to disk (in this case luckily SSD) to fully re-read the entire resource(s). If it is 100% read, then no negative impact to SSD.

However, this is definitely one case where, depending on the efficiency of the OS' disk caching, could really put IMO absurdly unneeded amount of mechanical disk reads (I'm clocking GIGABYTES per day), when, given today's RAM standard desktop sizes, my ENTIRE Rainmeter 2-monitor multi-skin config with all BMP resources, could easily be cached in RAM for that exact same looping to occur.

In addition to virtually eliminating all mechanical disk (or other) IO, this would obvioulsy also have the side-effect of significantly enhancing performance greatly reducing the overhead in loop timing reductions.

As a side note (and a bit scary to think about), I have had several mechanical HD's fail on systems where Rainmeter sat in the tray running 24/7 for YEARS, with me never having any idea it sat literally in a loop every second moving the drive heads, once loaded just to perform a basic animation of a couple jpg's or png's files. Either way, it just seems very inefficient at the very least.

Hope this helps. If there is anything more I can do to further debug/test just let me know!
User avatar
CyberTheWorm
Posts: 860
Joined: August 22nd, 2016, 11:32 pm
Location: Surrey, B.C., Canada

Re: BUG: Excessive disk IO

Post by CyberTheWorm »

Does not look like a bug, it looks like the skin is doing it.
User avatar
Brian
Developer
Posts: 2673
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: BUG: Excessive disk IO

Post by Brian »

CollinChaffin wrote:... to fully re-read the entire resource(s).
I think you are confusing a complete reload of a resource with a query of that resource. All your video shows is Rainmeter querying the properties of the image to see if that image has changed. In other words, Rainmeter needs to know if the image has changed in order to re-load it and display the new image. This is fundamental to Rainmeter.

Rainmeter caches each image until it is no longer needed. So on each update of a image meter it asks Windows if that image has changed in some way, and if so it will load the new image. If the image has not changed, it uses the cached image from memory. During this update process if for some reason an image is no longer being referenced, it is removed from our cache.


In Procmon, when you see QueryStandardInformationFile, this is where the system has been asked to fetch some file information. It isn't loading that resource directly, it is just asking for some properties like (size, end of file marker, links, if a delete is pending, if it is a directory). It is only a few bytes of data and isn't going to damage your SSD. Rainmeter doesn't query this information directly, it asks Windows (through the Windows API) for this information.


Having said that (depending on the skin design), you can tweak some more efficiency out of Rainmeter by utilizing UpdateDivider=-1 on image meters, and manually updating meters when changing images instead of having Rainmeter check on every update if the image has changed. This will all depend on your overall skin design and whether you are using images that are the same over time, or images that change (even if in name only) over time.

-Brian
CollinChaffin
Posts: 2
Joined: May 7th, 2017, 6:34 am

Re: BUG: Excessive disk IO

Post by CollinChaffin »

Hi Brian!

Thanks for the reply but in procmon, in addition to simply doing the querystandardinformationfile, if you look it IS actually doing a full ReadFile operation with offsets of zero, calling for the entire length of the file (read all).

I've been in IT and DEV myself for almost 30yrs, so bear in mind I actually do have a good understanding of how this stuff works and wouldn't have posted if I merely saw a querystandinfofile etc. but what I see is (and if you slow down the scroll and look again), full file reads after that call when nothing has changed - and would actually indicate the opposite of what you stated which is EXACTLY how I would think it should function with caching the files until an actual file change. BTW, other than updating a meter, NONE of these PNG resources EVER change, and IMO why would they EVER need to be re-read a single time from the time the skin is loaded, anyway? Those are the files you see being read from offset 0 to EOF over and over with corresponding disk read counters showing those read bytes.

The first response I got that it is basically the skins fault or that the skin is doing what it should and I'm not sure if that was an actual Rainmeter DEV or just a user taking a guess but - I have parsed through these skins but will do so again but even though some of them ARE in fact looping for animations and again my point of posting here is, even if I write a skin that uses the rainmeter exposed API functions to ask for 30 png's to be re-read for an animation - I then look back to the rainmeter code behind that exposed API to ask "when the skins use this API, do I cache all resources called in the param in RAM, or is my default behavior that of doing a file io stream read of each file to EOF - which again is why I asked here since all I can see is (regardless of HOW the file refresh/animation is being requested back through rainmeter) procmon and other filemon utils absolutely showing all those bytes being re-read constantly.

Also, to back this up if you take a look in taskmgr or resmanager you can see it as well. I only had a min so quickly just captured my open rainmeter that's been open again for a couple days and yes fully understand that the taskmgr view IOs also includes net and device io from the rainmeter process, but either way, with a couple simple animations and the machine sitting idle, it's pretty tough to do the math and explain THAT level of both TOTAL bytes and most importantly the RATE at which the number of is climbing, if in fact all it was doing was a simple queryinfofile call which is a mere few byte call, right? Again, you've got procmon I showed matching taskmgr IO counters matching resmon disk counters but if I'm off here I apologize ahead of time just want to be real sure I'm wrong about what this looks like to me.

https://youtu.be/M_a3_myn_Vw

If you could maybe just take another peek and maybe verify in the code and then tell me what you think I would really appreciate it. Thanks again!

Collin
User avatar
theAzack9
Developer
Posts: 522
Joined: March 2nd, 2016, 6:23 pm

Re: BUG: Excessive disk IO

Post by theAzack9 »

I will not really make any large statement on this, since i'm still kinda new and don't know how all the cogs are turning yet. (In the code, that is ;) )

From what i'm understanding is this an issue with animations, where images (ranging from 0-n frames) are being constantly loaded. This would make sense as Rainmeter is "caching" the images in the sense that it keeps the images that are currently being used by a meter. So when a e.g Image meter changes the image (through changing the ImageName option) will the meter point to a new image and the old image will no longer be used by any meters. This makes it so the previous image is unloaded since it is no longer used. This continues until the animation hits frame n and starts over again and now reloads frame 0 from the drive. This is just my 5 cents though and i might be wrong, brian probably has a better idea than me.

Assuming i'm correct. While the caching could be better by e.g keeping the images in ram for a certain time frame will i say that i don't really think rainmeter was created with animations in mind. Even though it is a nice feature to have. ;)
Last edited by theAzack9 on May 9th, 2017, 10:11 pm, edited 1 time in total.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: BUG: Excessive disk IO

Post by jsmorley »

I would concur with theAzack9 that it makes some sense that an animation based on an Image meter would do a ton of file reading. While we cache the image used by an Image meter so it isn't read on each update and redraw of the skin, the Image meter is going to react to any change in ImageName by assuming it is a different image, and getting it from the disk. It has no way to know that this is a repeating set of images that is being used in an animation. Any change to ImageName is just seen as a brand new image that must be read.

There may be something that can be done about this, but it must be with some careful thought. We can't just permanently cache every image used in a a skin while it is loaded, or a WebParser skin getting "random" images from some site every 10 minutes is eventually going to cause Rainmeter to use all available memory. We can't assume that the same "name" of a file, particularly when it's in the Rainmeter "temp" folder Download=1 on WebParser uses, is really the same image. It likely isn't. Again, I don't doubt that this could be more efficient, but the answer may not be as simple as all that...
User avatar
Brian
Developer
Posts: 2673
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: BUG: Excessive disk IO

Post by Brian »

CollinChaffin wrote:Thanks for the reply but in procmon, in addition to simply doing the querystandardinformationfile, if you look it IS actually doing a full ReadFile operation with offsets of zero, calling for the entire length of the file (read all).
I only saw a 'ReadFile' used sparingly in your video and I am not convinced it is being used inappropriately.

We only call 'ReadFile' when first loading an image, or if that images changes in some way (size, file date, etc.). If the image has not changed, we use the image already loaded in memory.
Our image loading function: https://github.com/rainmeter/rainmeter/blob/master/Library/TintedImage.cpp#L298-L410
More specifically: https://github.com/rainmeter/rainmeter/blob/master/Library/TintedImage.cpp#L328-L333
Our LoadImageFromHandle function: https://github.com/rainmeter/rainmeter/blob/master/Library/TintedImage.cpp#L190-L292

After reviewing your video again, it appears that the ReadFile is happening when some image meter changes. This indicates some animation (or the meter is changing what image is displayed) in the various skins you are using. I saw images change from li34.png to li36.png and Dna64.png to Dna66.png (and various other numbered images). The reason you see it switch between those images is because your skin is changing those meters dynamically. Once the previous image is no longer needed (by OTHER meters), we unload it. Rainmeter can't guess that you will need that image in the future.

Rainmeter only does what you tell it to do.

I am not suggesting that 'our code' is perfect or cannot be tweaked for better efficiency, but we do cache all images WHILE being used. If the meter moves on to another image and is NOT being referenced by other meters, it is unloaded - otherwise it stays in memory for usage in those other meters.

-Brian
User avatar
SilverAzide
Rainmeter Sage
Posts: 2588
Joined: March 23rd, 2015, 5:26 pm
Contact:

Re: BUG: Excessive disk IO

Post by SilverAzide »

Sorry to interject into this interesting thread... but assuming one wanted to do animation with an image meter AND minimize disk I/O, what would be the official recommended approach?

My first thought would be to use a single bitmap that combines all images/frames of the animation, then use ImageCrop to display the image you want. This would probably necessitate DynamicVariables=1 as well. In this way, the ImageName on the meter would never change. Would this methodology minimize disk I/O whether there is or is not a bug/issue in Rainmeter? If so, would a general rule be that if an Image meter changes often, you should not reset the ImageName, but instead use ImageCrop and etc.? (Specifically I'm thinking of system monitoring skins that use blinking status lights and the like.)
User avatar
Brian
Developer
Posts: 2673
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: BUG: Excessive disk IO

Post by Brian »

SilverAzide wrote:Sorry to interject into this interesting thread... but assuming one wanted to do animation with an image meter AND minimize disk I/O, what would be the official recommended approach?

My first thought would be to use a single bitmap that combines all images/frames of the animation, then use ImageCrop to display the image you want. This would probably necessitate DynamicVariables=1 as well. In this way, the ImageName on the meter would never change. Would this methodology minimize disk I/O whether there is or is not a bug/issue in Rainmeter? If so, would a general rule be that if an Image meter changes often, you should not reset the ImageName, but instead use ImageCrop and etc.? (Specifically I'm thinking of system monitoring skins that use blinking status lights and the like.)
This is one way you could reduce disk I/O, but I would think the overall size of the combined image might be troublesome, and the necessary calculations might be difficult to get right. Also, with DynamicVariables=1, you will lose a little in CPU usage.


Another approach would be create a 'dummy' image meter for every image you intend to display with your skin (obviously this won't work for images downloaded with Webparser). By setting an appropriate negative X and Y with the meter, you essentially load the image without displaying it. The image should stay cached, and you should be able to use the image in other meters without any disk I/O. In fact, you could create a dummy skin that loads all your images for use in other skins.

Either option will obviously increase the memory Rainmeter uses, but should reduce disk I/O if that is your goal. At the end of the day, Rainmeter was not designed for animation and any attempt to do animations (or dynamically change images) will result in a combination of disk I/O, CPU usage and memory usage.


Just as an aside, this thread has got us thinking about some sort of animation meter that is specifically designed to cache a known set of images - or maybe we will automatically cache all images in the @Resources\Images folder or something similar. Either way, it is just hypothetical at this stage.

-Brian

Edit - Interesting article on file caching. https://msdn.microsoft.com/en-us/library/windows/desktop/aa364218(v=vs.85).aspx
Microsoft wrote:By default, Windows caches file data that is read from disks and written to disks. This implies that read operations read file data from an area in system memory known as the system file cache, rather than from the physical disk.
Microsoft wrote:This ensures that file data that is frequently read will stay accessible in the system file cache for the maximum amount of time.
User avatar
jsmorley
Developer
Posts: 22628
Joined: April 19th, 2009, 11:02 pm
Location: Fort Hunt, Virginia, USA

Re: BUG: Excessive disk IO

Post by jsmorley »

Procmon is seeing reads from the Windows File Cache (remember Smartdrive from the day?) as I/O calls, which they are, but this should not be misinterpreted as reads from the physical disk, which they are not...

If you download the .rmskin on this page:

https://docs.rainmeter.net/tips/animated-gif-files/

And load it, you will see that although Rainmeter is going to request a file read for each image every time, in reality the image "frame" files are physically read once, when you load the skin, and never again. Windows itself is caching the files, and it's only Procmon that is seeing that I/O from the caches as a read of the file(s). There is in fact no actual read as long as the files themselves are not changed on the disk.

Even if you unload and reload the skin, or even exit and restart Rainmeter entirely, those image files are not physically read again until they "age out" of the Windows cache (no idea how long that takes, although I suspect NEVER if the file is being constantly asked for) or you reboot your system.

Certainly any animation in Rainmeter that is dynamically changing the ImageName option on an Image meter once a second to achieve the animation is going to be requesting the file(s) once a second. Don't worry though, Rainmeter isn't thrashing your hard drive. Any assertion that running a Rainmeter skin for a long period of time wore out your hard drive due to "reads" is almost certainly false.

https://technet.microsoft.com/en-us/sysinternals/diskmon.aspx
Post Reply