It is currently May 19th, 2022, 2:27 pm

[CRASH to SUGGESTION] Draw to memory Shape or Image

Report bugs with the Rainmeter application and suggest features.
User avatar
Yincognito
Rainmeter Sage
Posts: 4694
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

[CRASH to SUGGESTION] Draw to memory Shape or Image

Post by Yincognito »

I'll spare you of the details of what I want to achieve, but a while ago I found out a way to get the value of all or any pixels in an image, using a simple combination of batch and Python, like this (assuming a [SkinFolder]\@Resources\Image.tif or any other image type exists, of course)...

[SkinFolder]\@Resources\ImagePixelRGB.py:

Code: Select all

# Needs "pip install pillow" to be run in CMD first, on a new Python installation
from PIL import Image
icon = Image.open('Image.tif')
size = icon.size
data = list(icon.getdata())
print(size, data)
[SkinFolder]\@Resources\ImagePixelRGB.bat:

Code: Select all

@echo off
ImagePixelRGB.py > ImagePixelRGB.inc 2>&1
which, after running the batch, produces a nice .inc file with the RGB values of all pixels in the image and its resolution as a bonus, similar to...

[SkinFolder]\@Resources\ImagePixelRGB.inc:

Code: Select all

(1288, 644) [(12, 16, 29), (7, 10, 25), (4, 8, 22), (4, 7, 22), (3, 7, 22), ... , (242, 242, 242)]
With minor editing of the Python code to something like:

Code: Select all

# Needs "pip install pillow" to be run in CMD first, on a new Python installation
from PIL import Image
icon = Image.open('Image.tif')
size = icon.size
data = list(icon.getdata())
print("[Foreground]")
print("Meter=Shape")
for y in range(0, size[1]):
  for x in range(0, size[0]):
    print("Shape"+str("" if size[0]*y+x==0 else size[0]*y+x+1)+"=Rectangle "+str(x)+","+str(y)+",1,1 | StrokeWidth 0 | Stroke Color "+str(data[size[0]*y+x][0])+","+str(data[size[0]*y+x][1])+","+str(data[size[0]*y+x][2])+",255")
and a bit more processing time, one can even get a perfectly valid .inc file like:

Code: Select all

[Foreground]
Meter=Shape
Shape=Rectangle 0,0,1,1 | StrokeWidth 0 | Stroke Color 12,16,29,255
Shape2=Rectangle 1,0,1,1 | StrokeWidth 0 | Stroke Color 7,10,25,255
Shape3=Rectangle 2,0,1,1 | StrokeWidth 0 | Stroke Color 4,8,22,255
...
Shape829472=Rectangle 1287,643,1,1 | StrokeWidth 0 | Stroke Color 242,242,242,255
that is theoretically, ready to draw in Rainmeter, with a simple code similar to...

[SkinFolder]\ImagePixelRGB.ini:

Code: Select all

[Variables]

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1

---Meters---

[Background]
Meter=Shape
Shape=Rectangle 0,0,1288,644 | Fill Color 0,0,0,255 | Stroke Color 255,0,0,255 | StrokeWidth 1
UpdateDivider=-1

[Foreground]
@IncludeForeground=#@#Foreground.inc
UpdateDivider=-1
Similarly, if changing the (version 3.0+ recommended, to easily write stuff on the same line) Python code to:

Code: Select all

# Needs "pip install pillow" to be run in CMD first, on a new Python installation
from PIL import Image
icon = Image.open('Image.tif')
size = icon.size
data = list(icon.getdata())
print("color = ",end="")
print("{",end="")
for x in range(0, size[0]):
  print(("" if x==0 else ",")+"{",end="")
  for y in range(0, size[1]):
    print(("" if y==0 else ",")+"{"+str(data[size[0]*y+x][0])+","+str(data[size[0]*y+x][1])+","+str(data[size[0]*y+x][2])+","+str(255)+"}",end="")
  print("}",end="")
print("}")
and the batch code to:

Code: Select all

@echo off
ImagePixelRGB.py > ImagePixelRGB.lua 2>&1
one could even build a Lua dofile similar to...

[SkinFolder]\@Resources\ImagePixelRGB.lua:

Code: Select all

color = {{{12,16,29,255},{14,18,32,255},{7,10,25,255},...,{242,242,242,255}}}
and have a Lua script similar to...

[SkinFolder]\@Resources\Script.lua:

Code: Select all

function Initialize()
  dofile(SKIN:GetVariable('@') .. 'ImagePixelRGB.lua')
end

function Color(x, y)
  return tostring(color[x][y][1]) .. ',' .. tostring(color[x][y][2]) .. ',' .. tostring(color[x][y][3]) .. ',' .. tostring(255)
end
which could then be used in the .ini file like:

Code: Select all

[Variables]

[Rainmeter]
Update=1000
DynamicWindowSize=1
AccurateText=1

---Measures---

[Script]
Measure=Script
ScriptFile=#@#Script.lua
UpdateDivider=-1

---Meters---

[Background]
Meter=Shape
Shape=Rectangle 0,0,1288,644 | Fill Color 0,0,0,255 | Stroke Color 255,0,0,255 | StrokeWidth 1
UpdateDivider=-1

[Foreground]
Meter=Shape
Shape=Rectangle 0,0,1,1 | StrokeWidth 0 | Stroke Color [&Script:Color(0,0)]
Shape2=Rectangle 1,0,1,1 | StrokeWidth 0 | Stroke Color [&Script:Color(1,0)]
Shape3=Rectangle 2,0,1,1 | StrokeWidth 0 | Stroke Color [&Script:Color(2,0)]
...
Shape829472=Rectangle 1287,643,1,1 | StrokeWidth 0 | Stroke Color [&Script:Color(1287,643)]
UpdateDivider=-1
DynamicVariables=1
However, in practice, even though everything is as it's supposed to be, such a skin will crash Rainmeter, due to too many shapes, too much code or something along those lines - so be aware to comment a too large [Foreground] meter content like:

Code: Select all

[Foreground]
; @IncludeForeground=#@#Foreground.inc
and similar in the case of Lua from the spoiler, to avoid unpleasant surprises. Naturally, all the other ways of drawing such a construct in a skin by automatically building the .inc file using the method above (like using Line shapes with hard stop gradients to simulate pixels, or using 1 px sized Image meters, etc.) will crash as well. Basically, one is not able to draw a "normal" (i.e. larger than an icon sized) image pixel by pixel in Rainmeter.

If you got this far (yeah, I know, it's longer than expected cause I always cover all possibilities, to be sure I'm not halucinating when I say something), this is the point where you may wonder and outright reject any meaningful discussion about it on the grounds that "why don't you just use the Image meter for this" or "Rainmeter was not designed for this" and so on - but the thing is, I considered using these methods because I want to dynamically transform an image in ways that are not possible using the current set of tools available in Rainmeter (this is where I spare you of all the details, like mentioned above). So, forget about those ways to say "no" for a moment and let's focus on ways to possibly say "yes": is or can it be possible to alter specific pixels in an image, or draw an image pixel by pixel using a feasible speed in Rainmeter? And if not, could a "buffer" or "pointer" measure be created for Rainmeter that will do these operations in the very fast RAM/GPU memory and let the user then display the (converted to an image, internally) result in an Image meter?

Thanks, and sorry for taking from your precious time.
User avatar
death.crafter
Rainmeter Sage
Posts: 1364
Joined: April 24th, 2021, 8:13 pm

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by death.crafter »

Tbh, I would just use adobe illustrator to make a set of sprites and use it for animation rather than using this, though yes, it strikes some interest.

Also, due to too many shapes, as you say, and as anyone would guess, will crash Rainmeter. And I believe the crash is due to too many shapes and not the size of the inc file.

Now, what strike my interest is whether I could use an plugin to do it. I, ofc, can create a bitmap, set pixels and save the image. But it would be too resource intensive to save an image every update, given it's gonna run for a long time. So, that's where we come to the load to memory part.

But I wonder if the devs would be interested to expose the location of the cached image of an image meter, which we can in turn manipulate to inject the bitmap created by the plugin and change the shown image. This way might be more efficient than "the plugin making the image, saving the image, and then the image meter loading the image to memory".

But, this is just an theory, one which could violate memory safety. So I'll just wait for what the devs think about it.
If they do, ofc :lol:
from the Realm of Death
User avatar
Yincognito
Rainmeter Sage
Posts: 4694
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by Yincognito »

death.crafter wrote: April 2nd, 2022, 6:29 pm Tbh, I would just use adobe illustrator to make a set of sprites and use it for animation rather than using this, though yes, it strikes some interest.

Also, due to too many shapes, as you say, and as anyone would guess, will crash Rainmeter. And I believe the crash is due to too many shapes and not the size of the inc file.

Now, what strike my interest is whether I could use an plugin to do it. I, ofc, can create a bitmap, set pixels and save the image. But it would be too resource intensive to save an image every update, given it's gonna run for a long time. So, that's where we come to the load to memory part.

But I wonder if the devs would be interested to expose the location of the cached image of an image meter, which we can in turn manipulate to inject the bitmap created by the plugin and change the shown image. This way might be more efficient than "the plugin making the image, saving the image, and then the image meter loading the image to memory".

But, this is just an theory, one which could violate memory safety. So I'll just wait for what the devs think about it.
If they do, ofc :lol:
Actually, I thought about proposing the idea of such a plugin to you, given that you jumped quite successfully into the plugin makers camp. :D I would do it myself if C++ wouldn't be hard to decipher for me, and C# a language I didn't program in for a long time (last time, incidentally, also to make a collaborative plugin, this time for Sony Vegas, where luckily the format was a bit more comprehensible to me).

Since you're open minded and you can think about it at least as a possibility, I can share the purpose to you. I have my rotating 3D globe skin working perfectly for a while now, but while I can rotate on the X (aka desktop plane) axis via ImageRotate, TransformationMatrix and such without any resource investment bar the one required for the animation, I have to use about 4.5 GB of hard drive space to store the rotations on the Y and Z axes designed in Blender, as 360 degrees x 360 degrees x 32 KB JPGs of 414 x 414 px. I'd like to avoid that and do the plane to spherical projection myself in Lua, using the pixel color data above and then somehow transfer that "buffered" info to Rainmeter using only the fast RAM (or GPU?) memory so that a regular Image meter can use it to display the image from memory instead of loading it slowly from disk.

I tried a rudimentary way using the codes above for a plain, untransformed image, but couldn't even see it trying to draw due to not being able to have that many shapes or meters. It was rudimentary because obviously doing this pixel by pixels is in effect going to be way more resource intensive than even loading such an image from disk (which is why I included the Lua table version as well). Ideally, all the pixel color data would be loaded into memory in a single piece and step (hence the "buffer" word usage), formatted as a (fast accessible) image format linked to an Image meter or a potential Buffer measure, which could then be similarly drawn in a single step by the corresponding Image meter as usual - but without involving the disk at all.

Anyway, it's a nice dream, but nobody stops me from having it... 8-)
User avatar
Brian
Developer
Posts: 2414
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by Brian »

Yincognito wrote: April 2nd, 2022, 2:39 pm is or can it be possible to alter specific pixels in an image, or draw an image pixel by pixel using a feasible speed in Rainmeter?
I am not even sure how that would even work in an ini-style format. I am not even sure how the parsing would work. I mean, presumably it would take 2 coordinates, and color information....for each pixel? Seems like a ton of parsing/processing. I mean, isn't this why images are in some form of a binary format? I about died when your example meter had nearly 830,000 shapes.

Man, I thought my 5,287 shape (5,032 paths) meter was ridiculous. :D

Even if there is a feasible way, generally setting/getting specific pixel information is slowwwww with D2D. Even our mouse detection for Shapes is relatively slow given how D2D access's pixel data on the GPU (we basically have to ending the drawing process, access the "back buffer" bitmap directly, then re-enable drawing).




death.crafter wrote: April 2nd, 2022, 6:29 pm But I wonder if the devs would be interested to expose the location of the cached image of an image meter, which we can in turn manipulate to inject the bitmap created by the plugin and change the shown image. This way might be more efficient than "the plugin making the image, saving the image, and then the image meter loading the image to memory".
As far as "exposing" internal memory locations of images (even to plugins), this isn't really feasible due to the risk of memory corruption, and potential security issues (as you eluded to). We won't be giving away "drawing" operations to code beyond Rainmeter's control. Even some sort of "load my binary bitmap data, I promise it is an image" type of interface to the Image meter, still runs the risk of trying to load corrupted data that Rainmeter does not control.

This reminds me of some older plugins (like the old FileList plugin) that directly tried to access some of Rainmeter's internal bitmaps for the image meter (to draw icons). The plugin didn't work correctly, and would eventually (over some time) corrupt the drawing of windows in other programs and the OS itself. Ultimately the plugin author disappeared, and we had to deal with the aftermath (hence the birth of the FileView plugin).



I am not trying to say "No" to all of this, I just don't see a way to do it in a safe-enough way that allows direct manipulation of "drawing" functions/data from an outside source.

-Brian
User avatar
death.crafter
Rainmeter Sage
Posts: 1364
Joined: April 24th, 2021, 8:13 pm

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by death.crafter »

Fair enough. It ofc would have been cool, but safety is of greater concern here. So, let's let this topic rest in peace :lol:
from the Realm of Death
User avatar
Yincognito
Rainmeter Sage
Posts: 4694
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by Yincognito »

Brian wrote: April 3rd, 2022, 7:13 am I am not even sure how that would even work in an ini-style format. I am not even sure how the parsing would work. I mean, presumably it would take 2 coordinates, and color information....for each pixel? Seems like a ton of parsing/processing. I mean, isn't this why images are in some form of a binary format? I about died when your example meter had nearly 830,000 shapes.

Man, I thought my 5,287 shape (5,032 paths) meter was ridiculous. :D

Even if there is a feasible way, generally setting/getting specific pixel information is slowwwww with D2D. Even our mouse detection for Shapes is relatively slow given how D2D access's pixel data on the GPU (we basically have to ending the drawing process, access the "back buffer" bitmap directly, then re-enable drawing).

As far as "exposing" internal memory locations of images (even to plugins), this isn't really feasible due to the risk of memory corruption, and potential security issues (as you eluded to). We won't be giving away "drawing" operations to code beyond Rainmeter's control. Even some sort of "load my binary bitmap data, I promise it is an image" type of interface to the Image meter, still runs the risk of trying to load corrupted data that Rainmeter does not control.

This reminds me of some older plugins (like the old FileList plugin) that directly tried to access some of Rainmeter's internal bitmaps for the image meter (to draw icons). The plugin didn't work correctly, and would eventually (over some time) corrupt the drawing of windows in other programs and the OS itself. Ultimately the plugin author disappeared, and we had to deal with the aftermath (hence the birth of the FileView plugin).

I am not trying to say "No" to all of this, I just don't see a way to do it in a safe-enough way that allows direct manipulation of "drawing" functions/data from an outside source.

-Brian
Believe it or not, your shape skin actually gave me the idea of trying to transform the original image this way. Yeah, I know, my definition for ridiculous isn't quite mainstream, but I loved your skin from that point of view. :lol:

Unfortunately, decreasing the number of shapes for an Earth like map is difficult due to the details involved, which is why I ended up with a pixel by pixel attempt using the tools at my disposal. The fact that it has almost a million shapes is not that important in the context of this example, because that's only the number of "input" pixels for a couple of initial images that once I have the pixel data for, I won't need anymore (they represent the "material" to be projected, to be clearer). After the envisioned processing to project the pixel data to a 3D sphere, there would "only" be 414 x 414, aka 171,396 pixels / shapes (or a 414 x 414 px image) to draw, so a much smaller amount.

I perfectly understand what you're saying, I was only proposing one way of doing it via memory access, which might not be the best one due to security concerns, of course. If you were not mentioning how slow is setting/getting specific pixel information is with D2D, I would have thought of something like an "Image measure" that you'd be able to set its content via a widh and height along with pixel by pixel data, and which could be used as source for the Image meter, like:

Code: Select all

[MeasureImage]
Measure=Image
Width=414
Height=414
X0Y0=12,16,29,255
X1Y0=7,10,25,255
X2Y0=4,8,22,255
...
X414Y414=242,242,242,255

...
...
...
[MeterImage]
Meter=Image
MeasureName=MeasureImage
...
This is just another way this could be approached, and it doesn't involve security concerns regarding memory, since we only set some pixel color data (that can be set from Lua too, so we could transform the image any way we like if desired, in terms of strict math). Now that you mentioned how slow is the D2D get/set pixel info, I'm not so sure it would be feasible in terms of speed (considering I also do animation, so the transforming would be done on a low update rate).

From my part, bar the exasperatingly slow access of a folder having 129600 images, I'm ok with needing 4.5 GB to store the Y and Z rotation images for the Earth image in my skin, but that obviously won't be that suited for distribution, so I guess it will remain a personal skin, after all. :confused:
death.crafter wrote: April 3rd, 2022, 7:30 amFair enough. It ofc would have been cool, but safety is of greater concern here. So, let's let this topic rest in peace :lol:
Well, it seems being in the plugin developer boat now, it's easier to rest topics in peace than before... :D
User avatar
death.crafter
Rainmeter Sage
Posts: 1364
Joined: April 24th, 2021, 8:13 pm

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by death.crafter »

Yincognito wrote: April 3rd, 2022, 5:51 pm

Code: Select all

[MeasureImage]
Measure=Image
Width=414
Height=414
X0Y0=12,16,29,255
X1Y0=7,10,25,255
X2Y0=4,8,22,255
...
X414Y414=242,242,242,255

...
...
...
[MeterImage]
Meter=Image
MeasureName=MeasureImage
...
Imagine even if we got this kind of thing to work, imagine the number of setoptions you would have to call to change the image.

Tho, this could be a thing...

Code: Select all

[MeasureImage]
Measure=Image
Resolution=414x414
Frame1=[(a1, r1, g1, b1) (a1, r1, g1, b1) ... (a1, r1, g1, b1)]
Frame2=[(a2, r2, g2, b2) (a2, r2, g2, b2) ... (a2, r2, g2, b2)]
...
FrameN=[(aN, rN, gN, bN) (aN, rN, gN, bN) ... (aN, rN, gN, bN)]
I will see if I can cook something up... But I wonder if the size of these files with defined frames would be any smaller than the images them selves.
from the Realm of Death
User avatar
Yincognito
Rainmeter Sage
Posts: 4694
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by Yincognito »

death.crafter wrote: April 3rd, 2022, 9:15 pm Imagine even if we got this kind of thing to work, imagine the number of setoptions you would have to call to change the image.

Tho, this could be a thing...

Code: Select all

[MeasureImage]
Measure=Image
Resolution=414x414
Frame1=[(a1, r1, g1, b1) (a1, r1, g1, b1) ... (a1, r1, g1, b1)]
Frame2=[(a2, r2, g2, b2) (a2, r2, g2, b2) ... (a2, r2, g2, b2)]
...
FrameN=[(aN, rN, gN, bN) (aN, rN, gN, bN) ... (aN, rN, gN, bN)]
I will see if I can cook something up... But I wonder if the size of these files with defined frames would be any smaller than the images them selves.
As I said, that was just a sketch of what it would look like, by no means definitive or set in stone. From my part, the pixel data could be defined in a single option too, like:

Code: Select all

Pixels=A1,R1,G1,B1;A2,R2,G2,B2;A3,R3,G3,B3;...
Yes, the size of the files would be greater than an image of similar size, because no compression would be applied to the raw string data, not to mention that it won't be in binary form - that's one of the reasons I mentioned "buffering" earlier, as doing stuff exclusively in memory would have avoided writing them in an .ini file explicitly.

My problem is not with the frames, by the way: in my case, I would store / use the pixels of a single frame or image (414 x 414). It's just that I would need a way to change the color value of the pixels in that frame / image, aka "transform" the image, that's all. Not even I would dare to store zillions and zillions of frames in an .ini file, LMAO. After all, only a single frame or image is visible at a specific time in the skin - no need to deal with more at once. As I showed above, I can deal with getting the original pixel color data myself via Python, it's only "exporting" the (Lua) processed data back to a Rainmeter Image meter that I can display in my skin which is unavailable to me at this time.

If you can examine the possibilities, that would be great. I'm not expecting a full fledged plugin or asking that from you or anyone else - just see if it can somehow be done, for the moment. ;-)

P.S. The other alternative would be to have a 3DShape meter in Rainmeter, where one could project an image on a 3D mesh and display it, but that would put too much weight on the developers and would be a bit unrealistic for what Rainmeter is supposed to do. I'm willing to take that weight on my shoulders, if only I had a way to "set" an Image meter to use the image/pixel color data that I process via other means (in this case, Lua, given its interoperability with Rainmeter).
User avatar
death.crafter
Rainmeter Sage
Posts: 1364
Joined: April 24th, 2021, 8:13 pm

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by death.crafter »

Yincognito wrote: April 3rd, 2022, 11:19 pm P.S. The other alternative would be to have a 3DShape meter in Rainmeter, where one could project an image on a 3D mesh and display it, but that would put too much weight on the developers and would be a bit unrealistic for what Rainmeter is supposed to do. I'm willing to take that weight on my shoulders, if only I had a way to "set" an Image meter to use the image/pixel color data that I process via other means (in this case, Lua, given its interoperability with Rainmeter).
It's possible with Plugin WebView by khanhas, which uses Edge Webview to render web pages in Rainmeter. But that is too incomplete for general use. You can use that to render 3D objects.

But an 3D shape meter sounds cool, ignoring the work involved :lol:
from the Realm of Death
User avatar
Yincognito
Rainmeter Sage
Posts: 4694
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: [CRASH to SUGGESTION] Draw to memory Shape or Image

Post by Yincognito »

death.crafter wrote: April 4th, 2022, 4:54 am It's possible with Plugin WebView by khanhas, which uses Edge Webview to render web pages in Rainmeter. But that is too incomplete for general use. You can use that to render 3D objects.

But an 3D shape meter sounds cool, ignoring the work involved :lol:
Hmmm... if that does what I guess it does, it could very well be the solution to my problem, assuming it can use a Three.js script to draw a 3D globe in a skin, and the rendering and rotation are acceptable in terms of CPU usage (I'm OK with RAM usage anyway) - many thanks for the link! It introduces some dependencies that I normally avoid, but if it can feasibly do the above and save me from storing stuff in several GB on disk, it's worth to try. I can minimize the rendering work by using a single source material image with the bump and cloud maps already bundled in the source image and simulate both light and atmospheric differences via regular semi transparent radial Rainmeter gradients if necessary, as long rotating takes reasonable CPU usage. Not sure how it deals with scaling which I also use in the skin, but we'll see, when I'll get back to my computer.

By the way, why did you say it's incomplete?