It is currently July 10th, 2020, 1:21 pm

distribute skins via github and release via github actions

Report bugs with the Rainmeter application and suggest features.
User avatar
2bndy5
Posts: 19
Joined: February 25th, 2015, 2:38 am

distribute skins via github and release via github actions

Post by 2bndy5 »

:welcome: Has the idea of using github to distribute skins been standardized? Since DeviantArt.com doesn't seem to support Markdown or reStructuredText in the descriptions, I thought maybe I can use a github repo's readme file. Then it would be easier to include some documentation on certain features of a skin that aren't too obvious or have conditional quirks. I suppose we could still use DeviantArt to host the preview image (although that can be done through github also). Additionally, it would be easier to encourage group collaboration or bug fixes on a certain skin. 8-)

Now I'm wondering how we could use github actions to package a skin for release using the same method I use to create a .rmskin file through rainmeter. Of course I'm assuming one skin per repo, but the possibility to split a suit of skins located in one repo may also be desirable. I did find out that we can specify "windows-latest" on the github-hosted runners. Now what would I have to do to make github actions package a skin into a .rmskin file? Would it be as easy as creating a zip file and renaming the extension rmskin (in that case - why not use linux env)?
User avatar
Brian
Developer
Posts: 2024
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: distribute skins via github and release via github actions

Post by Brian »

2bndy5 wrote:
May 18th, 2020, 8:32 am
Has the idea of using github to distribute skins been standardized?
I am not sure any standardization is really needed as many skin authors already release their skins on github. I don't think there is any "right" way of storing skins on github....but it would be better to add your packaged rmskin as an asset when publishing a "release". The creation of a "readme" or documentation is totally up to the skin author.

The real problem with github is there is no real "group" feature or easy search (like an image gallery). There is a "search topic", but it requires a search term such as rainmeter or rainmeter-skin...which basic users may not stumble upon. Even then, you literally have to click on each one to "see" what it is all about. Github is really geared for developers, so it may not jive well for the normal basic user of Rainmeter.

I guess we could make a static "gallery" github page where users could submit a link to their github skin project, then we could link to it....but this would be a complicated project to get going, plus a lot of "approval" actions from the team. We currently don't have the time to do this at the moment. If someone wants to submit a proof of concept type of thing to us, I think we would at least discuss it.


2bndy5 wrote:
May 18th, 2020, 8:32 am
Now I'm wondering how we could use github actions to package a skin for release using the same method I use to create a .rmskin file through rainmeter. Of course I'm assuming one skin per repo, but the possibility to split a suit of skins located in one repo may also be desirable. I did find out that we can specify "windows-latest" on the github-hosted runners. Now what would I have to do to make github actions package a skin into a .rmskin file? Would it be as easy as creating a zip file and renaming the extension rmskin (in that case - why not use linux env)?
Yes, this is possible, but it is not as simple as renaming a zip file. We write a custom "footer" to the file in an effort to force authors to use the skin packager in Rainmeter. This is why when you rename your rmskin to zip and add a file, the rmskin is no longer valid. While there is no "automatic" skin packaging in Rainmeter proper, I did create a "proof of concept" automatic rmskin packager using github actions that would zip up the skin files (in rmskin format), then it writes the Rainmeter footer, then automatically upload the rmskin to the github release. This made it really easy to push any changes to your skin files, then just create a github release and the github action would do the rest. My version did use "windows-latest" since I mainly used a powershell script to create the rmskin. The only problem with my version, was it was non-validating - meaning it would zip up any files I wanted then create a valid rmskin that Rainmeter would not understand. I am sure a "validating" script can be made.

We are still debating the pros and cons of an "official" Rainmeter automatic skin package for github actions. We made the rmskin format in an effort to provide a bit more security from malware (which was a real problem several years ago [link] [link] [link]) - so we really encourage skin authors to use the built-in skin packager that comes with Rainmeter. We believe that user-interaction is key...but we do understand that sometimes the effort to create a rmskin package can be difficult for larger skins or skin suites. Since Rainmeter is open source, anyone can figure out how to write our custom "footer", so maybe providing a way using github actions might be acceptable. Time will tell.

-Brian
User avatar
2bndy5
Posts: 19
Joined: February 25th, 2015, 2:38 am

Re: distribute skins via github and release via github actions

Post by 2bndy5 »

I guess we could make a static "gallery" github page where users could submit a link to their github skin project...
I'm not too interested in creating a "gallery" for skins (the search term "rainmeter-skin" seems to work well enough), but I am adamant about the github actions idea as using the "create package dialogue" gets rather tedious.
I did create a "proof of concept" automatic rmskin packager using github actions...
Could you give a link to your work if it still exists?
We write a custom "footer" to the file in an effort to force authors to use the skin packager in Rainmeter.
I appreciate the need for authentication when packaging rmskin files. Correct me if I'm wrong, but it looks like rainmeter's dialoguePackage.cpp adds a string, "RMSKIN", after the CRC data. while also creating a RMSKIN.ini at root of the archive.

My idea for a CLI-type program (IDK powershell script) that takes a skin (path-to-root-dir) and spits out a validating rmskin file (in same dir if output destination is not specified by an argument). Use the metadata section from the first ini file in root dir of package (sought alphabetically if the ini file to load after install is not specified by a cmd arg) to fill in "author" and "version" info. Let the skin's root dir name be the package name if not specified by a cmd arg. If the metadata section doesn't exist and is not specified via cmd args, then abort. If metadata is specified by cmd args, then I'd like the program to add/update it appropriately to skin's on-first-install-loaded ini file (of course this wouldn't apply when a layout is specified to load on first install); in this case semantic versioning may need implementation. My go-to language has recently grown to be python, so I'd probably create a python package (installed via pip) that the github runner could install and use for workflows (this might allow the github runner to use Ubuntu env).

Would it be agreeable to save the metadata to a separate metadata.inc file (similar to the RMSKIN.ini file) in skin's root dir for when the skin is actually a suite of skins with no ini file at root dir (could also apply to skins with an ini file at root)? Is the current implementation of having a metadata section per each skin file supposed to preserve original authors metadata when their work is copied into other skin packages?
User avatar
Brian
Developer
Posts: 2024
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: distribute skins via github and release via github actions

Post by Brian »

2bndy5 wrote:
May 24th, 2020, 12:14 am
Could you give a link to your work if it still exists?
I did not save my "test" repo on github, but I do still have a local copy including my workflow file. I haven't had the time to come up with a proper validating script yet. I could re-upload the repo, but I would like to get a validating script working before publishing to github.

The tricky part with rmskin validation is checking the folder structure especially for plugins and layouts. I think it will be hard to check the proper "bitness" of plugins - which might be a deal breaker for this whole thing. Documenting the RMSKIN.ini file and its options should be easy part.

2bndy5 wrote:
May 24th, 2020, 12:14 am
I appreciate the need for authentication when packaging rmskin files. Correct me if I'm wrong, but it looks like rainmeter's dialoguePackage.cpp adds a string, "RMSKIN", after the CRC data. while also creating a RMSKIN.ini at root of the archive.
Right. The "custom footer" Rainmeter appends to the zip file is basically 3 components. The size of the zip (without the footer), a null byte (for future use) and the string "RMSKIN" (null terminated). You can write that custom footer with 7 lines of powershell code (maybe less).

2bndy5 wrote:
May 24th, 2020, 12:14 am
My idea for a CLI-type program (IDK powershell script) that takes a skin (path-to-root-dir) and spits out a validating rmskin file (in same dir if output destination is not specified by an argument). Use the metadata section from the first ini file in root dir of package (sought alphabetically if the ini file to load after install is not specified by a cmd arg) to fill in "author" and "version" info. Let the skin's root dir name be the package name if not specified by a cmd arg. If the metadata section doesn't exist and is not specified via cmd args, then abort. If metadata is specified by cmd args, then I'd like the program to add/update it appropriately to skin's on-first-install-loaded ini file (of course this wouldn't apply when a layout is specified to load on first install); in this case semantic versioning may need implementation.
This was similar to my initial thought process as well for this project except the semver stuff. My test repo was fairly basic. It looked something like this:

Code: Select all

.github\workflows\release.yml
RMSKIN\RMSKIN.ini
RMSKIN\Skins\ ...
RMSKIN\Layouts\ ...
RMSKIN\Plugins\ ...
README
As you can see, the "RMSKIN" folder is in the same format structure as a unzipped rmskin. My "release.yml" workflow ran when creating a github release. So when I would create a release, the action would checkout the code, create the zip archive, apply the footer, change the .zip extension to a .rmskin, then automatically upload the "asset" to the release that was already created. It's like 54 lines (that is with logging).

In my version, it used the name of the repo as the name of the rmskin. The "tag" created during the release was used as the version of the rmskin. After I got it all done, I realized all that information was already in the RMSKIN.ini file - so I am not sure it makes much sense to make another "config" style file. You could even have the github action commit the "tag" version to the RMSKIN.ini file afterward so you didn't have to update that file before the release.

2bndy5 wrote:
May 24th, 2020, 12:14 am
My go-to language has recently grown to be python, so I'd probably create a python package (installed via pip) that the github runner could install and use for workflows (this might allow the github runner to use Ubuntu env).
The beauty of github actions is you can switch environments and use different types of scripts. While, I don't think python or Ubuntu is necessary in this case, you may be able to use them if you are more comfortable with them. I tried using a Ubuntu runner initially, but I ran into trouble uploading the rmskin to the release. It was interpreting the rmskin as a zip and erasing the custom footer. I never figured out if it was the Ubuntu runner I was using, or my test of github artifacts. I gave up on both of those.

2bndy5 wrote:
May 24th, 2020, 12:14 am
Would it be agreeable to save the metadata to a separate metadata.inc file (similar to the RMSKIN.ini file) in skin's root dir for when the skin is actually a suite of skins with no ini file at root dir (could also apply to skins with an ini file at root)? Is the current implementation of having a metadata section per each skin file supposed to preserve original authors metadata when their work is copied into other skin packages?
I really don't think this is necessary at all. Just create a proper RMSKIN.ini file, and all the metadata needed should be there. Your workflow can just "read" the data from the RMSKIN.ini file itself. I see no reason to create another "config" file that creates the RMSKIN.ini file or the use of any arguments.

...but, documenting the contents of a proper RMSKIN.ini will certainly be needed. There is only a few options, so it shouldn't be too difficult.

-Brian
User avatar
2bndy5
Posts: 19
Joined: February 25th, 2015, 2:38 am

Re: distribute skins via github and release via github actions

Post by 2bndy5 »

I think it will be hard to check the proper "bitness" of plugins - which might be a deal breaker for this whole thing.
This is a little worrisome. Googled it and found this answer on stackoverflow (it also has a link to a whitepaper about the PE file format) which means it is possible. Does Rainmeter actually perform a check on the plugins' .dll files? All I could find in Rainmeter source code was IsWin32(), and that function only returns true if the system is running on a a 32-bit architecture (I think - couldn't find where _WIN64 was defined). Also, the m_ArchivePlugins option seems to just arbitrarily copy the .dll files to the @Vault\Plugins\ folder. Am I missing a validation check about the "bitness" of plugin?

PS I played with Powershell a bit, and its so wordy it reminds me of visual basic :uhuh:. Python comes with a zipfile & zlib modules as part of the builtin std libs that can read/write rmskin files, but I'm still exploring how to write the custom footer.
User avatar
2bndy5
Posts: 19
Joined: February 25th, 2015, 2:38 am

Re: distribute skins via github and release via github actions

Post by 2bndy5 »

Proof of concept using Python3
The following script writes a validating rmskin package (tested on windows 10 w/ only 1 skin in repo). NOTE: you must first install a 3rd-party module for detirmining "bitness" of plugin dll files; do this by typing "pip install pefile" at a cmd prompt. Run this scipt either in the repo root folder or pass the repo's root folder path to the script's argument "--path" (run "script.py -h")

Code: Select all

"""
A script to run on github release action that will
attempt to assemble a validating Rainmeter skin
package for quick and easy distibution.

ideal repo structure
********************

    - root directory
        * ``Skins`` (a folder to contain all necessary skins)
        * ``RMSKIN.ini`` (list of options specific to installing the
          skin).
        * ``Layouts``(a folder that contains rainmeter layout files)
        * ``Plugins``(a folder that contains rainmeter plugins)
        * ``@Vault`` (resources folder accessible by all installed
          skins)
"""
import os
import argparse
import configparser
import zipfile
import struct
import pefile

parser = argparse.ArgumentParser(
    description="""
    A script that will attempt to assemble a 
    validating Rainmeter skin package for 
    quick and easy githuib distibution."""
)
parser.add_argument(
    "--path",
    metavar='"STR"',
    type=str,
    default=os.getcwd(),
    help="base path of a git repository. Defaults to working directory.",
)

HAS_COMPONENTS = {
    "RMSKIN.ini": False,
    "Skins": 0,
    "Layouts": 0,
    "Plugins": 0,
    "@Vault": 0,
}


def main():
    # collect cmd args
    args = parser.parse_args()
    root_path = args.path
    # truncate trailing path seperator
    if root_path.endswith(os.sep):
        root_path = root_path[:-1] 
    
    # capture the directory tree
    for dirpath, dirnames, filenames in os.walk(root_path):
        dirpath = dirpath.replace(root_path, "")
        if dirpath.endswith("Skins"):
            HAS_COMPONENTS["Skins"] = len(dirnames)
            print("Found {} possible Skin(s)".format(HAS_COMPONENTS["Skins"]))
        elif dirpath.endswith("@Vault"):
            HAS_COMPONENTS["@Vault"] = len(dirnames) + len(filenames)
            print("Found {} possible @Vault item(s)".format(HAS_COMPONENTS["@Vault"]))
        elif dirpath.endswith("Plugins"):
            HAS_COMPONENTS["Plugins"] = len(dirnames)
            print("Found {} possible Plugin(s)".format(HAS_COMPONENTS["Plugins"]))
        elif dirpath.endswith("Layouts"):
            HAS_COMPONENTS["Layouts"] = len(filenames) + len(dirnames)
            print("Found {} possible Layout(s)".format(HAS_COMPONENTS["Layouts"]))
        elif len(dirpath) == 0 and "RMSKIN.ini" in filenames:
            HAS_COMPONENTS["RMSKIN.ini"] = True
            print("Found RMSKIN.ini file")
            for d in dirnames:  # exclude hidden directories
                if d.startswith("."):
                    del d
        # set depth of search to shallow (2 folders deep)
        if len(dirpath) > 0:
            dirnames.clear()

    # quite if bad dir struct
    if not (
        HAS_COMPONENTS["Layouts"]
        or HAS_COMPONENTS["Skins"]
        or HAS_COMPONENTS["Plugins"]
        or HAS_COMPONENTS["@Vault"]
    ):
        raise RuntimeError(
            f"repository structure for {root_path} is malformed. Found no Skins,"
            " Layouts, or Plugins!"
        )
    # read options from RMSKIN.ini
    arc_name = "package_name"
    version = "auto"  # TODO use *github Action tag*
    config = configparser.ConfigParser()
    if HAS_COMPONENTS["RMSKIN.ini"]:
        config.read(root_path + os.sep + "RMSKIN.ini")
        if "rmskin" in config:
            version = config["rmskin"]["Version"]
            # config["rmskin"]["Author"] = **github username** + ?Aggregated list from discovered skins' metadata->Author fields?
            # Name should be like (<github repo name> - "Rainmeter" - "Skin")
            arc_name = config["rmskin"]["Name"]
            print(f"Found Name ({arc_name}) & Version ({version}) from RMSKIN.ini")
            load_t = config["rmskin"]["LoadType"]  # ex: "Skin"
            load = config["rmskin"]["Load"]  # ex: "Skin_Root\\skin.ini"
            if len(load_t):  # if a file set to load on-install
                # exit early if loaded file does not exist
                with open(
                    root_path + os.sep + load_t + "s" + os.sep + load, "r"
                ) as temp:
                    if temp is None:
                        raise RuntimeError("On-install loaded file does not exits.")
        else:
            raise RuntimeError("RMSKIN.ini is malformed")
    else:
        raise RuntimeError(
            f"repository structure for {root_path} is malformed. RMSKIN.ini file not found!"
        )

    # Now get to creating an archive
    compressed_size = 0
    with zipfile.ZipFile(
        arc_name + "_" + version + ".rmskin", "w", compression=zipfile.ZIP_DEFLATED,
    ) as arc_file:
        # write RMSKIN.ini first
        arc_file.write(root_path + os.sep + "RMSKIN.ini", arcname="RMSKIN.ini")
        for key in HAS_COMPONENTS:
            if key.endswith(".ini"):
                pass
            elif HAS_COMPONENTS[key]:
                for dirpath, dirnames, filenames in os.walk(root_path + os.sep + key):
                    if key.endswith("Plugins"):
                        # check bitness of plugins here & archive accordingly
                        for n in filenames:
                            if n.lower().endswith(".dll"):
                                # let plugin_name be 2nd last folder name in dll's path
                                plugin_name = dirpath.split(os.sep)
                                plugin_name = plugin_name[len(plugin_name) - 2]
                                bitness = pefile.PE(
                                    dirpath + os.sep + n,
                                    fast_load=True,  # just get headers
                                )
                                bitness.close()  # do this now to copy file safely later
                                # pylint: disable=no-member
                                if bitness.FILE_HEADER.Machine == 0x014C:
                                    # archive this 32-bit plugin
                                    arc_file.write(
                                        dirpath + os.sep + n,
                                        arcname=key
                                        + os.sep
                                        + plugin_name
                                        + os.sep
                                        + "32bit"
                                        + os.sep
                                        + n,
                                    )
                                else:
                                    # archive this 64-bit plugin
                                    arc_file.write(
                                        dirpath + os.sep + n,
                                        arcname=key
                                        + os.sep
                                        + plugin_name
                                        + os.sep
                                        + "64bit"
                                        + os.sep
                                        + n,
                                    )
                                # pylint: enable=no-member
                                del bitness
                            else:  # for misc files in plugins folders like READMEs
                                arc_file.write(
                                    dirpath + os.sep + n,
                                    arcname=dirpath.replace(root_path + os.sep, "")
                                    + os.sep
                                    + n,
                                )
                    else:
                        for n in filenames:
                            arc_file.write(
                                dirpath + os.sep + n,
                                arcname=dirpath.replace(root_path + os.sep, "")
                                + os.sep
                                + n,
                            )
        # archive assembled; closing file
    compressed_size = os.path.getsize(arc_name + "_" + version + ".rmskin")
    print(f"archive size = {compressed_size} ({hex(compressed_size)})")
    # convert size to a bytes obj & prepend to custom footer
    custom_footer = struct.pack("q", compressed_size) + b"\x00RMSKIN\x00"

    # append footer to archive
    with open(arc_name + "_" + version + ".rmskin", "a+b") as arc_file:
        print(f"appending footer: {custom_footer}")
        arc_file.write(custom_footer)


if __name__ == "__main__":
    main()

moving forward
Currently I'm not adding the header info that Rainmeter seems to, although I think that may be an artifact from the old package format. I might also use the .gitignore file to skip certain files in the process. I intend to incorporate this into a cookiecutter or simply just a github action repo, but much more testing is needed... Spread the word
User avatar
Brian
Developer
Posts: 2024
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: distribute skins via github and release via github actions

Post by Brian »

Nice work.

I haven't had time to work on my validating powershell script yet. My todo list is getting larger every day. O.O

I think I will re-publish my non-validating script and work on it when I can (or maybe someone will send a PR or something). While we haven't discussed this much yet, we may publish an official github action so users can call on our action to create their rmskin. They then can have their action upload the rmskin automatically as an asset to their newly created release.

-Brian
User avatar
2bndy5
Posts: 19
Joined: February 25th, 2015, 2:38 am

Re: distribute skins via github and release via github actions

Post by 2bndy5 »

Update
I managed to make my own action with the script. made some alteration to the script to integrate it with github actions' env vars. Also my action repo contains some dummy information that would pass a CI test. Additionally I borrowed JSMorely's ConfigActive plugin (latest stable release) to throw actual dll files at the script; I have verified the archive's plugin folder structure is coherent with any plugins' "bitness". The script also can be run locally, but it overwrites the RMSKIN.ini file (using cmd args) in the repo that it is packaging.
User avatar
Brian
Developer
Posts: 2024
Joined: November 24th, 2011, 1:42 am
Location: Utah

Re: distribute skins via github and release via github actions

Post by Brian »

Good work. I hope to do some thorough testing soon.

After some skin encoding issues with github, I finally published my non-validating script to github.
https://github.com/brianferguson/auto-rmskin-package

Hopefully, I will get time to work on it some more.

-Brian
User avatar
2bndy5
Posts: 19
Joined: February 25th, 2015, 2:38 am

Re: distribute skins via github and release via github actions

Post by 2bndy5 »

Had a look at your workflow... So clean! Aside from lack of experience, the other reason I went away from PS is because my system isn't set up run PS scripts (outside of direct CLI execution). So, I'm afraid I won't be able to help with your PS approach, though I'm still jealous because of the no-dependencies appeal.

As for my python approach, I added another dependency ("pip install pillow") to convert "RMSKIN.bmp" into a 400x60 bmp with "RGB" color space, ("RGBA" color space does not register as a valid bmp file using Rainmeter's skin packager). At the moment, the only thing my python script can't do is ignore the hidden system files/folders in a skin's root config folder. Well... it could if I switch the GitHub runner to "windows-latest", and that way I could have python read the files' & folders' attributes managed by Windows.

That said, it runs fine on an Ubuntu runner as well as locally on Windows. I'm going to call this thread solved for now. Also, I'll hold back on publishing my action repo to the GitHub Marketplace until you RM devs reach a decision about having an official Rainmeter "actions/rmskin-packager" (no-rush). This has been quite a learning experience in terms of GitHub actions for me; I'm glad I ventured.


EDIT: Just found out the hard way that Github Actions running a Docker container must use a Linux-based OS image. This limitation may be a problem in the future when distributing an action that uses a PowerShell script file. As of this writing, Brian's repo is mainly a workflow that "runs" the PS script directly without a Docker container.