It is currently December 3rd, 2022, 1:17 pm

Lua string encoding and calling !WriteKeyValue bang

Discuss the use of Lua in Script measures.
User avatar
creewick
Posts: 37
Joined: March 4th, 2022, 2:43 pm
Location: Yekaterinburg, Russia

Lua string encoding and calling !WriteKeyValue bang

Post by creewick »

UPD: Hi! I'm creating a to-do list skin. I attached the package in the comments below

Long story short, I'm passing $UserInput$ from InputText measure to a Lua function
The way Lua interprete this string is not the same as original input

I had tried different combinations of encodings, here are the results:

Case 1:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-16 LE

Input string: "✅ Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks

Case 2:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-16 LE

Input string: "✅ Привет!"
Lua string: "✅ Пр�"

Emojis are dislaying correctly, but cyrillic text is getting truncated

Case 3:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-8

Input string: "✅ Привет!"
Lua string: "? Пр?"

Emojis are replaced with question marks, and cyrillic text is getting truncated

Case 4:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-8

Input string: "✅ Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks
Last edited by creewick on March 22nd, 2022, 5:35 pm, edited 2 times in total.
User avatar
balala
Rainmeter Sage
Posts: 14758
Joined: October 11th, 2010, 6:27 pm
Location: Gheorgheni, Romania

Re: Lua string encoding and calling !WriteKeyValue bang

Post by balala »

Would be nice to have your skin. Don't expect us to build all your files. So please pack the config you have and upload the package here.
User avatar
creewick
Posts: 37
Joined: March 4th, 2022, 2:43 pm
Location: Yekaterinburg, Russia

Re: Lua string encoding and calling !WriteKeyValue bang

Post by creewick »

Yeah, sorry. It was a bit complicated to isolate a single skin from the suite. Here it is!

In any skin variant, except Small.ini, you can click on the background to enter a new item. But right after I hit "Enter", it replaces emoji with a question mark (if Reminders.lua encoding is UTF-8) or truncates part of the text (if Reminders.lua encoding is UTF-16 LE)

ImageImage
You do not have the required permissions to view the files attached to this post.
User avatar
Yincognito
Rainmeter Sage
Posts: 4817
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: Lua string encoding and calling !WriteKeyValue bang

Post by Yincognito »

creewick wrote: March 22nd, 2022, 4:08 pm UPD: Hi! I'm creating a to-do list skin. I attached the package in the comments below

Long story short, I'm passing $UserInput$ from InputText measure to a Lua function
The way Lua interprete this string is not the same as original input

I had tried different combinations of encodings, here are the results:

Case 1:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-16 LE

Input string: "✅ Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks

Case 2:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-16 LE

Input string: "✅ Привет!"
Lua string: "✅ Пр�"

Emojis are dislaying correctly, but cyrillic text is getting truncated

Case 3:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-8

Input string: "✅ Привет!"
Lua string: "? Пр?"

Emojis are replaced with question marks, and cyrillic text is getting truncated

Case 4:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-8

Input string: "✅ Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks
I believe this (or at least case 2 and the truncated text above) is related to the bug reported by Jax here. You might want to try sending the emojis and the string separately to Lua (emojis as escaped character variables, if possible) and do the needed conversion back to emojis in Lua, assuming the encoding properly displays the text. If this is even possible, that is.

Or, let Rainmeter properly set a variable with both the emoji and the text, then use SKIN:ReplaceVariables('#VariableHoldingUserInput#') in Lua.

That being said, you should know that character variables above xFFFE are not supported by Rainmeter. Some "emojis" are beyong that limit.

EDIT: For example, setting the Command1 option in [NewItemInput] from Medium.ini to:

Code: Select all

Command1=[!SetVariable NewItemInput "$UserInput$"][!Update][&Lua:addReminder('NewItemInput')][!Refresh]
and the addReminder function in Reminders.lua to:

Code: Select all

function addReminder(textvar)
    local count = getRemindersCount()
    SKIN:Bang('!WriteKeyValue', 'Variables', 'Item'..(count+1), SKIN:ReplaceVariables('#'..textvar..'#'), '#@#Variables/Reminders.inc')

    return 0
end
will properly display, after converting the Lua file encoding to UCS2-LE BOM in Notepad++ (can't remember if it's UTF-8 or UTF-16 LE in the other notation, and won't bother to remember):
Reminders.jpg
As you can see, the + is properly displayed. Don't know about cyrillic text though, you'd have to try it for yourself, but I reckon it will work as well since the variable is taken from Rainmeter already set up properly.
You do not have the required permissions to view the files attached to this post.
User avatar
creewick
Posts: 37
Joined: March 4th, 2022, 2:43 pm
Location: Yekaterinburg, Russia

Re: Lua string encoding and calling !WriteKeyValue bang

Post by creewick »

Yincognito wrote: March 22nd, 2022, 6:10 pm EDIT: For example, setting the Command1 option in [NewItemInput] from Medium.ini to:

Code: Select all

Command1=[!SetVariable NewItemInput "$UserInput$"][!Update][&Lua:addReminder('NewItemInput')][!Refresh]
and the addReminder function in Reminders.lua to:

Code: Select all

function addReminder(textvar)
    local count = getRemindersCount()
    SKIN:Bang('!WriteKeyValue', 'Variables', 'Item'..(count+1), SKIN:ReplaceVariables('#'..textvar..'#'), '#@#Variables/Reminders.inc')

    return 0
end
It works great, emojis and cyrillics are displaying correctly with this code and both files encoded as UTF-16 LE
Thank you so much! :thumbup:
User avatar
balala
Rainmeter Sage
Posts: 14758
Joined: October 11th, 2010, 6:27 pm
Location: Gheorgheni, Romania

Re: Lua string encoding and calling !WriteKeyValue bang

Post by balala »

creewick wrote: March 22nd, 2022, 8:04 pm and both files encoded as UTF-16 LE
Actually the file in which you're writing with lua should be encoded as UTF-8: https://forum.rainmeter.net/viewtopic.php?f=5&t=34523&p=170918#p170918
User avatar
creewick
Posts: 37
Joined: March 4th, 2022, 2:43 pm
Location: Yekaterinburg, Russia

Re: Lua string encoding and calling !WriteKeyValue bang

Post by creewick »

balala wrote: March 22nd, 2022, 8:38 pm Actually the file in which you're writing with lua should be encoded as UTF-8: https://forum.rainmeter.net/viewtopic.php?f=5&t=34523&p=170918#p170918
Maybe I'm wrong, but on the link you shared, it says the opposite :confused:
jsmorley wrote: January 27th, 2020, 3:03 am Lua has challenges with Unicode, but not impossible by any means. Skin as UTF-16, .Lua file as UTF-16, any external file you read or write WITH the Lua as UTF-8.
I tried to use Yincognito's code with Lua file encoded as UTF-8, and it gives me the same question marks

Input string: "➕ Reminder"
Lua string: "? Reminder"

UPD: Sorry, I realised that you mean the Variable file. Give me a sec, I'll try it
User avatar
creewick
Posts: 37
Joined: March 4th, 2022, 2:43 pm
Location: Yekaterinburg, Russia

Re: Lua string encoding and calling !WriteKeyValue bang

Post by creewick »

balala wrote: March 22nd, 2022, 8:38 pm Actually the file in which you're writing with lua should be encoded as UTF-8: https://forum.rainmeter.net/viewtopic.php?f=5&t=34523&p=170918#p170918
Image

Cyrillic characters are not displaying correctly in that case.
I think it happens because the file I write to using Lua is the same file Rainmeter uses as @include file
But as UTF-16 it works fine
User avatar
balala
Rainmeter Sage
Posts: 14758
Joined: October 11th, 2010, 6:27 pm
Location: Gheorgheni, Romania

Re: Lua string encoding and calling !WriteKeyValue bang

Post by balala »

creewick wrote: March 22nd, 2022, 8:51 pm Cyrillic characters are not displaying correctly in that case.
I think it happens because the file I write to using Lua is the same file Rainmeter uses as @include file
But as UTF-16 it works fine
Yep, right. I realized just now that as you did this in fact you write to the external file (@Resources\Variables\Reminders.inc) through a !WriteKeyValue bang, even if it is called from a lua script. So, you're writing it from a skin, not from a lua script. Accordingly the file in which you're writing has to be indeed encoded az UTF-16 LE.
Sorry, my bad...
User avatar
Yincognito
Rainmeter Sage
Posts: 4817
Joined: February 27th, 2015, 2:38 pm
Location: Terra Yincognita

Re: Lua string encoding and calling !WriteKeyValue bang

Post by Yincognito »

creewick wrote: March 22nd, 2022, 8:04 pm It works great, emojis and cyrillics are displaying correctly with this code and both files encoded as UTF-16 LE
Thank you so much! :thumbup:
No problem, glad to help - let those emojis and cyrillics have a party there! 8-)