Lua string encoding and calling !WriteKeyValue bang

[creewick]

UPD: Hi! I'm creating a to-do list skin. I attached the package in the comments below

Long story short, I'm passing $UserInput$ from InputText measure to a Lua function
The way Lua interprete this string is not the same as original input

I had tried different combinations of encodings, here are the results:

Case 1:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-16 LE

Input string: " Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks

Case 2:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-16 LE

Input string: " Привет!"
Lua string: " Пр�"

Emojis are dislaying correctly, but cyrillic text is getting truncated

Case 3:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-8

Input string: " Привет!"
Lua string: "? Пр?"

Emojis are replaced with question marks, and cyrillic text is getting truncated

Case 4:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-8

Input string: " Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks

[balala]

Would be nice to have your skin. Don't expect us to build all your files. So please pack the config you have and upload the package here.

[creewick]

Yeah, sorry. It was a bit complicated to isolate a single skin from the suite. Here it is!

In any skin variant, except Small.ini, you can click on the background to enter a new item. But right after I hit "Enter", it replaces emoji with a question mark (if Reminders.lua encoding is UTF-8) or truncates part of the text (if Reminders.lua encoding is UTF-16 LE)

[Yincognito]

UPD: Hi! I'm creating a to-do list skin. I attached the package in the comments below

Long story short, I'm passing $UserInput$ from InputText measure to a Lua function
The way Lua interprete this string is not the same as original input

I had tried different combinations of encodings, here are the results:

Case 1:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-16 LE

Input string: " Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks

Case 2:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-16 LE

Input string: " Привет!"
Lua string: " Пр�"

Emojis are dislaying correctly, but cyrillic text is getting truncated

Case 3:
Scripts\Reminders.lua - UTF-16 LE
Variables\Reminders.inc - UTF-8

Input string: " Привет!"
Lua string: "? Пр?"

Emojis are replaced with question marks, and cyrillic text is getting truncated

Case 4:
Scripts\Reminders.lua - UTF-8
Variables\Reminders.inc - UTF-8

Input string: " Привет!"
Lua string: "? Привет!"

String is not truncated, but all emojis are replaced with question marks

I believe this (or at least case 2 and the truncated text above) is related to the bug reported by Jax here. You might want to try sending the emojis and the string separately to Lua (emojis as escaped character variables, if possible) and do the needed conversion back to emojis in Lua, assuming the encoding properly displays the text. If this is even possible, that is.

Or, let Rainmeter properly set a variable with both the emoji and the text, then use SKIN:ReplaceVariables('#VariableHoldingUserInput#') in Lua.

That being said, you should know that character variables above xFFFE are not supported by Rainmeter. Some "emojis" are beyong that limit.

EDIT: For example, setting the Command1 option in [NewItemInput] from Medium.ini to:

Code: Select all

Command1=[!SetVariable NewItemInput "$UserInput$"][!Update][&Lua:addReminder('NewItemInput')][!Refresh]

and the addReminder function in Reminders.lua to:

Code: Select all

function addReminder(textvar)
    local count = getRemindersCount()
    SKIN:Bang('!WriteKeyValue', 'Variables', 'Item'..(count+1), SKIN:ReplaceVariables('#'..textvar..'#'), '#@#Variables/Reminders.inc')

    return 0
end

will properly display, after converting the Lua file encoding to UCS2-LE BOM in Notepad++ (can't remember if it's UTF-8 or UTF-16 LE in the other notation, and won't bother to remember):

Reminders.jpg

As you can see, the + is properly displayed. Don't know about cyrillic text though, you'd have to try it for yourself, but I reckon it will work as well since the variable is taken from Rainmeter already set up properly.

[creewick]

EDIT: For example, setting the Command1 option in [NewItemInput] from Medium.ini to:

Code: Select all

Command1=[!SetVariable NewItemInput "$UserInput$"][!Update][&Lua:addReminder('NewItemInput')][!Refresh]

and the addReminder function in Reminders.lua to:

Code: Select all

function addReminder(textvar)
    local count = getRemindersCount()
    SKIN:Bang('!WriteKeyValue', 'Variables', 'Item'..(count+1), SKIN:ReplaceVariables('#'..textvar..'#'), '#@#Variables/Reminders.inc')

    return 0
end

It works great, emojis and cyrillics are displaying correctly with this code and both files encoded as UTF-16 LE
Thank you so much!

[balala]

and both files encoded as UTF-16 LE

Actually the file in which you're writing with lua should be encoded as UTF-8: https://forum.rainmeter.net/viewtopic.php?f=5&t=34523&p=170918#p170918

[creewick]

Actually the file in which you're writing with lua should be encoded as UTF-8: https://forum.rainmeter.net/viewtopic.php?f=5&t=34523&p=170918#p170918

Maybe I'm wrong, but on the link you shared, it says the opposite

Lua has challenges with Unicode, but not impossible by any means. Skin as UTF-16, .Lua file as UTF-16, any external file you read or write WITH the Lua as UTF-8.

I tried to use Yincognito's code with Lua file encoded as UTF-8, and it gives me the same question marks

Input string: " Reminder"
Lua string: "? Reminder"

UPD: Sorry, I realised that you mean the Variable file. Give me a sec, I'll try it

[creewick]

Actually the file in which you're writing with lua should be encoded as UTF-8: https://forum.rainmeter.net/viewtopic.php?f=5&t=34523&p=170918#p170918

Cyrillic characters are not displaying correctly in that case.
I think it happens because the file I write to using Lua is the same file Rainmeter uses as @include file
But as UTF-16 it works fine

[balala]

Cyrillic characters are not displaying correctly in that case.
I think it happens because the file I write to using Lua is the same file Rainmeter uses as @include file
But as UTF-16 it works fine

Yep, right. I realized just now that as you did this in fact you write to the external file (@Resources\Variables\Reminders.inc) through a !WriteKeyValue bang, even if it is called from a lua script. So, you're writing it from a skin, not from a lua script. Accordingly the file in which you're writing has to be indeed encoded az UTF-16 LE.
Sorry, my bad...

[Yincognito]

It works great, emojis and cyrillics are displaying correctly with this code and both files encoded as UTF-16 LE
Thank you so much!

No problem, glad to help - let those emojis and cyrillics have a party there!