Question:
How should I encode passwords with v4 security handler if I am using Unicode strings?
Answer:
v4 security handler of PDF requires the password string to be encoded to PDFDocEncoding. If the password string contains characters that cannot be encoded to PDFDocEncoding, then those characters must first be normalized to codepage encoding (meaning the current codepage the OS uses), then convert the codepage encoding to PDFDocEncoding.
In Windows, you may choose to compile your app in Unicode rather than MCS. The problem when compiling your app in Unicode is that all user input will be given to you by Windows in UTF-16 encoding. And as your experiments have proven, there is no ideal way to obtain the user’s input locale. With MBCS, however, Windows will map the source codepage encoding to the current system encoding so no conversion is necessary. For example, with MBCS, when your user enters the characters “тест” (regardless of the source codepage), if your current system encoding is using CP1252, then you will get 0xF2E5F1F2 as the coded characters for the original input. You can then use this to pass to UString and specify winansi encoding.
If your application is compiled with Unicode, however, your best bet is to find out the user input locale by using GetKeyboardLayout API (https://msdn.microsoft.com/en-us/library/windows/desktop/ms646296.aspx). Get the user input locale from that API, then convert Unicode with WideCharToMultiByte API (https://msdn.microsoft.com/en-us/library/windows/desktop/dd374130.aspx). There are, however, some pitfalls in using this method (e.g. user can change keyboard layout at any time leading to the app obtaining the incorrect source input locale).