Custom security handling

Q: We have our own MySecurityHandler which handles documents with
custom security. This is an extention to the standard security but with
filtername 'MY', so it can't be opened by Acrobat.
When our own documents are loaded, I detect this because the security
filter has our name, and I provide a fixed password to open the
document. This is the PDF user password so I don't have permissions to
change security settings. Now, if I want to change the security
settings, I want my application to ask for the owner password. How
would I do that? I handle GetAuthorizationData(e_doc_secure) but it's
never called by PDFNet SDK. Should I call this myself if
Authorize(e_doc_secure) fails?

You could implement a dialog where the user can input the owner
password. Afterwards you can call sec_hdlr.Authorize() which will in
turn call GetAuthorizationData() in the derived security handler
(similar to code in PDFNet/Samples/EncTest).

Q: Should I convert my unicode password to ANSI characters or to UTF8
encoded characters to be compatible with Acrobat?

A: Since you are implementing the custom security handler you can do it
either way. You can also pass UTF16 encoded strings.