Possibly hex-encoded but really I forgot the details. You can always look at the code.

@vishalnandagopal
It is not as simple as "base64" since coded length is no indication of number of characters
one letter password is
36280035e9f422e34c31b7cc455ac71026027a544f3be5fbe55781f65943bd9200000000000000000000000000000000
and eight letters is
c07adc198d89bee6bf53ea292f7c3217fe91f15bda233bc15dce25bde5e4abbb00000000000000000000000000000000

4 digits is easier
dc4eabcc07eef6f69f0d33d0c29457635139b4213a000000000000000000000000000000000000000000000000000000
but it is not the PDF file internal hash key which is longer
E72D0940E6DD12AF935A08BEA7B46B2CDA6F53F3173A9F403602F0ABD34DCEE8

so it may be a simpler cypher reflecting user input.
However comments in code says

// Store a password securely (if data encryption is provided by OS)
// or obfuscated to make search for password in memory dump less trivial.

and in part may be from MuPDF code base too as that says "padding"

When I choose to remember the PDF password option, why does it not work? I still need to paste the password each time I open the PDF file.