Thanks for the comments - I have replaced malloc/free in the mapping utilities with pkg_malloc()/pkg_free(). Re: "I did not fully understand why you need this here, maybe you can
elaborate a bit on the requirements of the HSM child_init."

Background: For soft keys, we initialize the SSL_CTX d->ctx[i] in PROC_INIT, then fork(). All SSL_CTX objects are now completely initialized and can be used as-is by SSL* objects.

For the HSM case, the private key handle from ENGINE_load_private_key() is a proxy for the private key in the HSM. This proxy object is not guaranteed to be valid in a different process whether by fork() or shared memory.

During mod_child() we also cannot load the private key into SSL_CTX *d->ctx[i], as the next child will simply overwrite the value since these are in shared memory. In my first implementation I forgot about shared memory so the keys are what the final child loaded. These keys would only work for the last child and the handle was invalid when other children tried to use them.

In my current implementation, I leave the SSL_CTXs as keyless. Instead loading the private key into a local set, keyed by the SSL_CTX pointer. Therefore SSL_CTX for HSM domains are keyless and incomplete for SSL_accept(). At runtime, just before

SSL_accept(ssl)

I retrieve the SSL_CTX* from ssl, and check if it is indeed keyless, i.e, it exists in the local key set. Then we retrieve the HSM key in just-in-time mode and load into the SSL* ssl object.

Summary:
Soft key: d->ctx[i] fully initialized with private key from PEM file

HSM key: d->ctx[i] keyless, the privatekey is stored in a local set: as hash table
{ domain0->ctx[0]: EVP_PKEY*
domain0->ctx[1]: EVP_PKEY*
....
domain1->ctx[0]: EVP_PKEY*
domain1->ctx[2]: EVP_PKEY*
...
}.

Notes:

1. Most engines that deal with HSM private keys are wrappers around a vendor or OpenSC PKCS 11 library.
2. AWS CloudHSM engine (libgem.so) a wrapper around SafeNet Luna HSM libCryptoki2_64.so actually produces private keys that work even in a different process. If the last child/master loads its private key into d->ctx[i] all the other children can use this SSL_CTX. This type of behaviour is due to their special care in implementaion and not mandated.
3. OpenSC/libp11: this engine can wrap any PKCS11 library into an engine. When wrapping libCryptoki2_64.so its private keys didn't work in another process. That is why I chose to use a local set.
4. Interestingly the NGINX developers refused to accept HSM private key handling to the child. They state that is the role of the PKCS 11 library to make sure its objects are valid across a fork(). Sadly, this principle does not accord with reality.
5. Although HAProxy has engine support, they do not use engine private keys yet, so have not addressed this issue.
6. GNUTLS recommends that all PKCS 11 objects be used only in one process; i.e don't try to leak handles across fork() or shared memory.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.