[kamailio/kamailio] TLS certificate decode error / ee key to small with tls_threads_mode = 1 (Issue #3764)
<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-users.lists.kamailio....
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-dev.lists.kamailio.or...
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
Note that an issue report may be closed automatically after about 2 months if there is no interest from developers or community users on pursuing it, being considered expired. In such case, it can be reopened by writing a comment that includes the token `/notexpired`. About two weeks before considered expired, the issue is marked with the label `stale`, trying to notify the submitter and everyone else that might be interested in it. To remove the label `stale`, write a comment that includes the token `/notstale`. Also, any comment postpone the `expire` timeline, being considered that there is interest in pursuing the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description
While trying latest kamailio 5.7 branch, when tls_threads_mode is set to 1, it fails to load self signed certificates. Setting tls_threads_mode to 0 works as expected. Certificates are self signed for a local test env, generated with openssl 3.x.
### Troubleshooting
The issue is very similar to https://github.com/kamailio/kamailio/issues/3737 but in my case the openssl config seems correct, and happens only enabling the tls_threads_mode
#### Reproduction
Certs have been generated with `openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -out server.pem -keyout server.key`
[server.pem.txt](https://github.com/kamailio/kamailio/files/14384611/server.pem.txt) [server.key.txt](https://github.com/kamailio/kamailio/files/14384612/server.key.txt)
(these are self signed cert for testing, nothing that cannot be shared)
My tls.cfg is very simple: ``` [server:default] method = TLSv1.2+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/server.key certificate = /etc/kamailio/server.pem
[client:default] method = TLSv1.2+ verify_certificate = no require_certificate = no ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` 1(35) NOTICE: tls [tls_domain.c:1168]: ksr_tls_fix_domain(): registered server_name callback handler for socket [:0], server_name='<default>' ... 1(35) ERROR: tls [tls_domain.c:590]: load_cert(): TLSs<default>: Unable to load certificate file '/etc/kamailio/server.pem' 1(35) ERROR: tls [tls_util.h:49]: tls_err_ret(): load_cert:error:03000072:digital envelope routines::decode error (sni: unknown) 1(35) ERROR: tls [tls_util.h:49]: tls_err_ret(): load_cert:error:0A00018F:SSL routines::ee key too small (sni: unknown) 1(35) ERROR: <core> [core/sr_module.c:913]: init_mod_child(): error while initializing module tls (/usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so) ```
### Possible Solutions
Don't use tls_threads_mode for now.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.7.4 (x86_64/linux) a0dfb8 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: a0dfb8 compiled with gcc 11.4.0 ```
Actually this is built from 5.7 branch, on commit a0dfb8cbdf4282040351e9dc014d9ef13e0e77fd
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
Containerized Ubunu jammy, updated as of today.
After further digging in my setup, as soon as I disabled the mqtt module everything works.
So there's maybe someting in the mqtt module that's not yet adapted to the new tls setup?
After further digging in my setup, as soon as I disabled the mqtt module everything works.
So there's maybe someting in the mqtt module that's not yet adapted to the new tls setup?
That's good info - can you put the mqtt config here - maybe it is using SSL.
Yes, mqtt is using TLS, the config is pretty trivial
``` modparam("mqtt", "host", MQTT_HOST) modparam("mqtt", "port", 8883) modparam("mqtt", "keepalive", 5) modparam("mqtt", "id", MQTT_NODE_NAME) modparam("mqtt", "username", MQTT_USER) modparam("mqtt", "password", MQTT_PASS) modparam("mqtt", "will_topic", "kamailio123") modparam("mqtt", "will", "gone") modparam("mqtt", "verify_certificate", 0) // this will enable TLS modparam("mqtt", "ca_file", "/ssl/ca.crt") ```
Does it make an immediate connection to the broker or does that only happen during operations - i.e., after startup but before handling traffic is there a connection to 8883?
it makes an immediate connection to the broker on startup
it makes an immediate connection to the broker on startup
Can you run kamailio under gdb with tls_thread_mode=1 with the following script You will need to start kamailio first as thread 1 is not started yet something like
gdb --args /usr/local/sbin/kamailo ................ break main r
then use the following script:
break CRYPTO_THREAD_set_local thread 1 commands bt 16 cont end
...then continue
I'll try, since my setup is dockerized I'll need to play a bit with it. Will report as soon as I'm able to do it, should not be too hard.
I can't reproduce any error with, can you try this too. ``` #!define MQTT_HOST "test.mosquitto.org" loadmodule "mqtt.so" modparam("mqtt", "host", MQTT_HOST) modparam("mqtt", "port", 8883) modparam("mqtt", "keepalive", 5) #modparam("mqtt", "id", MQTT_NODE_NAME) #modparam("mqtt", "username", MQTT_USER) #modparam("mqtt", "password", MQTT_PASS) modparam("mqtt", "will_topic", "kamailio123") modparam("mqtt", "will", "gone") modparam("mqtt", "verify_certificate", 0) // download crt file from https://test.mosquitto.org/ssl/mosquitto.org.crt modparam("mqtt", "ca_file", "/etc/kamailio/certs/mosquitto.org.crt") ```
I've tried and gdb stops at ``` Thread 1 "kamailio" hit Breakpoint 2, 0x00007ff30a1c2a70 in CRYPTO_THREAD_set_local () from /lib/x86_64-linux-gnu/libcrypto.so.3 (gdb) bt ```
I can't reproduce any error with the config below.
well, I have various other modules loaded, but only disabling mqtt makes it work, that's why I pointed at it.
Can you print the backtrace when it stops?
Yep, sorry
``` Thread 1 "kamailio" hit Breakpoint 2, 0x00007fb1301c2a70 in CRYPTO_THREAD_set_local () from /lib/x86_64-linux-gnu/libcrypto.so.3 #0 0x00007fb1301c2a70 in CRYPTO_THREAD_set_local () from /lib/x86_64-linux-gnu/libcrypto.so.3 #1 0x00007fb1301bfad3 in OPENSSL_thread_stop () from /lib/x86_64-linux-gnu/libcrypto.so.3 #2 0x00007fb1301bfb43 in OPENSSL_cleanup () from /lib/x86_64-linux-gnu/libcrypto.so.3 #3 0x00005638fca17d48 in destroy_tls () at core/tls_hooks.c:75 #4 cleanup (show_status=1) at /usr/local/src/pkg/src/main.c:595 #5 0x00005638fce45a8d in shutdown_children.constprop.0 (show_status=show_status@entry=1, sig=15) at /usr/local/src/pkg/src/main.c:722 #6 0x00005638fca12cd5 in handle_sigs () at /usr/local/src/pkg/src/main.c:822 #7 0x00005638fca1baa4 in main_loop () at /usr/local/src/pkg/src/main.c:1989 #8 0x00005638fca0cffc in main (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/pkg/src/main.c:3213 0(13170) INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized 0(13170) DEBUG: <core> [core/mem/shm.c:287]: shm_destroy_manager(): destroying memory manager: q_malloc 0(13170) DEBUG: <core> [core/mem/q_malloc.c:1278]: qm_shm_lock_destroy(): destroying the shared memory lock 0(13170) DEBUG: <core> [core/mem/pkg.c:95]: pkg_destroy_manager(): destroying memory manager: q_malloc [Inferior 1 (process 13170) exited normally] Thread-specific breakpoint 2 deleted - thread 1 no longer in the thread list. (gdb) ```
Not much information there... Take 2 - what modules do you load?
yes, that break reports only shutdown routines.
Well actually a lot of modules: ``` loadmodule "tls.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "db_postgres.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "dialog.so" loadmodule "maxfwd.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "textops.so" loadmodule "textopsx.so" loadmodule "siputils.so" loadmodule "dmq.so" loadmodule "htable.so" loadmodule "dispatcher.so" loadmodule "ipops.so" loadmodule "kex.so" loadmodule "auth.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "nathelper.so" loadmodule "http_async_client.so" loadmodule "ruxc.so" loadmodule "ctl.so" loadmodule "json.so" loadmodule "jansson.so" loadmodule "presence.so" loadmodule "presence_dialoginfo.so" loadmodule "presence_reginfo.so" loadmodule "pua.so" loadmodule "pua_reginfo.so" loadmodule "pua_dialoginfo.so" loadmodule "debugger.so" loadmodule "sqlops.so" loadmodule "sdpops.so" loadmodule "rtpengine.so" loadmodule "corex.so" loadmodule "timer.so" loadmodule "rtimer.so" loadmodule "cfgutils.so" loadmodule "mqtt.so" loadmodule "xhttp.so" loadmodule "jsonrpcs.so" loadmodule "uac.so" ```
Ok, further progress: my setup has `enable_tls=yes` in the config. removing it (which disables SIP TLS) kamailio works with mqtt too. So seems something related to concurrent startup of mqtt tls and core tls?
while disabling TLS on mqtt (let it go in plain) and keeping enable_tls=yes works too
further infos: launching with "strace -ff kamailio" which slows down things... it works. so seems a race condition?
last for today: switching to `tlsa` works, too. Unfortunately I'm not expert enough to perform further analysys. Let me know if I can help in some way.
yes, that break reports only shutdown routines.
Well actually a lot of modules:
loadmodule "tls.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "db_postgres.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "dialog.so" loadmodule "maxfwd.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "textops.so" loadmodule "textopsx.so" loadmodule "siputils.so" loadmodule "dmq.so" loadmodule "htable.so" loadmodule "dispatcher.so" loadmodule "ipops.so" loadmodule "kex.so" loadmodule "auth.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "nathelper.so" loadmodule "http_async_client.so" loadmodule "ruxc.so" loadmodule "ctl.so" loadmodule "json.so" loadmodule "jansson.so" loadmodule "presence.so" loadmodule "presence_dialoginfo.so" loadmodule "presence_reginfo.so" loadmodule "pua.so" loadmodule "pua_reginfo.so" loadmodule "pua_dialoginfo.so" loadmodule "debugger.so" loadmodule "sqlops.so" loadmodule "sdpops.so" loadmodule "rtpengine.so" loadmodule "corex.so" loadmodule "timer.so" loadmodule "rtimer.so" loadmodule "cfgutils.so" loadmodule "mqtt.so" loadmodule "xhttp.so" loadmodule "jsonrpcs.so" loadmodule "uac.so"
Can you attach your config too
@space88man seems that fixes in current 5.7 branch after #3765 makes it work, so I think this can be closed.
Closed #3764 as completed.
-
Matteo -
space88man