<!--
Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment).
-->
### Description
I have a cluster of 2 Kamailios nodes on debian 9 + Kamailio 5.5.4.
We have installed a 3rd server on debian 11 + Kamailio 5.6.1 but it keeps crashing due to an issue with DMQ USRLOC
### Troubleshooting
#### Reproduction
We don't know how to reproduce but this is currently happening constantly on our new server
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
bt full
info locals
list
If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
-->
Kamailio generates a core on each segfault:
```
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7507.csbc03.example.com.1663227725
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7576.csbc03.example.com.1663227733
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7642.csbc03.example.com.1663227739
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7708.csbc03.example.com.1663227746
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7772.csbc03.example.com.1663227753
```
But they seem to be empty?
```
root@csbc03:/var/tmp# gdb core.kamailio.7772.csbc03.example.com.1663227753
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"0x7ffec5273ce0s": not in executable format: file format not recognized
(gdb) bt full
No stack.
(gdb) quit
root@csbc03:/var/tmp#
```
Very similar output for any of the other core files when In run `bt full`.
I confirmed debug symbols were installed:
```
root@csbc03:/var/tmp# dpkg -l | grep kam | grep dbg
ii kamailio-dbg:amd64 5.6.1+bpo11 amd64 very fast and configurable SIP server [debug symbols]
root@csbc03:/var/tmp#
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
/var/log/syslog
```
[ 501.251476] kamailio[7772]: segfault at f8 ip 00007f86ded9bf72 sp 00007ffd892eb7c0 error 4 in dmq_usrloc.so[7f86ded89000+17000]
[ 501.251496] Code: 40 38 01 d0 89 05 12 95 00 00 48 8b 05 cf 8f 00 00 8b 00 83 f8 01 0f 85 80 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 <8b> 90 f8 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 48 8b
```
/var/log/kamailio/kamailio.lorg
```
Sep 15 02:42:34 csbc03 csbc[7792]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 59
Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:774]: handle_sigs(): child process 7772 exited by a signal 11
Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:777]: handle_sigs(): core was generated
```
#### SIP Traffic
<!--
If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
(paste your sip traffic here)
```
### Possible Solutions
<!--
If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix.
-->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
root@csbc03:/var/tmp# kamailio -v
version: kamailio 5.6.1 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 10.2.1
root@csbc03:/var/tmp#
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
Debian 11 Bullseye
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3238
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3238(a)github.com>
Hello,
I am planning to release a new version from branch 5.4, respectively
5.4.9, sometime soon, maybe on Friday or Monday.
Unless a major regression comes out with this release, this is going to
be the last official packaged version from branch 5.4. The latest two
stable branches 5.5 and 5.6 will be maintained further.
As usual, if anyone is aware of issues not yet reported to bug tracker
or missing backports, report them in order to try to get the fixes in
this release.
Cheers,
Daniel
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
Hi,
The "--cfg-print" CLI option is not documented in Kamailio 5.6 Cookbook.
I'm sorry but I don't know how to add it by myself...
Leonid
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3239
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3239(a)github.com>
Module: kamailio
Branch: 5.6
Commit: 2b1a5ffb75566fb9411b23be23b263986b3eafaa
URL: https://github.com/kamailio/kamailio/commit/2b1a5ffb75566fb9411b23be23b2639…
Author: Daniel-Constantin Mierla <miconda(a)gmail.com>
Committer: Daniel-Constantin Mierla <miconda(a)gmail.com>
Date: 2022-09-15T10:28:03+02:00
dmq: docs - note about using same Kamailio version
(cherry picked from commit 1fd789246cc7c58237b9a903a198521cd5b3494d)
---
Modified: src/modules/dmq/doc/dmq_admin.xml
---
Diff: https://github.com/kamailio/kamailio/commit/2b1a5ffb75566fb9411b23be23b2639…
Patch: https://github.com/kamailio/kamailio/commit/2b1a5ffb75566fb9411b23be23b2639…
---
diff --git a/src/modules/dmq/doc/dmq_admin.xml b/src/modules/dmq/doc/dmq_admin.xml
index 661fd4b8471..7a2c933c62f 100644
--- a/src/modules/dmq/doc/dmq_admin.xml
+++ b/src/modules/dmq/doc/dmq_admin.xml
@@ -47,6 +47,11 @@
Nodes that are not responding to KDMQ reguests are removed the from local
list of active nodes, no longer being considered to be part of the KDMQ cluster.
</para>
+ <para>
+ IMPORTANT: DMQ must be used only between Kamailio instances having same
+ major version. Internal structures can be incompatible between different
+ major versions and can lead to crashes or unexpected behaviour.
+ </para>
<example>
<title>KDMQ Request Example</title>
<para>
Module: kamailio
Branch: 5.6
Commit: 961b62fc1ced8e7015052c97bf288a560d50e7f7
URL: https://github.com/kamailio/kamailio/commit/961b62fc1ced8e7015052c97bf288a5…
Author: Kamailio Dev <kamailio.dev(a)kamailio.org>
Committer: Kamailio Dev <kamailio.dev(a)kamailio.org>
Date: 2022-09-15T10:46:34+02:00
modules: readme files regenerated - modules ... [skip ci]
---
Modified: src/modules/dmq/README
---
Diff: https://github.com/kamailio/kamailio/commit/961b62fc1ced8e7015052c97bf288a5…
Patch: https://github.com/kamailio/kamailio/commit/961b62fc1ced8e7015052c97bf288a5…
---
diff --git a/src/modules/dmq/README b/src/modules/dmq/README
index 10f0f7f80a..b79a6c2759 100644
--- a/src/modules/dmq/README
+++ b/src/modules/dmq/README
@@ -28,6 +28,9 @@ Charles Chance
1. Admin Guide
1. Overview
+
+ 1.1. KDMQ Request
+
2. Dependencies
2.1. Kamailio Modules
@@ -99,6 +102,9 @@ Chapter 1. Admin Guide
Table of Contents
1. Overview
+
+ 1.1. KDMQ Request
+
2. Dependencies
2.1. Kamailio Modules
@@ -131,6 +137,8 @@ Chapter 1. Admin Guide
1. Overview
+ 1.1. KDMQ Request
+
The DMQ module implements a distributed message queue on top of
Kamailio in order to facilitate data propagation and replication
between multiple instances, referred as "nodes" (or "peers").
@@ -166,11 +174,16 @@ Chapter 1. Admin Guide
different major versions and can lead to crashes or unexpected
behaviour.
- Example 1.1. KDMQ Request Example
+1.1. KDMQ Request
+
+ KDMQ messages can have different format and content for R-URI username,
+ headers and body, being specific to each component that leverages DMQ
+ to replicate data, such as htable, dialog or usrloc modules.
- This request is generated for DMQ peers availability notifications.
- Other messages can be produced by various modules, with different R-URI
- username and body content, such as htable, dialog or usrloc modules.
+ Next request is generated by DMQ module for peers availability
+ notifications.
+
+ Example 1.1. KDMQ Request Example
...
KDMQ sip:notification_peer@192.168.40.15:5090 SIP/2.0
Module: kamailio
Branch: master
Commit: 95e0fe85ff9149c80249e848a437deb5dae2e463
URL: https://github.com/kamailio/kamailio/commit/95e0fe85ff9149c80249e848a437deb…
Author: Kamailio Dev <kamailio.dev(a)kamailio.org>
Committer: Kamailio Dev <kamailio.dev(a)kamailio.org>
Date: 2022-09-15T10:46:17+02:00
modules: readme files regenerated - dmq ... [skip ci]
---
Modified: src/modules/dmq/README
---
Diff: https://github.com/kamailio/kamailio/commit/95e0fe85ff9149c80249e848a437deb…
Patch: https://github.com/kamailio/kamailio/commit/95e0fe85ff9149c80249e848a437deb…
---
diff --git a/src/modules/dmq/README b/src/modules/dmq/README
index 10f0f7f80a..b79a6c2759 100644
--- a/src/modules/dmq/README
+++ b/src/modules/dmq/README
@@ -28,6 +28,9 @@ Charles Chance
1. Admin Guide
1. Overview
+
+ 1.1. KDMQ Request
+
2. Dependencies
2.1. Kamailio Modules
@@ -99,6 +102,9 @@ Chapter 1. Admin Guide
Table of Contents
1. Overview
+
+ 1.1. KDMQ Request
+
2. Dependencies
2.1. Kamailio Modules
@@ -131,6 +137,8 @@ Chapter 1. Admin Guide
1. Overview
+ 1.1. KDMQ Request
+
The DMQ module implements a distributed message queue on top of
Kamailio in order to facilitate data propagation and replication
between multiple instances, referred as "nodes" (or "peers").
@@ -166,11 +174,16 @@ Chapter 1. Admin Guide
different major versions and can lead to crashes or unexpected
behaviour.
- Example 1.1. KDMQ Request Example
+1.1. KDMQ Request
+
+ KDMQ messages can have different format and content for R-URI username,
+ headers and body, being specific to each component that leverages DMQ
+ to replicate data, such as htable, dialog or usrloc modules.
- This request is generated for DMQ peers availability notifications.
- Other messages can be produced by various modules, with different R-URI
- username and body content, such as htable, dialog or usrloc modules.
+ Next request is generated by DMQ module for peers availability
+ notifications.
+
+ Example 1.1. KDMQ Request Example
...
KDMQ sip:notification_peer@192.168.40.15:5090 SIP/2.0