[kamailio/kamailio] Core generated by Kamailio when a bad char is found in header (#1687)
### Description
Kamailio dies complaining about a bad char in some header. I can say that I just updated from the nightly-devel apt repo, this problem never happened in latest v5.1 release.
### Troubleshooting
#### Reproduction
I don't know how to reproduce yet.
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
[BT-FULL_core.kamailio.3329.1540415468.txt](https://github.com/kamailio/kamailio/files/2512570/BT-FULL_core.kamailio.332...) [BT-FULL_core.kamailio.3330.1540415471.txt](https://github.com/kamailio/kamailio/files/2512571/BT-FULL_core.kamailio.333...) [BT-FULL_core.kamailio.3331.1540415467.txt](https://github.com/kamailio/kamailio/files/2512572/BT-FULL_core.kamailio.333...) [BT-FULL_core.kamailio.3333.1540415469.txt](https://github.com/kamailio/kamailio/files/2512573/BT-FULL_core.kamailio.333...) [INFO-LOCALS_core.kamailio.3329.1540415468.txt](https://github.com/kamailio/kamailio/files/2512574/INFO-LOCALS_core.kamailio...) [INFO-LOCALS_core.kamailio.3330.1540415471.txt](https://github.com/kamailio/kamailio/files/2512575/INFO-LOCALS_core.kamailio...) [INFO-LOCALS_core.kamailio.3331.1540415467.txt](https://github.com/kamailio/kamailio/files/2512576/INFO-LOCALS_core.kamailio...) [INFO-LOCALS_core.kamailio.3333.1540415469.txt](https://github.com/kamailio/kamailio/files/2512577/INFO-LOCALS_core.kamailio...) [LIST_core.kamailio.3329.1540415468.txt](https://github.com/kamailio/kamailio/files/2512578/LIST_core.kamailio.3329.1...) [LIST_core.kamailio.3330.1540415471.txt](https://github.com/kamailio/kamailio/files/2512579/LIST_core.kamailio.3330.1...) [LIST_core.kamailio.3331.1540415467.txt](https://github.com/kamailio/kamailio/files/2512580/LIST_core.kamailio.3331.1...) [LIST_core.kamailio.3333.1540415469.txt](https://github.com/kamailio/kamailio/files/2512581/LIST_core.kamailio.3333.1...)
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> The relevant log lines are:
``` Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/parse_addr_spec.c:718]: parse_addr_spec(): unexpected char [<] in status 6: ["910609864" <sip:;:929] . Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:164]: get_hdr_field(): bad to header Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:337]: parse_headers(): bad header field [To: "910609864" <sip:;:929<>64@138.99.136.3>;tag=as664d068c#015#012Call-ID: 032f010653fed9170365045a4e1002] Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/parse_addr_spec.c:718]: parse_addr_spec(): unexpected char [<] in status 6: ["910609864" <sip:;:929] . Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:164]: get_hdr_field(): bad to header Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:337]: parse_headers(): bad header field [To: "910609864" <sip:;:929<>64@138.99.136.3>;tag=as664d068c#015#012Call-ID: 032f010653fed9170365045a4e1002] Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/parse_addr_spec.c:718]: parse_addr_spec(): unexpected char [<] in status 6: ["910609864" <sip:;:929] . Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:164]: get_hdr_field(): bad to header Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:337]: parse_headers(): bad header field [To: "910609864" <sip:;:929<>64@138.99.136.3>;tag=as664d068c#015#012Call-ID: 032f010653fed9170365045a4e1002] Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: pv [pv_core.c:1892]: pv_get_hdr(): error parsing headers Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/parse_addr_spec.c:718]: parse_addr_spec(): unexpected char [<] in status 6: ["910609864" <sip:;:929] . Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:164]: get_hdr_field(): bad to header Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: <core> [core/parser/msg_parser.c:337]: parse_headers(): bad header field [To: "910609864" <sip:;:929<>64@138.99.136.3>;tag=as664d068c#015#012Call-ID: 032f010653fed9170365045a4e1002] Oct 24 16:11:11 sbc01 sbc[3330]: ERROR: pv [pv_core.c:704]: pv_get_callid(): cannot parse Call-Id header ```
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
I have enabled a mirror server to capture all traffic, once the problem happens again I will be able to update this issue with SIP traffic.
### Possible Solutions
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` root@sbc01:/var/tmp# kamailio -v version: kamailio 5.2.0-pre1 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 6.3.0 root@sbc01:/var/tmp# ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) -->
``` root@sbc01:/var/tmp# uname -a Linux sbc01.vozelia.com.pa 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux root@sbc01:/var/tmp#
root@sbc01:/var/tmp# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.5 (stretch) Release: 9.5 Codename: stretch root@sbc01:/var/tmp# ```
To add a more specific version as `5.2.0-pre1` doesn't say much...:
``` root@sbc01:/var/tmp# dpkg -l | grep kam ii kamailio 5.2.0~pre1+0~20181020005754.1238+stretch amd64 very fast, dynamic and configurable SIP server ii kamailio-dbg:amd64 5.2.0~pre1+0~20181020005754.1238+stretch amd64 very fast and configurable SIP server [debug symbols] ii kamailio-extra-modules:amd64 5.2.0~pre1+0~20181020005754.1238+stretch amd64 Extra modules for the Kamailio SIP Server ii kamailio-geoip2-modules:amd64 5.2.0~pre1+0~20181020005754.1238+stretch amd64 The geoip2 module for the Kamailio SIP Server ii kamailio-mysql-modules:amd64 5.2.0~pre1+0~20181020005754.1238+stretch amd64 MySQL database connectivity module for Kamailio ii kamailio-outbound-modules:amd64 5.2.0~pre1+0~20181020005754.1238+stretch amd64 SIP Outbound module for the Kamailio SIP server ii kamailio-phonenum-modules:amd64 5.2.0~pre1+0~20181020005754.1238+stretch amd64 phonenum modules for the Kamailio SIP server ii kamailio-utils-modules:amd64 5.2.0~pre1+0~20181020005754.1238+stretch amd64 Utility functions for the Kamailio SIP server root@sbc01:/var/tmp# ```
The first two backtraces led to a function related to a new feature added in 5.2, respectively statistics for replies per method. The issue was accessing CSeq header fields without any checks if the header was properly parsed. I pushed a commit for it, referenced above. You can try with latest master and see if now all ok.
Hi @miconda, I've just deployed K with this patch, I'll let you know if this happens again.
Out of curiosity, if I wanted to test with the nightly builds, how can I know if for example:
`5.2.0~pre1+0~20181025005757.1239+stretch` deb package contains commit 5e76302?
Thanks!
One idea: execute kamailio -V, this should output the build revision "id:". Then you can see e.g. with git log if this revision already contains your wanted commit.
Yeah this is my bad, it was there all the time, and only because you said so now I actually see it:
``` root@sbc02:~# kamailio -V version: kamailio 5.2.0-pre1 (x86_64/linux) 2ecf60-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 2ecf60 -dirty compiled with gcc 6.3.0 root@sbc02:~# ```
Commit --> 2ecf60
Thanks @henningw
Closed #1687.
Isn't "sip:;:929<>64@138.99.136.3" in the To header what's making it fail?
On Thu, Oct 25, 2018, 22:05 Joel Serrano notifications@github.com wrote:
Closed #1687 https://github.com/kamailio/kamailio/issues/1687.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kamailio/kamailio/issues/1687#event-1927363924, or mute the thread https://github.com/notifications/unsubscribe-auth/AF36ZTx5k_vZi4fa0c4qEQAhUmCj0hawks5uoidagaJpZM4X5L1n . _______________________________________________ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
From the logs I also thought it was something in the To: header but I don't know how to read the backtraces :(
Reopening for now!
Reopened #1687.
@joelsdc - did you get any new crash after the commit I did? Even the error message is related to To header, the issue was not checking if the CSeq is parsed. The core tries to parse the mandatory headers like To, Call-ID, CSeq, ... if one is broken, parsing stops there.
Unless you got new crash with the commit, there is no reason to open this issue. In general, a log message may or may not be related to a crash, the gdb backtrace shows where the crash happens. Otherwise you would apply a logic that if memory manager prints "no more memory", then it is a bug inside it, not in the code using it.
Closed #1687.
-
Daniel-Constantin Mierla -
David Villasmil -
Henning Westerholt
-
Joel Serrano