13 Dec
2022
13 Dec
'22
2:13 p.m.
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and
today our monitoring system triggered an alarm for lack of disk space on /.
Indeed, kamailio.log started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following
line was frequently logged in the logs (no instances of it before that
time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine:
bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not
found!
They don't only concern rtpengine, core has produced similar "free fragment
not found" errors as well. There's additional multiple "had header
field",
"bad header" and "invalid header name" errors appearing after that
too, but
I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with
almost identical traffic and configuration, but running on 5.6.1, which has
never exhibited such failures in several months uptime. So I'm wondering,
is there anything in 5.6.2 that could explain this? Thanks!
Best regards,
George
13 Dec
13 Dec
8:31 p.m.
Hello,
difficult to say, it could be also some random memory error or virtual/physical hardware
issue. Are you seeing more error related to that, maybe it’s a memory leak? Make probably
sense to observe it further. If it happens again, a diff between the rtpengine in 5.6.1
and 5.6.2 might get more insight.
Regarding a memory leak, you can observe the memory pool by kamcmd or different tools by
looking at statistics or memory status.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: sr-users <sr-users-bounces(a)lists.kamailio.org> On Behalf Of George
Diamantopoulos
Sent: Tuesday, December 13, 2022 12:14 PM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: [SR-Users] Logs suddenly inundated with qm_malloc errors
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and today our
monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log
started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following line was
frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c:
__bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free fragment not
found" errors as well. There's additional multiple "had header field",
"bad header" and "invalid header name" errors appearing after that
too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with almost
identical traffic and configuration, but running on 5.6.1, which has never exhibited such
failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that
could explain this? Thanks!
Best regards,
George
9:50 p.m.
Maybe there is a memory leak or maybe the system is not provisioned
with enough memory to cope with the traffic. Increase the amount of
allocated memory and monitor it, see how it evolves over time.
Regards,
Ovidiu Sas
On Tue, Dec 13, 2022 at 6:14 AM George Diamantopoulos
<georgediam(a)gmail.com> wrote:
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and today our
monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log
started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following line was
frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c:
__bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free fragment not
found" errors as well. There's additional multiple "had header field",
"bad header" and "invalid header name" errors appearing after that
too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with almost
identical traffic and configuration, but running on 5.6.1, which has never exhibited such
failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that
could explain this? Thanks!
Best regards,
George
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
16 Dec
16 Dec
1:57 p.m.
Hello Henning, Ovidiu,
Thank you for your feedback.
I downgraded to 5.6.1 and the issue hasn't reappeared since then (three
days ago). Note that it did manifest twice in a day when running on 5.6.2
so it seems possible this is a regression.
I'm not sure if the culprit lies in the rtpengine module. There were
similar messages for other components as well later on in the logs (e.g.
core).
Can you think of any changes between 5.6.1 and 5.6.2 that could be
potential triggers? Should I open an issue over at github? Thanks!
Best regards,
George
On Tue, 13 Dec 2022 at 20:53, Ovidiu Sas <osas(a)voipembedded.com> wrote:
Maybe there is a memory leak or maybe the system is
not provisioned
with enough memory to cope with the traffic. Increase the amount of
allocated memory and monitor it, see how it evolves over time.
Regards,
Ovidiu Sas
On Tue, Dec 13, 2022 at 6:14 AM George Diamantopoulos
<georgediam(a)gmail.com> wrote:
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and
today
our monitoring system triggered an alarm for lack of disk space on /.
Indeed, kamailio.log started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following
line was
frequently logged in the logs (no instances of it before that
time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine:
bencode.c:
__bencode_piece_new(79), module: rtpengine; Free fragment not
found!
They don't only concern rtpengine, core has produced similar "free
fragment not found" errors as well. There's additional multiple "had header
field", "bad header" and "invalid header name" errors appearing
after that
too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with
almost identical traffic and configuration, but running on 5.6.1, which has
never exhibited such failures in several months uptime. So I'm wondering,
is there anything in 5.6.2 that could explain this? Thanks!
Best regards,
George
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the
sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
2:27 p.m.
Hi George,
thanks for the feedback. Maybe some insight can be got from logs related to the error, as
mentioned, maybe there are more error messages related to memory allocation problems in
the log?
Other idea is to run it again with the “bad” version, and after some time run the memory
debug to get the allocated memory details.
Further details are documented here:
http://www.kamailio.org/wiki/tutorials/troubleshooting/memory
If you get more feedback, it would be surely good to open an issue to fix the regression,
if its one.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: George Diamantopoulos <georgediam(a)gmail.com>
Sent: Friday, December 16, 2022 11:58 AM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: [SR-Users] Re: Logs suddenly inundated with qm_malloc errors
Hello Henning, Ovidiu,
Thank you for your feedback.
I downgraded to 5.6.1 and the issue hasn't reappeared since then (three days ago).
Note that it did manifest twice in a day when running on 5.6.2 so it seems possible this
is a regression.
I'm not sure if the culprit lies in the rtpengine module. There were similar messages
for other components as well later on in the logs (e.g. core).
Can you think of any changes between 5.6.1 and 5.6.2 that could be potential triggers?
Should I open an issue over at github? Thanks!
Best regards,
George
On Tue, 13 Dec 2022 at 20:53, Ovidiu Sas
<osas@voipembedded.com<mailto:osas@voipembedded.com>> wrote:
Maybe there is a memory leak or maybe the system is not provisioned
with enough memory to cope with the traffic. Increase the amount of
allocated memory and monitor it, see how it evolves over time.
Regards,
Ovidiu Sas
On Tue, Dec 13, 2022 at 6:14 AM George Diamantopoulos
<georgediam@gmail.com<mailto:georgediam@gmail.com>> wrote:
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and today our
monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log
started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following line was
frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c:
__bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free fragment not
found" errors as well. There's additional multiple "had header field",
"bad header" and "invalid header name" errors appearing after that
too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with almost
identical traffic and configuration, but running on 5.6.1, which has never exhibited such
failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that
could explain this? Thanks!
Best regards,
George
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
707
days inactive
710
days old
4 comments
3 participants
participants (3)
-
George Diamantopoulos -
Henning Westerholt -
Ovidiu Sas