Hello,
as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated.
I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine.
A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch.
Cheers, Daniel
Hi Daniel. Thank you for back porting the acc and dialog changes to 5.6 -- I continue to test, but can report that the updates on the 5.6 branch through bfc5c2aacb272e91da80096ef27fbe6fcfe9a746 are working properly for acc/dialog- based CDR generation. -A
On Monday, September 26, 2022 6:30:04 AM CDT Daniel-Constantin Mierla wrote:
Hello,
as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated.
I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine.
A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch.
Cheers, Daniel
Thanks for testing and reporting back!
Cheers, Daniel
On 28.09.22 16:34, Anthony Messina wrote:
Hi Daniel. Thank you for back porting the acc and dialog changes to 5.6 -- I continue to test, but can report that the updates on the 5.6 branch through bfc5c2aacb272e91da80096ef27fbe6fcfe9a746 are working properly for acc/dialog- based CDR generation. -A
On Monday, September 26, 2022 6:30:04 AM CDT Daniel-Constantin Mierla wrote:
Hello,
as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated.
I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine.
A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch.
Cheers, Daniel
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated.
I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine.
A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest commit 6f400a8074fe60916867596431ca26dff00435d1. I usually leave a commit load test running for 2 months before consider it ready for production release. I will report any crash/problem.
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi mayamatakeshi@gmail.com wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated.
I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine.
A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest commit 6f400a8074fe60916867596431ca26dff00435d1. I usually leave a commit load test running for 2 months before consider it ready for production release. I will report any crash/problem.
After a few hours of load test, all 4 load test environments start to log memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log |head 2022-10-13T01:36:10.429809+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.923609+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.961677+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983281+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:14.983537+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983665+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.861558+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:20.864388+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.878469+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:23.174159+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6 commit 61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests, then again, kamailio master commit 6f400a8074fe60916867596431ca26dff00435d1 started to write the above logs but this doesn't happen with 5.6.
Hello,
are you using accounting to generate CDRs with dialog module (records in acc_cdrs table)? Or only for getting the event records in the acc table?
Cheers, Daniel
On 13.10.22 23:19, mayamatakeshi wrote:
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi mayamatakeshi@gmail.com wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla <miconda@gmail.com> wrote: Hello, as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation. With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated. I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine. A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time. In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch. I have started 4 load test environments today with latest commit 6f400a8074fe60916867596431ca26dff00435d1. I usually leave a commit load test running for 2 months before consider it ready for production release. I will report any crash/problem.After a few hours of load test, all 4 load test environments start to log memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log |head 2022-10-13T01:36:10.429809+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.923609+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.961677+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983281+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:14.983537+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983665+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.861558+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:20.864388+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.878469+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:23.174159+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6 commit 61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests, then again, kamailio master commit 6f400a8074fe60916867596431ca26dff00435d1 started to write the above logs but this doesn't happen with 5.6.
Hi, I am using acc for CDR generation. Here is the module configuration I'm using:
modparam("acc", "db_url", "mysql://kamailio:kamailiorw@localhost/kamailio") modparam("acc", "db_flag", 1) modparam("acc", "db_missed_flag", 2) modparam("acc", "failed_transaction_flag", 3) modparam("acc", "cdr_enable", 1) modparam("acc", "cdrs_table", "cdr") modparam("acc", "cdr_extra", "callid=$ci;caller_domain=$dlg_var(caller_domain);callee_domain=$dlg_var(callee_domain);caller_username=$dlg_var(caller_username);callee_username=$dlg_var(callee_username);calling_number=$dlg_var(calling_number);destination=$dlg_var(destination);anonymous=$dlg_var(anonymous);forwarding=$dlg_var(forwarding);tracing=$dlg_var(tracing);relay=$dlg_var(relay);sip_code=$dlg_var(sip_code);status_code=$dlg_var(status_code);start_time=$dlg_var(start_time)") modparam("acc", "cdr_start_on_confirmed", 1) modparam("acc", "cdr_start_id", "answer_time") modparam("acc", "log_level", 9) modparam("acc", "log_facility", "LOG_LOCAL0") modparam("acc", "log_flag", 10) modparam("acc", "cdr_facility", "LOG_LOCAL0") modparam("acc", "cdr_log_enable", 1)
On Fri, Oct 14, 2022 at 3:57 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
are you using accounting to generate CDRs with dialog module (records in acc_cdrs table)? Or only for getting the event records in the acc table?
Cheers, Daniel On 13.10.22 23:19, mayamatakeshi wrote:
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi mayamatakeshi@gmail.com wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated.
I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine.
A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest commit 6f400a8074fe60916867596431ca26dff00435d1. I usually leave a commit load test running for 2 months before consider it ready for production release. I will report any crash/problem.
After a few hours of load test, all 4 load test environments start to log memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log |head 2022-10-13T01:36:10.429809+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.923609+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.961677+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983281+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:14.983537+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983665+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.861558+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:20.864388+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.878469+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:23.174159+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6 commit 61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests, then again, kamailio master commit 6f400a8074fe60916867596431ca26dff00435d1 started to write the above logs but this doesn't happen with 5.6.
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training - Online Nov 7-10, 2022 (Europe Timezone)
OK. Can you run the tests for a while and then execute:
kamctl rpc corex.pkg_summary idx 1
The value for idx should be the kamailio process expected to handle sip traffic. If it is mostly udp and the first socket is the corresponding udp socket, then the value 1, like above, is ok. If it is tcp/tls or you have different listen parameters on udp, see the processes index with:
kamctl ps
It should not be required to wait till you get out of memory errors, but be sure you run it long enough to have many accounting records written by the process that is going to print the pkg summary. You can eventually set the children to 1 or 2, to "force" traffic via a smaller group of processes.
Cheers, Daniel
On 14.10.22 10:12, mayamatakeshi wrote:
Hi, I am using acc for CDR generation. Here is the module configuration I'm using:
modparam("acc", "db_url", "mysql://kamailio:kamailiorw@localhost/kamailio") modparam("acc", "db_flag", 1) modparam("acc", "db_missed_flag", 2) modparam("acc", "failed_transaction_flag", 3) modparam("acc", "cdr_enable", 1) modparam("acc", "cdrs_table", "cdr") modparam("acc", "cdr_extra", "callid=$ci;caller_domain=$dlg_var(caller_domain);callee_domain=$dlg_var(callee_domain);caller_username=$dlg_var(caller_username);callee_username=$dlg_var(callee_username);calling_number=$dlg_var(calling_number);destination=$dlg_var(destination);anonymous=$dlg_var(anonymous);forwarding=$dlg_var(forwarding);tracing=$dlg_var(tracing);relay=$dlg_var(relay);sip_code=$dlg_var(sip_code);status_code=$dlg_var(status_code);start_time=$dlg_var(start_time)") modparam("acc", "cdr_start_on_confirmed", 1) modparam("acc", "cdr_start_id", "answer_time") modparam("acc", "log_level", 9) modparam("acc", "log_facility", "LOG_LOCAL0") modparam("acc", "log_flag", 10) modparam("acc", "cdr_facility", "LOG_LOCAL0") modparam("acc", "cdr_log_enable", 1)
On Fri, Oct 14, 2022 at 3:57 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello, are you using accounting to generate CDRs with dialog module (records in acc_cdrs table)? Or only for getting the event records in the acc table? Cheers, Daniel On 13.10.22 23:19, mayamatakeshi wrote:On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi <mayamatakeshi@gmail.com> wrote: On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla <miconda@gmail.com> wrote: Hello, as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation. With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated. I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine. A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time. In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch. I have started 4 load test environments today with latest commit 6f400a8074fe60916867596431ca26dff00435d1. I usually leave a commit load test running for 2 months before consider it ready for production release. I will report any crash/problem. After a few hours of load test, all 4 load test environments start to log memory allocation problems: [root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log |head 2022-10-13T01:36:10.429809+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.923609+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.961677+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983281+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:14.983537+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983665+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.861558+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:20.864388+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.878469+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:23.174159+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed I reverted 2 of the load test envs to previous kamailio 5.6 commit 61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests, then again, kamailio master commit 6f400a8074fe60916867596431ca26dff00435d1 started to write the above logs but this doesn't happen with 5.6.-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio Advanced Training - Online Nov 7-10, 2022 (Europe Timezone) * https://www.asipto.com/sw/kamailio-advanced-training-online/
Based on your parameters, it seems that acc should send cdrs also to syslog. I think I identified an issue on that code and pushed a fix, can you test with latest master git branch?
Cheers, Daniel
On 14.10.22 10:25, Daniel-Constantin Mierla wrote:
OK. Can you run the tests for a while and then execute:
kamctl rpc corex.pkg_summary idx 1
The value for idx should be the kamailio process expected to handle sip traffic. If it is mostly udp and the first socket is the corresponding udp socket, then the value 1, like above, is ok. If it is tcp/tls or you have different listen parameters on udp, see the processes index with:
kamctl ps
It should not be required to wait till you get out of memory errors, but be sure you run it long enough to have many accounting records written by the process that is going to print the pkg summary. You can eventually set the children to 1 or 2, to "force" traffic via a smaller group of processes.
Cheers, Daniel
On 14.10.22 10:12, mayamatakeshi wrote:
Hi, I am using acc for CDR generation. Here is the module configuration I'm using:
modparam("acc", "db_url", "mysql://kamailio:kamailiorw@localhost/kamailio") modparam("acc", "db_flag", 1) modparam("acc", "db_missed_flag", 2) modparam("acc", "failed_transaction_flag", 3) modparam("acc", "cdr_enable", 1) modparam("acc", "cdrs_table", "cdr") modparam("acc", "cdr_extra", "callid=$ci;caller_domain=$dlg_var(caller_domain);callee_domain=$dlg_var(callee_domain);caller_username=$dlg_var(caller_username);callee_username=$dlg_var(callee_username);calling_number=$dlg_var(calling_number);destination=$dlg_var(destination);anonymous=$dlg_var(anonymous);forwarding=$dlg_var(forwarding);tracing=$dlg_var(tracing);relay=$dlg_var(relay);sip_code=$dlg_var(sip_code);status_code=$dlg_var(status_code);start_time=$dlg_var(start_time)") modparam("acc", "cdr_start_on_confirmed", 1) modparam("acc", "cdr_start_id", "answer_time") modparam("acc", "log_level", 9) modparam("acc", "log_facility", "LOG_LOCAL0") modparam("acc", "log_flag", 10) modparam("acc", "cdr_facility", "LOG_LOCAL0") modparam("acc", "cdr_log_enable", 1)
On Fri, Oct 14, 2022 at 3:57 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello, are you using accounting to generate CDRs with dialog module (records in acc_cdrs table)? Or only for getting the event records in the acc table? Cheers, Daniel On 13.10.22 23:19, mayamatakeshi wrote:On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi <mayamatakeshi@gmail.com> wrote: On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla <miconda@gmail.com> wrote: Hello, as I am not a user of dialog variables, I am turning to community to ask for help testing the current master branch with configurations that make use of dialog variables and acc dialog-based cdr generation. With a few reports of issues related to dialog modules and unexpected crashes, I looked over the code and noticed that the access of the value for dialog variables was not protected, making them vulnerable of invalid memory access in case of the variable was updated by another process or dialog was terminated. I introduced a couple of new functions to try to cover different use cases of getting the dlg variable values, dialog management code was not affected, but given that these commits need to be backported to stable branch (5.6), I want to get proper feedback from community that things work fine. A previous attempt of a simpler fix was not enough, having side effects to acc module for dialog-based cdr generation, because it was keeping referenced to many dlg variables at the same time. In short, it would be appreciated any feedback on testing dialog and acc with dialog-based cdr generation using git master branch. I have started 4 load test environments today with latest commit 6f400a8074fe60916867596431ca26dff00435d1. I usually leave a commit load test running for 2 months before consider it ready for production release. I will report any crash/problem. After a few hours of load test, all 4 load test environments start to log memory allocation problems: [root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log |head 2022-10-13T01:36:10.429809+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.923609+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:12.961677+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983281+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:14.983537+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:14.983665+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.861558+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed 2022-10-13T01:36:20.864388+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:20.878469+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-13T01:36:23.174159+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed I reverted 2 of the load test envs to previous kamailio 5.6 commit 61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests, then again, kamailio master commit 6f400a8074fe60916867596431ca26dff00435d1 started to write the above logs but this doesn't happen with 5.6.-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio Advanced Training - Online Nov 7-10, 2022 (Europe Timezone) * https://www.asipto.com/sw/kamailio-advanced-training-online/-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training - Online Nov 7-10, 2022 (Europe Timezone)
On Fri, Oct 14, 2022 at 6:21 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Based on your parameters, it seems that acc should send cdrs also to syslog. I think I identified an issue on that code and pushed a fix, can you test with latest master git branch?
OK. I have started 2 load test environments using latest master
commit 0d363cf1c1f09b4920e137eac74e1593e7120531.
On Mon, Oct 17, 2022 at 7:45 AM mayamatakeshi mayamatakeshi@gmail.com wrote:
On Fri, Oct 14, 2022 at 6:21 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Based on your parameters, it seems that acc should send cdrs also to syslog. I think I identified an issue on that code and pushed a fix, can you test with latest master git branch?
OK. I have started 2 load test environments using latest master
commit 0d363cf1c1f09b4920e137eac74e1593e7120531.
Load tests are running for more than 30 hours and the pkg problem doesn't happen anymore. So it seems your changes solved the issue. I will start 2 more load test envs and keep them running for 2 months. Thanks.
On Fri, Oct 14, 2022 at 5:25 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
OK. Can you run the tests for a while and then execute:
kamctl rpc corex.pkg_summary idx 1
The value for idx should be the kamailio process expected to handle sip traffic. If it is mostly udp and the first socket is the corresponding udp socket, then the value 1, like above, is ok. If it is tcp/tls or you have different listen parameters on udp, see the processes index with:
kamctl ps
I tried calling kamctl rpc corex.pkg_summary idx 1 before and after the memory problem started to be logged but it always showed up with result empty:
{ "jsonrpc": "2.0", "result": { }, "id": 3824999 }
Here is result for all processes: https://pastebin.com/AabwyJwx I tried several times but I never saw a non-empty result.
It should not be required to wait till you get out of memory errors, but be sure you run it long enough to have many accounting records written by the process that is going to print the pkg summary. You can eventually set the children to 1 or 2, to "force" traffic via a smaller group of processes.
By reducing children from children=12 to children=1 I started to get memory allocation error just a after a few minutes of load test:
[root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log |head 2022-10-17T06:20:41.314619+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:41.916596+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:41.916786+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:42.518013+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:42.518248+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.114454+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.163816+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.165863+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.167092+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.167247+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/data_lump.c:163]: insert_new_lump_before(): could not allocate private memory from pkg pool
I collected the pkg summary after 4000 CDRs were written to DB but again, i could not see a non-empty result: https://pastebin.com/CpaRYLAR
The rpc command is to arm the printing of the pkg usage summary in the syslog file. Then, when the kamailio process handles the next sip message, it is printing the summary in the syslog file, so look inside /var/log/syslog or /var/log/messages, you should spot the related log messages, they should contain "summary" or something similar...
Cheers, Daniel
On 16.10.22 23:48, mayamatakeshi wrote:
On Fri, Oct 14, 2022 at 5:25 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
OK. Can you run the tests for a while and then execute: kamctl rpc corex.pkg_summary idx 1 The value for idx should be the kamailio process expected to handle sip traffic. If it is mostly udp and the first socket is the corresponding udp socket, then the value 1, like above, is ok. If it is tcp/tls or you have different listen parameters on udp, see the processes index with: kamctl psI tried calling kamctl rpc corex.pkg_summary idx 1 before and after the memory problem started to be logged but it always showed up with result empty:
{ "jsonrpc": "2.0", "result": { }, "id": 3824999 }
Here is result for all processes: https://pastebin.com/AabwyJwx I tried several times but I never saw a non-empty result.
It should not be required to wait till you get out of memory errors, but be sure you run it long enough to have many accounting records written by the process that is going to print the pkg summary. You can eventually set the children to 1 or 2, to "force" traffic via a smaller group of processes.By reducing children from children=12 to children=1 I started to get memory allocation error just a after a few minutes of load test:
[root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log |head 2022-10-17T06:20:41.314619+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:41.916596+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:41.916786+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:42.518013+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:42.518248+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc [acc_extra.c:234]: extra2strar(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.114454+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.163816+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.165863+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.167092+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/rvalue.c:253]: rval_new_empty(): could not allocate private memory from pkg pool 2022-10-17T06:20:43.167247+09:00 lab002107-flip-server /usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core> [core/data_lump.c:163]: insert_new_lump_before(): could not allocate private memory from pkg pool
I collected the pkg summary after 4000 CDRs were written to DB but again, i could not see a non-empty result: https://pastebin.com/CpaRYLAR
On Mon, Oct 17, 2022 at 10:01 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
The rpc command is to arm the printing of the pkg usage summary in the syslog file. Then, when the kamailio process handles the next sip message, it is printing the summary in the syslog file, so look inside /var/log/syslog or /var/log/messages, you should spot the related log messages, they should contain "summary" or something similar...
OK. Understood. I will try it again the next time pkg memory happens.
-
Anthony Messina -
Daniel-Constantin Mierla -
mayamatakeshi