Thanks Jose.

We actually use DH cipher encryption between our clients and Kamailio and this is production setup so I can't use any other supported types.

I can for sure double check the cpu and memory but generally this server is not heavily used.

Is there anyway for Kamailio to log in the configs memory and cpu usage?

On Wed, Sep 26, 2018 at 5:03 AM José Seabra joseseabra4@gmail.com wrote:

Hi Andy,

The server where kamailio is running is not CPU overloaded? and what about memory consumption?

I would say that if you decode the sip tls packages, it would be more easy to debug the issue.

The link bellow can guide you how to decode SIP TLS packages:

https://www.kamailio.org/wiki/tutorials/tls/testing-and-debugging#decoding_o...

Hope that the information provided can help you.

Regards José

Andrew Chen achen@fuze.com escreveu no dia terça, 25/09/2018 à(s) 07:17:

...

Hi Kam community,

Facing an urgent issue here and not sure how to troubleshoot it further.

So our engineers been getting reports of unknown SIP timeouts and they suspect it's something on the Kamailio side. I've logged every transaction from receiving the message to done relaying it as shown below:

Sep 24 18:38:13 sjoprodfnsg01 /usr/local/sbin/kamailio[42608]: INFO:

<script>: [1589Fz-ses1-41d9a88aR70125425@xxx.xxx.xxx.xxx][4534123][CSeq 7] NG-FuzeMedia-Server/42ae58c/mediahub Relaying [UPDATE] from (xxx.xxx.xxx.xxx ) to ( xxx.xxx.xxx.xxx ) via (tls) Sep 24 18:38:13 sjoprodfnsg01 /usr/local/sbin/kamailio[42608]: INFO: <script>: [1589Fz-ses1-41d9a88aR70125425@ xxx.xxx.xxx.xxx ][4534123][CSeq 7] NG-FuzeMedia-Server/42ae58c/mediahub Done relay [UPDATE] from (xxx.xxx.xxx.xxx ) to ( xxx.xxx.xxx.xxx ) via (tls) When I pulled the 443 SIP trace from my tcpdump session, it shows the application data left the machine at 18:44 as shown here: [image: image.png] Note: This packet is encrypted and can't see the actual SIP message. This matches what was received on the client logs: 2018-09-24T18:38:44.032Z [40927] [INF] fuze.sip SipParser.cpp:549 LogSipMessage: xxx.xxx.xxx.xxx:443 - 4292B (local time 09/24 11:38:44) -------------------- RECEIVE: UPDATE sips:Luke_Surazski@ xxx.xxx.xxx.xxx:50795;id=37262838;inst=52241254 SIP/2.0 Via: SIP/2.0/TLS xxx.xxx.xxx.xxx:443;branch=z9hG4bK15f.5052c318e1766a0a9405f10708f43188.0;i=acbb7 Via: SIP/2.0/TCP xxx.xxx.xxx.xxx :38342;rport=38342;branch=z9hG4bK-55f913-t9-I214262897 What would cause such delays? I'm running Kamailio 4.1.5 Thanks. -- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ <achen@thinkingphones.com>fuze.com *Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.* _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Cumprimentos José Seabra _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

update to my last email.

If do you see that the SIP transaction(UPDATE) arrives at your client fast enough, it isn't a problem with TCP send Queue (ignore my last email). Do you can see the reply to the SIP transaction(UPDATE) sent from your client arriving to the host where kamailio is running? If not, seems that it isn't a problem in your Kamailio.

Regards José

José Seabra joseseabra4@gmail.com escreveu no dia quinta, 27/09/2018 à(s) 10:47:

Hi,

Kamailio has some configuration options that logs a msg to the syslog when some config script action takes more time that is expected, the options are:

• latency_limit_action=500
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_action
• latency_limit_db=500
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_db
• latency_log=2
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_log

The issue could also be related with the TCP write queue, maybe you have set low values on the following parameters:

• tcp_conn_wq_max
  • https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_conn_wq_max
• tcp_wq_max
  • https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_wq_max

Other hint is to look for the OS send Queue, the package could be delayed because the other end is not reading it fast enough:

• netstat -altp

it also will be important try to isolate the issue in terms of SIP, the transaction delayed(UPDATE) is always the same in all situation that the issue occurs?

Hope that the information above helped you.

Regards José

Hi, So, Kamailio receives the UPDATE msg from your application server and relays it to the client and the delay is on UPDATE msg relay between kamailio and the client.

As the wireshark capture the outgoing packages just after the OS processing them, then if you take a network trace, trigger the issue and try to identify the package sent by kamailio(UPDATE) in the network trace, there you can see if there is any delay between the application(Kamailio) and network level, comparing the time registered between the logs and the pcap.

If the 30 seconds is not between the kamailio and network level, i would say that the issue is not in your kamailio box.

Other way is trying to check out the OS sending queue status:

- netstat -altp

Regards

José

Andrew Chen achen@fuze.com escreveu no dia quinta, 27/09/2018 à(s) 14:17:

So I log when the server side sends an UPDATE request to the Kamailio and then I log when the t_relay is completed to the client application. Those were on the same second. It's when the message left the box and then arrived at the client side is where the 30 sec difference came from. So our engineering is questioning if the t_relay really did relay it properly out of the system as it was logged. They are suspecting there could be some issue with the t_relay and the OS level tcp queue.

On Thu, Sep 27, 2018 at 6:23 AM José Seabra joseseabra4@gmail.com wrote:

...

update to my last email.

If do you see that the SIP transaction(UPDATE) arrives at your client fast enough, it isn't a problem with TCP send Queue (ignore my last email). Do you can see the reply to the SIP transaction(UPDATE) sent from your client arriving to the host where kamailio is running? If not, seems that it isn't a problem in your Kamailio.

Regards José

José Seabra joseseabra4@gmail.com escreveu no dia quinta, 27/09/2018 à(s) 10:47:

...

Hi,

Kamailio has some configuration options that logs a msg to the syslog when some config script action takes more time that is expected, the options are:

• latency_limit_action=500
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_action
• latency_limit_db=500
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_db
• latency_log=2
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_log

The issue could also be related with the TCP write queue, maybe you have set low values on the following parameters:

• tcp_conn_wq_max
  https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_conn_wq_max
• tcp_wq_max
  • https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_wq_max

Other hint is to look for the OS send Queue, the package could be delayed because the other end is not reading it fast enough:

• netstat -altp

it also will be important try to isolate the issue in terms of SIP, the transaction delayed(UPDATE) is always the same in all situation that the issue occurs?

Hope that the information above helped you.

Regards José

-- Cumprimentos José Seabra _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com

*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.* _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

It is hard to me tell you if the issue is or not in kamailio, kamailio is like a sip framework and the way how it works depends a lot how do you had implemented it.

That's why i was giving you some hints in order to help you move forward on troubleshooting.

Anyway, if do you see that the package left the kamailio box more or less at same time that it was registered on the kamailio logs, i would say that the issue is not in kamailio box.

But if do you notice that the 30 seconds is in between the time registered on the kamailio logs and when the package left the kamailio box(wireshark), so probably it is something between kamailio and the OS and here is necessary investigate why kamailio is delaying the package.

Best Regards

Andrew Chen achen@fuze.com escreveu no dia quinta, 27/09/2018 à(s) 17:01:

The system does tcpdump real time on the system. We do see the delay going out the box to the destination client side by the 30 seconds. There is also suspicion about the network quality as well given we see retransmits of ACK packets during the time the UPDATE was to be delivered. I just want to make sure there is nothing on the Kamailio side would be causing this delay, which is what our engineering team is suspecting.

On Thu, Sep 27, 2018 at 11:23 AM José Seabra joseseabra4@gmail.com wrote:

...

Hi, So, Kamailio receives the UPDATE msg from your application server and relays it to the client and the delay is on UPDATE msg relay between kamailio and the client.

As the wireshark capture the outgoing packages just after the OS processing them, then if you take a network trace, trigger the issue and try to identify the package sent by kamailio(UPDATE) in the network trace, there you can see if there is any delay between the application(Kamailio) and network level, comparing the time registered between the logs and the pcap.

If the 30 seconds is not between the kamailio and network level, i would say that the issue is not in your kamailio box.

Other way is trying to check out the OS sending queue status:

• netstat -altp

Regards

José

Andrew Chen achen@fuze.com escreveu no dia quinta, 27/09/2018 à(s) 14:17:

...

So I log when the server side sends an UPDATE request to the Kamailio and then I log when the t_relay is completed to the client application. Those were on the same second. It's when the message left the box and then arrived at the client side is where the 30 sec difference came from. So our engineering is questioning if the t_relay really did relay it properly out of the system as it was logged. They are suspecting there could be some issue with the t_relay and the OS level tcp queue.

On Thu, Sep 27, 2018 at 6:23 AM José Seabra joseseabra4@gmail.com wrote:

...

update to my last email.

If do you see that the SIP transaction(UPDATE) arrives at your client fast enough, it isn't a problem with TCP send Queue (ignore my last email). Do you can see the reply to the SIP transaction(UPDATE) sent from your client arriving to the host where kamailio is running? If not, seems that it isn't a problem in your Kamailio.

Regards José

José Seabra joseseabra4@gmail.com escreveu no dia quinta, 27/09/2018 à(s) 10:47:

...

Hi,

Kamailio has some configuration options that logs a msg to the syslog when some config script action takes more time that is expected, the options are:

• latency_limit_action=500
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_action
• latency_limit_db=500
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_db
• latency_log=2
• https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_log

The issue could also be related with the TCP write queue, maybe you have set low values on the following parameters:

• tcp_conn_wq_max
  https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_conn_wq_max
• tcp_wq_max
  • https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_wq_max

Other hint is to look for the OS send Queue, the package could be delayed because the other end is not reading it fast enough:

• netstat -altp

it also will be important try to isolate the issue in terms of SIP, the transaction delayed(UPDATE) is always the same in all situation that the issue occurs?

Hope that the information above helped you.

Regards José

-- Cumprimentos José Seabra _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com

*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.* _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Cumprimentos José Seabra _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com

*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.* _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users