children monitoring

List overview All Threads
Download

newer

older

Crash on dialog_publish.

record_route() adding wrong sips...

Sergey Safarov

4 Jan 2025 4 Jan '25

2:31 a.m.

If TCP children are busy, Please correct me if I am wrong if the remote IP is not responding via TCP protocol, this leads to a timeout and this child is busy. If many remote endpoints are not reachable via TCP this can lead to the case when half or more children are by timeout operations. I think required to monitor how many children are busy. Is it possible to get such info using kamcmd utility, statistics module, or xprom module. Sergey

Attachments:

attachment.html (text/html — 552 bytes)

Show replies by date

Alex Balashov

4 Jan 4 Jan

12:01 p.m.

It's hard for children to introspect on and report their own busy status, and there's no distributor/supervisor process that has that kind of awareness about the children[1]. `ss -nl` (or `netstat` equivalent) will show you, in the RecvQ column, the number of packets (UDP) or connections (TCP) that are stacked up and going unreplied by any child. This figure should be 0 or substantially 0 if things are going well. -- Alex [1] Notwithstanding the recent 6.x change that optionally adds such a process.

...

On Jan 4, 2025, at 2:31 am, Sergey Safarov via sr-users <sr-users(a)lists.kamailio.org> wrote: If TCP children are busy, Please correct me if I am wrong if the remote IP is not responding via TCP protocol, this leads to a timeout and this child is busy. If many remote endpoints are not reachable via TCP this can lead to the case when half or more children are by timeout operations. I think required to monitor how many children are busy. Is it possible to get such info using kamcmd utility, statistics module, or xprom module. Sergey __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users(a)lists.kamailio.org To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!

-- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800

Henning Westerholt

1:30 p.m.

Hello Alex, indeed, this is easier to analyse with the usual system management tools. Small addition, for TCP and TLS there is actually a distributor or main process inside Kamailio: $ kamcmd ps |grep tcp |grep main 18439 tcp main process $ netstat -nap |grep 5060 | grep 18439 tcp 0 0 116.203.XXX.XXX:5060 0.0.0.0:* LISTEN 18439/kamailio tcp 0 0 116.203.XXX.XXX:5060 0.0.0.0:* LISTEN 18439/kamailio tcp 0 0 127.0.0.1:5060 0.0.0.0:* LISTEN 18439/kamailio tcp6 0 0 2a01:4f8:XXXX:XXXX::5060 :::* LISTEN 18439/kamailio Cheers, Henning

...

-----Original Message----- From: Alex Balashov via sr-users <sr-users(a)lists.kamailio.org> Sent: Samstag, 4. Januar 2025 18:01 To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> Cc: Alex Balashov <abalashov(a)evaristesys.com> Subject: [SR-Users] Re: children monitoring It's hard for children to introspect on and report their own busy status, and there's no distributor/supervisor process that has that kind of awareness about the children[1]. `ss -nl` (or `netstat` equivalent) will show you, in the RecvQ column, the number of packets (UDP) or connections (TCP) that are stacked up and going unreplied by any child. This figure should be 0 or substantially 0 if things are going well. -- Alex [1] Notwithstanding the recent 6.x change that optionally adds such a process.

On Jan 4, 2025, at 2:31 am, Sergey Safarov via sr-users <sr-

users(a)lists.kamailio.org> wrote:

If TCP children are busy, Please correct me if I am wrong if the remote IP is not responding via TCP

protocol, this leads to a timeout and this child is busy.

If many remote endpoints are not reachable via TCP this can lead to the case

when half or more children are by timeout operations.

I think required to monitor how many children are busy. Is it possible to get such info using kamcmd utility, statistics module, or

xprom module.

Sergey __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users(a)lists.kamailio.org To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the

sender! -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr- users(a)lists.kamailio.org To unsubscribe send an email to sr-users- leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!

Arsen Semenov

3:18 p.m.

Hello, I am just curious what is a scenario or configuration which can cause that and therefore need for such monitoring. The remote endpoints are not SIP endpoints? Are you monitoring async_workers? Arsen On Sat, Jan 4, 2025 at 7:55 PM Henning Westerholt via sr-users < sr-users(a)lists.kamailio.org> wrote:

...

status, and

there's no distributor/supervisor process that has that kind of awareness about the children[1]. `ss -nl` (or `netstat` equivalent) will show you, in the RecvQ column,

the

number of packets (UDP) or connections (TCP) that are stacked up and

going

unreplied by any child. This figure should be 0 or substantially 0 if

things are

going well. -- Alex [1] Notwithstanding the recent 6.x change that optionally adds such a process.

On Jan 4, 2025, at 2:31 am, Sergey Safarov via sr-users <sr-

users(a)lists.kamailio.org> wrote: > > If TCP children are busy, > Please correct me if I am wrong if the remote IP is not responding via

TCP

protocol, this leads to a timeout and this child is busy. > If many remote endpoints are not reachable via TCP this can lead to

the case

when half or more children are by timeout operations. > > I think required to monitor how many children are busy. > Is it possible to get such info using kamcmd utility, statistics

module, or

xprom module. > > Sergey > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions -- > sr-users(a)lists.kamailio.org To unsubscribe send an email to > sr-users-leave(a)lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only

to the

the

sender!

__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users(a)lists.kamailio.org To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!

-- Arsenii Semenov

Sergey Safarov

5 Jan 5 Jan

5:47 p.m.

yes, we need to check the remote SIP server's status using OPTIONS messages and TCP protocol. The children's message processing is delayed if the remote server is not reachable. For UDP children sent OPTIONS messages and forgot. But for TCP, it is required to establish a TCP connection first and then children send OPTIONS. If the remote server is not reachable the TCP connection cannot be established and OPTIONS cannot be sent and children are waiting. On Sat, Jan 4, 2025 at 10:40 PM Arsen Semenov via sr-users < sr-users(a)lists.kamailio.org> wrote:

...

status, and

there's no distributor/supervisor process that has that kind of

awareness

about the children[1]. `ss -nl` (or `netstat` equivalent) will show you, in the RecvQ column,

the

number of packets (UDP) or connections (TCP) that are stacked up and

going

unreplied by any child. This figure should be 0 or substantially 0 if

things are

going well. -- Alex [1] Notwithstanding the recent 6.x change that optionally adds such a process.

On Jan 4, 2025, at 2:31 am, Sergey Safarov via sr-users <sr-

users(a)lists.kamailio.org> wrote: > > If TCP children are busy, > Please correct me if I am wrong if the remote IP is not responding

via TCP

protocol, this leads to a timeout and this child is busy. > If many remote endpoints are not reachable via TCP this can lead to

the case

when half or more children are by timeout operations. > > I think required to monitor how many children are busy. > Is it possible to get such info using kamcmd utility, statistics

module, or

to the

sender!

-- Arsenii Semenov __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users(a)lists.kamailio.org To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!

Sergey Safarov

6:40 p.m.

I have modified the command and for my case, it will be like ss -n 'dport 5060 or dport 5061 or sport 5060 or sport 5061' ss -o -n state syn-sent 'dport 5060 or dport 5061 or sport 5060 or sport 5061' For the second command, the output will be like [root@bcf-e-0 ~]# ss -o -n state syn-sent 'dport 5060 or dport 5061 or sport 5060 or sport 5061' Netid Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp 0 1 100.85.64.198:48542 8.8.8.8:5060 timer:(on,9.980ms,4) [root@test ~]# ss -o -n state syn-sent 'dport 5060 or dport 5061 or sport 5060 or sport 5061' Netid Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp 0 1 100.85.64.198:48542 8.8.8.8:5060 timer:(on,8.690ms,4) [root@test ~]# ss -o -n state syn-sent 'dport 5060 or dport 5061 or sport 5060 or sport 5061' Netid Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp 0 1 100.85.64.198:48542 8.8.8.8:5060 timer:(on,7.230ms,4) [root@test ~]# ss -o -n state syn-sent 'dport 5060 or dport 5061 or sport 5060 or sport 5061' Netid Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp 0 1 100.85.64.198:48542 8.8.8.8:5060 timer:(on,6.140ms,4) [root@test ~]# ss -o -n state syn-sent 'dport 5060 or dport 5061 or sport 5060 or sport 5061' Netid Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp 0 1 100.85.64.198:48542 8.8.8.8:5060 timer:(on,5.050ms,4) [root@test ~]# ss -o -n state syn-sent 'dport 5060 or dport 5061 or sport 5060 or sport 5061' Netid Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp 0 1 100.85.64.198:48542 8.8.8.8:5060 timer:(on,3.860ms,4) On Sat, Jan 4, 2025 at 8:55 PM Henning Westerholt via sr-users < sr-users(a)lists.kamailio.org> wrote:

...

status, and

there's no distributor/supervisor process that has that kind of awareness about the children[1]. `ss -nl` (or `netstat` equivalent) will show you, in the RecvQ column,

the

number of packets (UDP) or connections (TCP) that are stacked up and

going

unreplied by any child. This figure should be 0 or substantially 0 if

things are

going well. -- Alex [1] Notwithstanding the recent 6.x change that optionally adds such a process.

On Jan 4, 2025, at 2:31 am, Sergey Safarov via sr-users <sr-

users(a)lists.kamailio.org> wrote: > > If TCP children are busy, > Please correct me if I am wrong if the remote IP is not responding via

TCP

protocol, this leads to a timeout and this child is busy. > If many remote endpoints are not reachable via TCP this can lead to

the case

when half or more children are by timeout operations. > > I think required to monitor how many children are busy. > Is it possible to get such info using kamcmd utility, statistics

module, or

to the

the

sender!

Arsen Semenov

6 Jan 6 Jan

2:37 a.m.

do you have tcp_async = yes (it should be enabled by default) ? What OS do you use? can you please print kamailio -v On Mon, Jan 6, 2025 at 1:02 AM Sergey Safarov via sr-users < sr-users(a)lists.kamailio.org> wrote:

...

status, and

there's no distributor/supervisor process that has that kind of

awareness

about the children[1]. `ss -nl` (or `netstat` equivalent) will show you, in the RecvQ column,

the

number of packets (UDP) or connections (TCP) that are stacked up and

going

unreplied by any child. This figure should be 0 or substantially 0 if

things are

going well. -- Alex [1] Notwithstanding the recent 6.x change that optionally adds such a process.

On Jan 4, 2025, at 2:31 am, Sergey Safarov via sr-users <sr-

users(a)lists.kamailio.org> wrote: > > If TCP children are busy, > Please correct me if I am wrong if the remote IP is not responding

via TCP

protocol, this leads to a timeout and this child is busy. > If many remote endpoints are not reachable via TCP this can lead to

the case

when half or more children are by timeout operations. > > I think required to monitor how many children are busy. > Is it possible to get such info using kamcmd utility, statistics

module, or

to the

sender!

-- Arsenii Semenov

days inactive

days old

sr-users@lists.kamailio.org

Manage subscription

6 comments

4 participants

tags (0)

participants (4)

Alex Balashov
Arsen Semenov
Henning Westerholt
Sergey Safarov