Hi,
I upgraded one of our systems today to 4.4.1. And when checking whether everything is okay, I noticed, that only the first UDP listener (from kamcmd core.ps) is actually sending lines to the log. The only other logs I see are from usrloc timer worker.
kamcmd> core.ps 10737 main process - attendant 10740 udp receiver child=0 sock=172.20.40.5:5060 10741 udp receiver child=1 sock=172.20.40.5:5060 10742 udp receiver child=2 sock=172.20.40.5:5060 10743 udp receiver child=3 sock=172.20.40.5:5060
We are sending quite a lot of information via xlog("L_NOTICE", "foo"); to syslog, but the only lines I see are from PID 10740. Kamailio works fine otherwise, handling all requests, the usrloc entries didn't drop, and customers don't complain. So I think it's only a logging issue. There are about 50k registrations on the machine and more than 10 Mbit of traffic.
When I downgrade to 4.3.5, I get output from all listeners again.
Has there been any change between 4.3 and 4.4 regarding logging? The upgrade document says nothing about this.
Best Regards, Sebastian
On Tue, May 24, 2016 at 4:31 PM, Sebastian Damm damm@sipgate.de wrote:
Hi,
I upgraded one of our systems today to 4.4.1. And when checking whether everything is okay, I noticed, that only the first UDP listener (from kamcmd core.ps) is actually sending lines to the log. The only other logs I see are from usrloc timer worker.
I have to add: This happens only on our stateful proxy server. The stateless load balancers in front are logging fine.
Does anybody have an idea?
Best Regards, Sebastian
Hello,
there were some changes in the logging system to allow custom "transport" for syslog messages -- like those implemented by the new log_* modules.
But they should not affect the old (and still default) mode. I just looked at one deployment running actually master branch (which has no change in that regard) and I get logs from the worker processes.
Are you running with systemd? Do you daemonize the main process?
Cheers, Daniel
On 24/05/16 17:06, Sebastian Damm wrote:
On Tue, May 24, 2016 at 4:31 PM, Sebastian Damm damm@sipgate.de wrote:
Hi,
I upgraded one of our systems today to 4.4.1. And when checking whether everything is okay, I noticed, that only the first UDP listener (from kamcmd core.ps) is actually sending lines to the log. The only other logs I see are from usrloc timer worker.
I have to add: This happens only on our stateful proxy server. The stateless load balancers in front are logging fine.
Does anybody have an idea?
Best Regards, Sebastian
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi,
On Tue, May 24, 2016 at 11:25 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
there were some changes in the logging system to allow custom "transport" for syslog messages -- like those implemented by the new log_* modules.
But they should not affect the old (and still default) mode. I just looked at one deployment running actually master branch (which has no change in that regard) and I get logs from the worker processes.
Are you running with systemd? Do you daemonize the main process?
The affected machines are running on Wheezy, so there is no systemd (and we didn't do anything to enable it). We are using the init script provided under pkg/debian/..., so yes, we are daemonizing.
Best Regards, Sebastian
Just to not leave the problem unresolved on the list, we finally found out the reason for it.
We did an strace on one of the UDP listeners, and there we could see that Kamailio was sending out log messages, but on the wrong file descriptor. Actually, it was sending the log lines to one of our RTP proxies.
After telling Daniel about our findings, he dug a bit into rtpengine module and found some issues there.
After applying the two patches
https://github.com/kamailio/kamailio/commit/d76761bf95a5745240d655ee63241b8d... https://github.com/kamailio/kamailio/commit/0fbac693b4d3ed913b64c2d000f27a96...
to our setup, now we have logging servers again.
We were not able to reproduce the problem in our dev environment, only on the production systems. We have an array of 12 rtp engines defined in our config.
Thanks Daniel for those two quick patches!
Best Regards, Sebastian
-
Daniel-Constantin Mierla -
Sebastian Damm