Hello,
Sorry, next try with posting this message. Attempts with attachment failed due to message size.
We are running OpenSER in a pilot project and unfortunately have some stability problems.
Any help or hints are appreciated.
Project information
OpenSER is used in a pilot project with
* Appr. 5000 subscriber accounts
* Appr. 1200 simultaneously registered users
* Signalling encrypted with TLS
* Media data encrypted with SRTP
* Clients: softphones and hardphones
* Re-registration time for clients: 3600 sec
OpenSER configuration
· Works as stateful SIP Proxy
1 mySQL database
2 Version 1.3.4.-TLS
3 Tcp_children: 100 --> is it recommended to increase this number?
4 Udp_children: 20
5 Tcp_connection_timeout: 3600
6 Shared memory:
· -m 512 when error occurred
1 Now set to 1024
Problems
* Shared memory consumption
Shared memory usage is permanently increasing (about 50 MB per day)
Application already crashed twice
First messages were, these, repeated thousands of times (5915 times):
Jun 17 08:54:52 si-.... /usr/local/sbin/openser[13921]: ERROR:core:tcpconn_new: shared memory allocation failure
Jun 17 08:54:52 si-... /usr/local/sbin/openser[13921]: ERROR:core:handle_new_connect: tcpconn_new failed, closing socket
And a few of these also (7613 times):
Jun 17 08:57:24 si-... /usr/local/sbin/openser[13880]: ERROR:core:tls_accept: some error in SSL:
Jun 17 08:57:24 si-... /usr/local/sbin/openser[13880]: ERROR:core:tls_print_errstack: error:1409C041:SSL routines:SSL3_SETUP_BUFFERS:malloc failure
* TCP errors, lost SIP messages
Examples from error messages:
14.100 times in log file from 17.06.09
Jun 17 04:03:15 si-... /usr/local/sbin/openser[13863]: ERROR:core:tcp_blocking_connect: poll error: flags 18
Jun 17 04:03:15 si-... /usr/local/sbin/openser[13863]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (111) Connection refused
Jun 17 04:03:15 si-... /usr/local/sbin/openser[13863]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed
Jun 17 04:03:15 si-... /usr/local/sbin/openser[13863]: ERROR:core:tcp_send: connect failed
Jun 17 04:03:15 si-.. /usr/local/sbin/openser[13863]: ERROR:tm:msg_send: tcp_send failed
Jun 17 04:03:15 si-... /usr/local/sbin/openser[13863]: ERROR:tm:t_forward_nonack: sending request failed
Appears at least 20 000 times; and in the day of the last shared memory errors, it was 225.794 times in the log file (note that the number in parenthesis is usually 1 or 2, but on that day it has reached 6):
Jun 17 09:01:27 si-.... /usr/local/sbin/openser[13921]: WARNING:core:send2child: no free tcp receiver, connection passed to the leastbusy one (6)
Jun 17 09:01:27 si-... /usr/local/sbin/openser[13921]: WARNING:core:send2child: no free tcp receiver, connection passed to the leastbusy one (5)
* Certificate validation problems
TCP traffic is currently significantly increased by some ( appr. 70) clients which failed to validate the TLS certificate. Registration is repeated every 5 sec.
Circa 30 thousand per day (on that day, it was 37.162 times in log)
Jun 17 04:03:10 si-024lc008 /usr/local/sbin/openser[13801]: ERROR:core:tls_accept: some error in SSL:
Jun 17 04:03:10 si-024lc008 /usr/local/sbin/openser[13801]: ERROR:core:tls_print_errstack: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
Mit besten Grüßen | Best regards
Albert Munder
Robert Bosch GmbH
IT Systems Engineering (CI/ISE)
Postfach 30 02 20
70442 Stuttgart
GERMANY
www.bosch.com
Tel. +49 711 811-40562
Fax +49 711 811-5113333
Albert.Munder(a)de.bosch.com
Robert Bosch GmbH, Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 14000
Aufsichtsratsvorsitzender: Hermann Scholl; Geschäftsführung: Franz Fehrenbach, Siegfried Dais;
Bernd Bohr, Wolfgang Chur, Rudolf Colm, Gerhard Kümmel, Wolfgang Malchow, Peter Marks;
Volkmar Denner, Peter Tyroller.
I am running SER/rtpproxy on some systems with
multiple cores and multiple ethernet interfaces.
I am running multiple copies of rtpproxy, each with
an IP address associated with a particular interface,
as in one rtpproxy is using em1 which is 208.33.44.3
and em2 is 208.33.44.4 which another rtpproxy listens
at, while ser listens in em0 at 208.33.44.2. These
three (actually ten) interfaces pass through ethernet
switches and then on to a router with a default gateway
of 208.33.44.1 and then off to the Internet or wherever.
(In my setup, each rtpproxy has a second interface pointing
into a private network, so only one side of a call appears
on any given interface and five face toward the Internet.)
SER+rtpproxy correctly divides RTP traffic for calls to
208.33.44.3 and 208.33.44.4 in the SDP payloads, and
callers send their RTP packets to the two rtproxies
IP addresses without problems. That part works just fine.
The problem is that rtpproxy is using the default route
for transmission, so everything that goes towards the
Internet goes out via em0, so em1 and em2 don't transmit
anything. This means that this one link is overwhelmed
by the traffic that should have been divided across
multiple links.
Rtpproxy is allowing the local route table and the
default route to handle the outgoing packets and so
the packets are all going out the same interface.
In my case, em0 was 100Mbit/sec (completely sufficient
for SIP messages, while em1 and em2 were gigabit links
to handle RTP, but they are getting used only in one
direction. Even if em0 was upgraded to Gbit, it would
still not be able to handle the potential for 2+ Gbit
of data from the rtproxies.
So, is there any way to force rtproxy to perform what
I believe they used to call "reflector mode", where
the answers go out the same interface they come in on?
(This used to be common on UDP-based NFS file servers.)
A number of communication programs also allow this
behavior as an option, but rtpproxy doesn't appear
to do it or it isn't turned on by default.
Because the destinations for the RTP are out on
the Internet somewhere, I don't believe I can solve this
via "route" or "ifconfig" settings, but would be
delighted if someone knows different or knows of
a setting change in rtpproxy that will do this
so that load balancing will work.
The OS in question is FreeBSD 7.2, but I have the same
issue on 6.2 and 6.3.
Thanks in advance!
2009/3/23 Alexandr Dubovikov <shurik(a)start4.info>:
> On Mon, Mar 23, 2009 at 11:15:52AM +0100, Iñaki Baz Castillo wrote:
>> 2009/4/1 Alexandr Dubovikov <shurik(a)start4.info>:
>> > On Fri, Mar 20, 2009 at 10:21:20PM +0100, Andreas Heise wrote:
>> >> Hello Iñaki,
>
> Hi Inaki,
>
>> >
>> > Hi all,
>> >>
>> >> you should ask Alexandr he has introdused this feature with rev5452,
>> >> but I'm not sure if he is on the lists all the time, so I'll forward your
>> >> question
>> >> to him...
>> >
>> > sorry, I am currently offline and couldn't answer directly to the list.
>> >
>> > anyway. I agreed, 180 seconds it's too big ping interval, but it couldn't be
>> > also less 32 seconds.
>> >
>> >
>> > http://www.ietf.org/rfc/rfc3261.txt
>> >
>> > 17.2.2 Non-INVITE Server Transaction
>> >
>> >
>> > T1 = 500 ms.
>> >
>> >
>> > Timer J 64*T1 for UDP Section 17.2.2 Wait time for
>> > 0s for TCP/SCTP non-INVITE request
>> >
>> > so, the timer for "completed" is 32 seconds and couldn't be less.
>> >
>> >
>> > of course, you can use less value, but on own risk :-)
>> > RFC 3261 are not recommendet it :)
>>
>> Thanks for pointing it out.
>> However, if a gw takes so long time (~32 seconds) to respond I prefer
>> to consider it as "offline" :)
>
> don't forget to disable ping for this gateway, because finaly you will have
> a DDOS effect of the "offline" gw. :-)
Well, If the host is already offline then it doesn't matter to kill it more XD
Thanks.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
Hi All,
First off - thank you for this module.
I have configured two redundant DB sources in my kamailio confiuration. The
DB's are accessed from Kamailio for redundancy pruposes.
Kamailio and the primary DB are on the same hardware node. The secondary DB
is on a remote host.
If Kamailio is up and both MySQL DB's are up, the config works like a
charm. If the primary or the secondary DB becomes unreachable while
Kamailio is up, the config handles the exception and all works well.
However, if any of the DB servers becomes unreachable and Kamailio is
restarted, Kamailio can no longer start.
Below is an excerpt from my config file:
# -- sqlops params
modparam("sqlops","sqlcon","pridb=>mysql://root:@localhost/db_table")
modparam("sqlops","sqlcon","secdb=>mysql://root:@192.168.1.231/db_table")
/var/log/messages
Jun 26 06:16:00 test-ser /sbin/kamailio[20217]:
ERROR:db_mysql:db_mysql_new_connection: driver error: Can't connect to local
MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Jun 26 06:16:00 test-ser /sbin/kamailio[20217]: ERROR:core:db_do_init: could
not add connection to the pool
Jun 26 06:16:00 test-ser /sbin/kamailio[20217]: ERROR:sqlops:sql_connect:
failed to connect to the database [pridb]
Jun 26 06:16:00 test-ser /sbin/kamailio[20217]: ERROR:core:init_mod_child:
failed to initializing module sqlops, rank 1
Jun 26 06:16:00 test-ser /sbin/kamailio[20217]: ERROR:core:main_loop:
init_child failed for UDP listener
What is the proper way of using multiple database servers with Kamailio?
Thanks in advance.
--JR
Hi all,
i'm pleased to announce that i give a talk about the kamailio project on
the linuxtag 2009 in Berlin.
The title of the talk is "Building carrier grade voice over IP systems
with kamailio", on wednesday the 24th june afternoon. Further details can
be found in the conference program at:
http://www.linuxtag.org/2009/de/program/freies-vortragsprogramm/mittwoch/vo…
I'll be also present on the linuxtag on thursday, so if you would like to
meet for some discussions, just drop me a mail (off-list).
With best regards,
Henning Westerholt
Hello,
there is a new module in repository that allow you to execute most of
the mi commands from Kamailio modules (see readme) via RPC interface
inherited from SER:
http://sip-router.org/docbook/sip-router/branch/master/modules/mi_rpc/mi_rp…http://sip-router.org/docbook/sip-router/branch/master/rpc/ser_rpc.html
One of the nicest things you can out of this is usage of sercmd -
command line interface tool. It is located in utils/sercmd. Thanks to
Andrei, sercmd does now tab completion for mi command.
How to use it:
- get sip router from git:
git clone --depth 1 git://git.sip-router.org/sip-router
cd sip-router
make all
- add to your config file:
loadmodule "ctl.so" # located in 'modules_s'
loadmodule "mi_rpc.so" # located in 'modules'
- use sercmd
cd utils/sercmd
make
./sercmd
Then you should get:
sercmd> mi <tab><tab>
arg nh_show_rtpp t_reply ul_rm_contact
cleanup ps t_uac_cancel ul_show_contact
debug pwd t_uac_dlg uptime
dum refreshWatchers ul_add version
kill shv_get ul_dump which
nh_enable_ping shv_set ul_flush
nh_enable_rtpp t_hash ul_rm
sercmd> mi uptime
200 OK
+ Now:: Tue Jun 30 12:47:54 2009
+ Up since:: Tue Jun 30 12:47:26 2009
+ Up time:: 28 [sec]
Via sercmd you can control lot of internals at runtime -- see 'help'
command -- more about:
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=blob_plain;f=ut…
Enjoy,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com/
The ZIP-attachment sent to you was blocked.
Please inform the original sender to rename ZIP-files.
Die Ihnen gesandte ZIP-Datei wurde geblockt.
Bitte informieren Sie den ursprünglichen Absender. (Dateiendung umbenennen).
Sender: Albert.Munder(a)de.bosch.com
Recipients: users(a)lists.kamailio.org
Subject: "OpenSER stability problems in pilot project"
Datei / Filename: OpenSER_stability.zip
Time: Jun 30, 2009 11:49:04 AM
The cleaned message is attached.
------------------------------------------------
Hello,
We are running OpenSER in a pilot project and unfortunately have some stability problems.
Related information are summarized in the attached document:
Any help or hints are appreciated.
Mit besten Grüßen | Best regards
Albert Munder
Robert Bosch GmbH
IT Systems Engineering (CI/ISE)
Postfach 30 02 20
70442 Stuttgart
GERMANY
www.bosch.com
Tel. +49 711 811-40562
Fax +49 711 811-5113333
Albert.Munder(a)de.bosch.com
Robert Bosch GmbH, Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 14000
Aufsichtsratsvorsitzender: Hermann Scholl; Geschäftsführung: Franz Fehrenbach, Siegfried Dais;
Bernd Bohr, Wolfgang Chur, Rudolf Colm, Gerhard Kümmel, Wolfgang Malchow, Peter Marks;
Volkmar Denner, Peter Tyroller.
I'll try first with the debug in kamailio... I'll let you know in a few
minutes...
2009/6/27 Raúl Alexis Betancor Santana <rabs(a)dimension-virtual.com>
> On Saturday 27 June 2009 02:10:36 Uriel Rozenbaum wrote:
> > Oki doki... how can I do that? I really don't know how to do it.
>
> I don't know witch server are you using, but on PostgreSQL, you have to set
> an
> option on the postgresql.conf file that allow your to dump by syslog all
> the
> DDL statements the server receives.
>
> I also beleive that kamailio, if you put it on debug level 6 or more, will
> dump the SQL statements it's sending out.
>
>
> --
> Raúl Alexis Betancor Santana
> Dimensión Virtual
>
hello all,
please any billing modules or billing application to use with Kamailio /
OpenSer?
i need a prepay / Postpay application sam to Asterisk to billing (A2Billing)
thanks for your help
__________ Information from ESET NOD32 Antivirus, version of virus signature database 4191 (20090626) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com