Hi Klaus and all other who read this.
It's not a problem of tcp socket - yesterday i rewrite gateway config for fowarding directed to extgateway requests to gateway itself. In tcp this problems not occured, but after i switched on tls, i see overfilled queues again. In that case i haven't extgateway, communications with remote sockets and etc. Did anybody test openser whith big payload? I think this problem occurs because in my case i have big traffic and several process that simultaniously use tls. Did anybody have the same problem?
Hi Serega!
I'm not sure - just a guess. openser uses "worker threads". This worker threads get a SIP message from the listener thread, process them, and then they send them.
If for some reason the sending fails - e.g. the gateway sends via TCP/TLS to extgateway and extgateway's receiving queue is full. Then gateway's sending buffer gets full and the threads wait until they can send. Thus, as the threads are all busy, also the receiving queue of gateway gets full. I guess after some time the threads give up sending and will read again.
The question is: why does the sending buffer gets full? Where is gateway sending the messages to?
The sending queue must not get flushed. The queue is in the OS, not in openser. As TCP guarantees that there is no loss/reordering the TCP stack must not flush the queue.
regards klaus
serega wrote:
Hi all.
I have next problem while using openser with tls. System consists of sip statefull server (in next time, simply, server) and stateless sip gateway (gateway). Server used for connect to jabber server. Server contains rewrote jabber module logic and also use presence module. Gateway used to connect to other sip gateway (extgateway) via tls protocol. Gateway use 1.1 openser version and doesn’t contain changed code. When server together with gateway restarts and server in its database contains above 260 subscriptions (in watcherinfo table) i have error. Socket used by gateway to connect with extgateway contains in receive and sent queues a lot of data (above 50kByte on each sides). This data never flushed out. This happens because after restart sip server through gateway send notifications to extgateway. When I attached using gdb to process that send data, I saw that it was in infinity loop because tls library returns SSL_ERROR_WANT_WRITE and I think it’s correct because we have overfilled send queue. In this case I don’t interest why receive queue not empty (I think it happens because sending process have got block on socket). But I don’t understand why sent queue not flushed. I test this behavior using tcp – all was correct. Socket not closed by other side because this status can be unchanged above one day. Anybody can help me? Why this happens?
I use openssl-0.9.8c and redhat os (Linux xdevel1 2.4.21-4.ELsmp #1 SMP Fri Oct 3 17:52:56 EDT 2003 i686 i686 i386 GNU/Linux). Computer have too Pentium 4 processors).
Gateway.cfg (sip.qa.hbex.com – gateway address):
# # $Id: router_qa.cfg,v 1.1 2006/08/02 18:14:34 ilya Exp $ # # simple quick-start config script #
# ----------- global configuration parameters
debug=9 # debug level (cmd line: -dddddddddd) log_facility=LOG_LOCAL0 fork=yes log_stderror=yes # (cmd line: -E)
check_via=no # (cmd. line: -v) dns=no # (cmd. line: -r) rev_dns=no # (cmd. line: -R) children=20 fifo="/tmp/openser_fifo" server_signature=no
fifo_db_url="mysql://root:@localhost/openser"
listen = udp:sip.qa.hbex.com:5060 #incoming sip server address listen = tcp:sip.qa.hbex.com:5060
#tls address disable_tls = 0 listen = tls:sip.qa.hbex.com:5061 listen = udp:sip.qa.hbex.com:5061
tls_certificate = "/ home/inop/ex.com-cert.pem" tls_private_key = "/home/inop/ex.com-privkey.pem" tls_ca_list = "/home/inop/ex.com-calist.pem" tls_require_client_certificate=0 tls_verify_client=0 tls_verify_server=0
# ------------------ module loading ---------------------------------- loadmodule "/home/interop/openser/lib/openser/modules/rr.so" loadmodule "/home/interop/openser/lib/openser/modules/xlog.so" loadmodule "/home/interop/openser/lib/openser/modules/textops.so" loadmodule "/home/interop/openser/lib/openser/modules/maxfwd.so" loadmodule "/home/interop/openser/lib/openser/modules/sl.so" loadmodule "/home/interop/openser/lib/openser/modules/mysql.so" loadmodule "/home/interop/openser/lib/openser/modules/tm_unchanged.so" loadmodule "/home/interop/openser/lib/openser/modules/usrloc.so" loadmodule "/home/interop/openser/lib/openser/modules/registrar.so"
# ----------------- setting module-specific parameters --------------- modparam("registrar", "default_expires", 120) modparam("registrar", "use_domain", 1)
modparam("usrloc", "use_domain", 1) modparam("usrloc", "db_mode", 0) # Uncomment this if you want to use SQL database # for persistent storage and comment the previous line #modparam("usrloc", "db_mode", 2)
# add value to ;lr param to make some broken UAs happy modparam("rr", "enable_full_lr", 1) modparam("maxfwd", "max_limit", 10) # ------------------------- request routing logic ------------------- # main routing logic route{ if (!mf_process_maxfwd_header("10")) { sl_send_reply("483","To Many Hops"); drop(); }; if (dst_port==5061) { if (search("^To:.*@.*cmnicator.com") || search("^To:.*@.*cihb.com")) { # rewrite destination and forward to jabber (sip server) route(1); return; }; sl_reply_error(); return; } if (method=="REGISTER") { xlog("XXX: saving location msg=$mb\n"); if(!save_noreply("location")) { log("XXX: Error saving location!\n"); sl_reply_error(); } sl_send_reply("200","OK"); return; } if (search("^To:.*@.*cmnicator.com") || search("^To:.*@.*cihb.com")) { #forward to self route(3); return; } ##forward to ext gateway route(2); }
route[1]{ if (method!="MESSAGE" && method!="SUBSCRIBE" && method!="NOTIFY") { log("XXX: Request not forwarded to sip server!\n"); sl_send_reply("202","Accepted"); return; } t_on_reply("1"); record_route();
if(!lookup("location")) { if (method!="SUBSCRIBE") { log("XXX: only subscribes are processed for user that are not registered\n"); sl_reply_error(); return; } subst_uri('/(.*)@(.*)/\1#\2*jabber.qa.im.hb.com@jabber01.qa.im.hb.com/ig'); if(!t_relay()) { log("XXX: error forwarding jabber01...\n"); sl_reply_error(); return; } else { sl_send_reply("200","OK"); return; } } else { log("XXX: forwarding to the address of record...\n"); if(!t_relay()) { log("XXX: error forwarding to address of record \n"); sl_reply_error(); return; } else { sl_send_reply("200", "OK"); return; } }
}
# forwarding to external gateway route[2]{ log("XXX: rewriting headers\n"); subst('/^(From:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
subst('/^(Contact:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
if(!forward("tls:43.123.141.166:3000")) { log("XXX: Error forwarding to external gateway!\n"); sl_reply_error(); } else { sl_send_reply("200", "Accepted"); }; }
# forwarding to SELF route[3]{ log("XXX: rewriting headers\n"); subst('/^(From:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
subst('/^(Contact:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
log("XXX: about to forward to self\n"); if(!forward("tcp:sip.qa.hbex.com:5061")) { log("XXX: Error forwarding to self!\n"); sl_reply_error(); } else { sl_send_reply("200", "Accepted"); }; }
# this is executed for replies onreply_route[1]{ subst('/^(To:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
subst('/^(Contact:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig'); route(2); }
_______________________________________________ Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Hello,
On 09/29/06 16:06, serega wrote:
Hi Klaus and all other who read this.
It's not a problem of tcp socket - yesterday i rewrite gateway config for fowarding directed to extgateway requests to gateway itself. In tcp this problems not occured, but after i switched on tls, i see overfilled queues again. In that case i haven't extgateway, communications with remote sockets and etc. Did anybody test openser whith big payload? I think this problem occurs because in my case i have big traffic and several process that simultaniously use tls. Did anybody have the same problem?
If you increase the number of the TLS/TCP workers do you get same results? In case of TLS, there is more CPU used because of encryption/decryption. How is the CPU usage in teh system while you are testing?
As Klaus said, the management of socket queue is done by OS.
Cheers, Daniel
Hi Serega!
I'm not sure - just a guess. openser uses "worker threads". This worker threads get a SIP message from the listener thread, process them, and then they send them.
If for some reason the sending fails - e.g. the gateway sends via TCP/TLS to extgateway and extgateway's receiving queue is full. Then gateway's sending buffer gets full and the threads wait until they can send. Thus, as the threads are all busy, also the receiving queue of gateway gets full. I guess after some time the threads give up sending and will read again.
The question is: why does the sending buffer gets full? Where is gateway sending the messages to?
The sending queue must not get flushed. The queue is in the OS, not in openser. As TCP guarantees that there is no loss/reordering the TCP stack must not flush the queue.
regards klaus
serega wrote:
Hi all.
I have next problem while using openser with tls. System consists of sip statefull server (in next time, simply, server) and stateless sip gateway (gateway). Server used for connect to jabber server. Server contains rewrote jabber module logic and also use presence module. Gateway used to connect to other sip gateway (extgateway) via tls protocol. Gateway use 1.1 openser version and doesn’t contain changed code. When server together with gateway restarts and server in its database contains above 260 subscriptions (in watcherinfo table) i have error. Socket used by gateway to connect with extgateway contains in receive and sent queues a lot of data (above 50kByte on each sides). This data never flushed out. This happens because after restart sip server through gateway send notifications to extgateway. When I attached using gdb to process that send data, I saw that it was in infinity loop because tls library returns SSL_ERROR_WANT_WRITE and I think it’s correct because we have overfilled send queue. In this case I don’t interest why receive queue not empty (I think it happens because sending process have got block on socket). But I don’t understand why sent queue not flushed. I test this behavior using tcp – all was correct. Socket not closed by other side because this status can be unchanged above one day. Anybody can help me? Why this happens?
I use openssl-0.9.8c and redhat os (Linux xdevel1 2.4.21-4.ELsmp #1 SMP Fri Oct 3 17:52:56 EDT 2003 i686 i686 i386 GNU/Linux). Computer have too Pentium 4 processors).
Gateway.cfg (sip.qa.hbex.com – gateway address):
# # $Id: router_qa.cfg,v 1.1 2006/08/02 18:14:34 ilya Exp $ # # simple quick-start config script #
# ----------- global configuration parameters
debug=9 # debug level (cmd line: -dddddddddd) log_facility=LOG_LOCAL0 fork=yes log_stderror=yes # (cmd line: -E)
check_via=no # (cmd. line: -v) dns=no # (cmd. line: -r) rev_dns=no # (cmd. line: -R) children=20 fifo="/tmp/openser_fifo" server_signature=no
fifo_db_url="mysql://root:@localhost/openser"
listen = udp:sip.qa.hbex.com:5060 #incoming sip server address listen = tcp:sip.qa.hbex.com:5060
#tls address disable_tls = 0 listen = tls:sip.qa.hbex.com:5061 listen = udp:sip.qa.hbex.com:5061
tls_certificate = "/ home/inop/ex.com-cert.pem" tls_private_key = "/home/inop/ex.com-privkey.pem" tls_ca_list = "/home/inop/ex.com-calist.pem" tls_require_client_certificate=0 tls_verify_client=0 tls_verify_server=0
# ------------------ module loading ---------------------------------- loadmodule "/home/interop/openser/lib/openser/modules/rr.so" loadmodule "/home/interop/openser/lib/openser/modules/xlog.so" loadmodule "/home/interop/openser/lib/openser/modules/textops.so" loadmodule "/home/interop/openser/lib/openser/modules/maxfwd.so" loadmodule "/home/interop/openser/lib/openser/modules/sl.so" loadmodule "/home/interop/openser/lib/openser/modules/mysql.so" loadmodule "/home/interop/openser/lib/openser/modules/tm_unchanged.so" loadmodule "/home/interop/openser/lib/openser/modules/usrloc.so" loadmodule "/home/interop/openser/lib/openser/modules/registrar.so"
# ----------------- setting module-specific parameters --------------- modparam("registrar", "default_expires", 120) modparam("registrar", "use_domain", 1)
modparam("usrloc", "use_domain", 1) modparam("usrloc", "db_mode", 0) # Uncomment this if you want to use SQL database # for persistent storage and comment the previous line #modparam("usrloc", "db_mode", 2)
# add value to ;lr param to make some broken UAs happy modparam("rr", "enable_full_lr", 1) modparam("maxfwd", "max_limit", 10) # ------------------------- request routing logic ------------------- # main routing logic route{ if (!mf_process_maxfwd_header("10")) { sl_send_reply("483","To Many Hops"); drop(); }; if (dst_port==5061) { if (search("^To:.*@.*cmnicator.com") || search("^To:.*@.*cihb.com")) { # rewrite destination and forward to jabber (sip server) route(1); return; }; sl_reply_error(); return; } if (method=="REGISTER") { xlog("XXX: saving location msg=$mb\n"); if(!save_noreply("location")) { log("XXX: Error saving location!\n"); sl_reply_error(); } sl_send_reply("200","OK"); return; } if (search("^To:.*@.*cmnicator.com") || search("^To:.*@.*cihb.com")) { #forward to self route(3); return; } ##forward to ext gateway route(2); }
route[1]{ if (method!="MESSAGE" && method!="SUBSCRIBE" && method!="NOTIFY") { log("XXX: Request not forwarded to sip server!\n"); sl_send_reply("202","Accepted"); return; } t_on_reply("1"); record_route();
if(!lookup("location")) { if (method!="SUBSCRIBE") { log("XXX: only subscribes are processed for user that are not registered\n"); sl_reply_error(); return; } subst_uri('/(.*)@(.*)/\1#\2*jabber.qa.im.hb.com@jabber01.qa.im.hb.com/ig'); if(!t_relay()) { log("XXX: error forwarding jabber01...\n"); sl_reply_error(); return; } else { sl_send_reply("200","OK"); return; } } else { log("XXX: forwarding to the address of record...\n"); if(!t_relay()) { log("XXX: error forwarding to address of record \n"); sl_reply_error(); return; } else { sl_send_reply("200", "OK"); return; } }
}
# forwarding to external gateway route[2]{ log("XXX: rewriting headers\n"); subst('/^(From:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
subst('/^(Contact:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
if(!forward("tls:43.123.141.166:3000")) { log("XXX: Error forwarding to external gateway!\n"); sl_reply_error(); } else { sl_send_reply("200", "Accepted"); }; }
# forwarding to SELF route[3]{ log("XXX: rewriting headers\n"); subst('/^(From:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
subst('/^(Contact:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
log("XXX: about to forward to self\n"); if(!forward("tcp:sip.qa.hbex.com:5061")) { log("XXX: Error forwarding to self!\n"); sl_reply_error(); } else { sl_send_reply("200", "Accepted"); }; }
# this is executed for replies onreply_route[1]{ subst('/^(To:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
subst('/^(Contact:[^@#]*)#([^@]*)*jabber.qa.im.hb.com@jabber...qa.im.hb.com(.*)/\1@\2\3/ig'); route(2); }
_______________________________________________ Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
-
Daniel-Constantin Mierla -
serega