25 Oct
2011
25 Oct
'11
5:11 p.m.
Hi All,
Just wondering, has anyone else experienced the behaviour described
below, where messages are still routed to a gateway that is in
Active-Probing state? From my understanding of the docs, when a
destination is in probing state, it should not be used in the
destination selection. I can see that Kamailio does set the state from
AX to AP (Active to Active-Probing) when a timeout occurs, and a "kamctl
dispatcher dump" does show that the state has changed to AP, however,
sip trace to the destination still shows messages being routed to the
destination that is in AP mode.
Just checking if anyone else is seeing this behaviour as well.
Thanks
On 24/10/2011 16:30, Asgaroth wrote:
OK, did a few more tests but have come accross
something, which I am
not sure is intended behaviour.
When setting a destingation as probing in failure route (due to
timeout), the destination still gets used in destination selection.
# ./kamailio -V
version: kamailio 3.3.0-dev0 (i386/linux) 25bedc
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 25bedc
compiled on 09:18:41 Oct 21 2011 with gcc 4.1.2
Dispatcher module parameters (in testing) are as follows
(SBC_PING_FROM is #defined previously):
modparam("dispatcher", "flags", 2)
modparam("dispatcher", "dst_avp", "$avp(AVP_DST)")
modparam("dispatcher", "grp_avp", "$avp(AVP_GRP)")
modparam("dispatcher", "cnt_avp", "$avp(AVP_CNT)")
modparam("dispatcher", "ds_ping_method", "OPTIONS")
modparam("dispatcher", "ds_ping_from", SBC_PING_FROM)
modparam("dispatcher", "ds_ping_interval", 10)
modparam("dispatcher", "ds_probing_threshhold", 1)
modparam("dispatcher", "ds_probing_mode", 0)
Main routing logic has following snippet to select destination (hash
table selects dispatcher setid based on request domain):
if(!ds_select_dst("$sht(which_sbc=>$rd)", "0")) {
sl_send_reply("500", "No destination
available");
xlog("route[MAIN] : $rm : No destinations
available for $rd");
exit;
}
Failure route has following logic to select next destination based on
timout/failure of destination:
if (t_branch_timeout() && !t_branch_replied())
{
xlog("route[TO_SBC] : $rm : timeout and no reply
($si:$sp->$Ri:$Rp->$du)\n");
xlog("route[TO_SBC] : $rm : setting $du to probing
state");
ds_mark_dst("p");
if(ds_next_dst())
{
xlog("route[TO_SBC] : $rm : next destination
select ($du)\n");
t_on_failure("TO_SBC");
t_relay();
exit;
} else {
send_reply("500", "No destination
available");
xlog("route[TO_SBC] : $rm : No destinations
available for $rd");
exit;
}
}
According to 3.2 module docs for dispatcher, when a destination is set
into probing state, it will not be used by ds_select_dst:
----
4.6. |ds_mark_dst("s")|
Mark the last used address from destination set as inactive
("i"/"I"/"0"), active
("a"/"A"/"1") or probing
("p"/"P"/"2"). With
this function, an automatic detection of failed gateways can be
implemented. When an address is marked as inactive or probing, it will
be ignored by 'ds_select_dst' and 'ds_select_domain'.
possible parameters:
*
/"i", "I" or "0"/ - the last destination should be set
to inactive
and will be ignored in future requests.
*
/"a", "A" or "1"/ - the last destination should be set
to active
and the error-counter should set to "0".
*
/"p", "P" or "2"/ - the last destination will be set to
probing.
Note: You will need to call this function "threshhold"-times,
before it will be actually set to probing.
This function can be used from REQUEST_ROUTE, FAILURE_ROUTE.
---
What happens here, for me, is:
[1] Gateway is in Active mode (state: AX).
[2] Request comes in and times out, destination is set to
Active/Probing (state: AP)
[3] Another request comes in and it selects gateway that is in AP
mode, times out, and then selects next dst in list.
[4] Another request comes in and it selects gateway that is in AP
mode, times out, and then selects next dst in list.
.
.
NOTE: The requests selecting the AP mode gateway may not be right
after each other (algorythm used is hash over callid) but I have
stripped those out in above steps. If I'm not mistaked, if 2
destination in a set, and 1 destination is marked as AP, then
remaining destination should always be selected as destination to send
to. The destination marked AP (active-probing) should not be selected
while in probing state.
When the gateway is set into AP mode at step [2], then, according to
docs, any new request coming in should not have the gateway selected
as it is marked as being in probing state.
Is this the intended behaviour or am I missing something in the
documentation, or is it a bug?
Thanks
On 22/10/2011 15:55, Daniel-Constantin Mierla wrote:
Hello,
yes, I will do it. Just happens to travel these days, but when I get
the time for it, I will do it.
Btw, did you test also the load balancing functionality? Was it
affected or all is ok when you change the states? I mean when you
disable/make inactive a gateway, is no longer used for routing,
right? Just double checking, to be sure it was not affected by this
change.
Thanks,
Daniel
On 10/22/11 3:42 PM, Asgaroth wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users Hi Daniel,
Just a reminder for this issue, to backport to 3.2.0 :)
Thanks
On 21/10/2011 10:53, Asgaroth wrote:
Hi Daniel,
It appears the change you made has fixed the issue. Below are my
tests:
# kamailio -V
version: kamailio 3.3.0-dev0 (i386/linux) 25bedc
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 25bedc
compiled on 09:18:41 Oct 21 2011 with gcc 4.1.2
Here we have the dispatcher state as loaded from db on startup in
kamailio memory.
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
Now try set state for destination to inactive (Worked)
# kamctl fifo ds_set_state i 1 sip:1.1.1.1:10001
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=IX priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
Now try set state for destination to active (Worked)
# kamctl fifo ds_set_state a 1 sip:1.1.1.1:10001
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
Now try set state for destination to inactive-probing (Worked)
# kamctl fifo ds_set_state ip 1 sip:1.1.1.1:10001
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
I waited for OPTIONS/INFO message and checked state (Worked), changed
state from IP -> AX
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
Now try set state to active-probing (Worked)
# kamctl fifo ds_set_state ap 1 sip:1.1.1.1:10001
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=AP priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
I waited for OPTIONS/INFO message and checked state (Worked), changed
state from AP -> AX
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
Now try set state to disabled (Worked). At this point no pinging
occured
as destination is admin down.
# kamctl fifo ds_set_state d 1 sip:1.1.1.1:10001
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=DX priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
Now try set state to inactive-probing again, from disabled state
(Worked)
# kamctl fifo ds_set_state ip 1 sip:1.1.1.1:10001
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
I waited for OPTIONS/INFO message and checked state (Worked), changed
state from IP -> AX
# kamctl dispatcher dump
SET_NO:: 2
SET:: 1
URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
SET:: 2
URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
Thanks.
On 20/10/2011 22:54, Daniel-Constantin Mierla wrote:
> Hello,
>
> indeed the setting of active state via MI command was wrong.
> Should be
> fixed now in master branch. Can you test it and see if works ok
> now (I
> had no option to test it myself yet). If all goes fine with your
> tests, then I will backport.
>
> Guidelines to install master branch can be found at:
> https://www.kamailio.org/wiki/install/devel/git
>
> Cheers,
> Daniel
>
>
> On 10/20/11 7:52 PM, Asgaroth wrote:
>> Hi All,
>>
>> Just been doing some testing with Kamailio 3.2 and the dispatcher
>> module, and have some strange behaviour, can we confirm if it is
>> expected behaviour or a bug of some sort.
>>
>> It appears that I can only set the dspatcher state via fifo command
>> once, and not reset it again, here is an example:
>>
>> version: kamailio 3.2.0 (i386/linux) 94d3b8
>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
>> USE_RAW_SOCKS,
>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
>> PKG_MALLOC,
>> DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
>> 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> id: 94d3b8
>> compiled on 12:34:43 Oct 20 2011 with gcc 4.1.2
>>
>>
>> Here we start with the dispatcher list as it was pulled from the
>> database and currently loaded in memory.
>>
>> # kamctl dispatcher dump
>> SET_NO:: 2
>> SET:: 1
>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>> SET:: 2
>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>
>> Now I try to set one gateway inactive:
>>
>> # kamctl fifo ds_set_state i 1 sip:1.1.1.1:10001
>> # kamctl dispatcher dump
>> SET_NO:: 2
>> SET:: 1
>> URI:: sip:1.1.1.1:10001 flags=IX priority=0 attrs=
>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>> SET:: 2
>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>
>> All normal so far, now if I try to set the same gateway active
>> again,
>> the state still stays inactive, i dont seem to be able to change the
>> state back to active. The same thing happens if I attempt to set
>> this
>> state to disabled and then try to set it back to active
>> afterwards. The
>> state does not change from what the first attempt was.
>>
>> # kamctl fifo ds_set_state a 1 sip:1.1.1.1:10001
>> # kamctl dispatcher dump
>> SET_NO:: 2
>> SET:: 1
>> URI:: sip:1.1.1.1:10001 flags=IX priority=0 attrs=
>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>> SET:: 2
>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>
>> Now I can try to set the state to inactive probing, that works.
>>
>> # kamctl fifo ds_set_state ip 1 sip:1.1.1.1:10001
>> # kamctl dispatcher dump
>> SET_NO:: 2
>> SET:: 1
>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>> SET:: 2
>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>
>> Now if I try to disable the same destination:
>>
>> # kamctl fifo ds_set_state d 1 sip:1.1.1.1:10001
>> # kamctl dispatcher dump
>> SET_NO:: 2
>> SET:: 1
>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>> SET:: 2
>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>
>> State still stays in Inactive/Probing mode.
>>
>> Now if I try to set the same gateway to active/probing mode, I cant:
>>
>> # kamctl fifo ds_set_state ap 1 sip:1.1.1.1:10001
>> # kamctl dispatcher dump
>> SET_NO:: 2
>> SET:: 1
>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>> SET:: 2
>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>
>> The only way for me to get the state back to active after the
>> above is
>> to do a reload on the dispatch table:
>>
>> # kamctl dispatcher reload
>> # kamctl dispatcher dump
>> SET_NO:: 2
>> SET:: 1
>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>> SET:: 2
>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>
>> Is the dispatcher expected to behave as above?
>>
>> I'm still trying to understand the different states assuming the
>> following options are set in the configuration file:
>>
>> modparam("dispatcher", "ds_ping_interval", 10)
>> modparam("dispatcher", "ds_probing_threshhold", 1)
>> modparam("dispatcher", "ds_probing_mode", 0)
>>
>> AX = Active/No Probing
>> (*) Gateway is used in ds_select_*, ds_next_*
>> (*) No OPTIONS/INFO messages are sent to destination to
>> check alive
>> state
>>
>> This appears to be the normal operating state with no issues
>> detected.
>>
>> AP = Active/Probing
>> (*) Gateway is NOT used in ds_select_*, ds_next_*
>> (*) OPTIONS/INFO messages ARE sent to destination to check
>> alive
>> state
>>
>> This appears to be the state taken when a gateway times out
>> and
>> ds_mark_dst("p") is set in failure route, the gateway is
"pinged"
>> while
>> down and set into AX mode once a response is recieved.
>>
>> IX = Inactive/No Probing
>> (*) Gateway is NOT used in ds_select_*, ds_next_*
>> (*) No OPTIONS/INFO messages sent to destination
>>
>> This appears to be the state taken when a gateway times out
>> and if
>> ds_mark_dst("i") is set in failure route, this gateway is set
>> inactive
>> and no "pinging" is performed.
>>
>> IP = Inactive/Probing
>> (*) Gateway is NOT used in ds_select_*, ds_next_*
>> (*) OPTIONS/INFO messages ARE sent to destination
>>
>> I'm not sure when this state would be reached unless it is
>> set by
>> fifo command. The ds_mark_dst function only allows "a","i"
or
>> "p". So
>> the probing method would only become active is ds_probing_mode =
>> 1. In
>> my case it is 0, so I'm not sure if I would ever reach this state
>> unless
>> I forced it. What would happen here is the gateway came back
>> alive once
>> in "IP" state, would it go back to "AX" state?
>>
>> DX = Disabled/No Probing
>> (*) Gateway is administratively set to down
>>
>> This is where manual intervention has occured and the
>> destination
>> has been set to administrativly down, for example, work being
>> carried
>> out on the destination gateway.
>>
>> All these states make sense, for the most part, I'm a little
>> unsure of
>> how IP/IX is meant to work unless you manually set it to
>> "inactive" in
>> your failure route.
>>
>> The main concern though is the the fifo commands dont appear to be
>> working as expected, I can only set the state for a destination
>> manually
>> once, and not again after that, unless I perform a dispatcher
>> reload.
>>
>> Has anyone else experienced this behaviour?
>>
>> Any comments/suggestions explanations are most welcome.
>>
>> Thanks
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>> mailing list
>> sr-users(a)lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users