I'm trying to debug an issue I've encountered where a variable that I've set using kamcmd cfg.set has an unexpected value when used in kamailio. The variable should be 0 or 1, but it looks like its getting set accidently somehow.
2018-05-11T22:02:58.344260+00:00 ip-172-31-129-45 kamailio[5253]: DEBUG: <core> [select.c:263]: resolve_select(): 'with_dynamodb_user_data'
2018-05-11T22:02:58.344264+00:00 ip-172-31-129-45 kamailio[5253]: DEBUG: <core> [cfg/cfg_select.c:174]: select_cfg_var(): DEBUG: select_cfg_var(): select fixup is postponed: features.with_dynamodb_user_data
2018-05-11T22:05:03.467110+00:00 ip-172-31-129-45 kamailio[5551]: INFO: <core> [cfg/cfg_ctx.c:608]: cfg_set_now(): INFO: cfg_set_now(): features.with_dynamodb_user_data has been changed to 1
2018-05-11T22:15:51.695118+00:00 ip-172-31-129-45 kamailio[5574]: [2c03c333-1705-41b1-aa26-6f10492a5a6f] DEBUG: with_dynamodb_user_data is 1157627905
I've tried to debug using gdb using breaks and set a watch on the memory address of the var, but it looks like it keeps using different memory locations. The value in the address pointed at by group.handle seems to change for each invocation of set_cfg_now, so when I try and put a watch on the address of *group.handle + var.offset it never gets hit.
(gdb) print *group $36 = {num = 25, mapping = 0x7f56ee4474f0, vars = 0x0, add_var = 0x0, size = 164, meta_offset = 968, var_offset = 984, handle = 0x7f56edf92fd0, orig_handle = 0x0, dynamic = 1 '\001', next = 0x0, name_len = 8, name = "f"}
(gdb) print *var $62 = {def = 0x7f56ee471668, name_len = 23, pos = 24, offset = 164, flag = 0}
Any ideas?
Malcolm
Hello,
can you provide more details about what you think is going wrong?
The value of this variable is updated by each process when that process does some particular operations (e.g., receiving a sip message), it is not propagated automatically to every kamailio process when you do the update via kamcmd.
Cheers, Daniel
On 14.05.18 19:43, Malcolm O'Hare wrote:
I'm trying to debug an issue I've encountered where a variable that I've set using kamcmd cfg.set has an unexpected value when used in kamailio. The variable should be 0 or 1, but it looks like its getting set accidently somehow.
2018-05-11T22:02:58.344260+00:00 ip-172-31-129-45 kamailio[5253]: DEBUG: <core> [select.c:263]: resolve_select(): 'with_dynamodb_user_data'
2018-05-11T22:02:58.344264+00:00 ip-172-31-129-45 kamailio[5253]: DEBUG: <core> [cfg/cfg_select.c:174]: select_cfg_var(): DEBUG: select_cfg_var(): select fixup is postponed: features.with_dynamodb_user_data
2018-05-11T22:05:03.467110+00:00 ip-172-31-129-45 kamailio[5551]: INFO: <core> [cfg/cfg_ctx.c:608]: cfg_set_now(): INFO: cfg_set_now(): features.with_dynamodb_user_data has been changed to 1
2018-05-11T22:15:51.695118+00:00 ip-172-31-129-45 kamailio[5574]: [2c03c333-1705-41b1-aa26-6f10492a5a6f] DEBUG: with_dynamodb_user_data is 1157627905
I've tried to debug using gdb using breaks and set a watch on the memory address of the var, but it looks like it keeps using different memory locations. The value in the address pointed at by group.handle seems to change for each invocation of set_cfg_now, so when I try and put a watch on the address of *group.handle + var.offset it never gets hit.
(gdb) print *group $36 = {num = 25, mapping = 0x7f56ee4474f0, vars = 0x0, add_var = 0x0, size = 164, meta_offset = 968, var_offset = 984, handle = 0x7f56edf92fd0, orig_handle = 0x0, dynamic = 1 '\001', next = 0x0, name_len = 8, name = "f"}
(gdb) print *var $62 = {def = 0x7f56ee471668, name_len = 23, pos = 24, offset = 164, flag = 0}
Any ideas?
Malcolm
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
So what actually happens when you do kamcmd cfg.set ? I assumed the variable is set in shared memory somewhere and all the processes reference the same shared memory variable.
What I'm seeing is that I set the value to 1 using cfg.set and then some time later I do
sbin/kamctl kamcmd cfg.get features with_dynamodb_user_data
and I see the value being some rather large integer, which is not equal to 1. Inside my kamailio.cfg code, I only ever use that value in if statements so it should never be changed.
example:
if (@cfg_get.features.with_dynamodb_user_data == 1)
When I attach gdb to the kamailio process that handles kamctl, and break inside the set_cfg_now function, I can see that the actual address where the variable is being saved to changes, so it makes me wonder if there is some race condition where a process could try and read the value of the variable while its memory location is being changed? Otherwise the only thing I can think of is that there is some sort of buffer overflow happening when writing and its changing the value of this variable as a consequence.
Thoughts?
From: Daniel-Constantin Mierla miconda@gmail.com Sent: Friday, May 18, 2018 8:59 AM To: Kamailio (SER) - Users Mailing List; Malcolm O'Hare Subject: Re: [SR-Users] Help on how to debug cfg_get variable having unexpected value?
Hello,
can you provide more details about what you think is going wrong?
The value of this variable is updated by each process when that process does some particular operations (e.g., receiving a sip message), it is not propagated automatically to every kamailio process when you do the update via kamcmd.
Cheers, Daniel
On 14.05.18 19:43, Malcolm O'Hare wrote: I'm trying to debug an issue I've encountered where a variable that I've set using kamcmd cfg.set has an unexpected value when used in kamailio. The variable should be 0 or 1, but it looks like its getting set accidently somehow.
2018-05-11T22:02:58.344260+00:00 ip-172-31-129-45 kamailio[5253]: DEBUG: <core> [select.c:263]: resolve_select(): 'with_dynamodb_user_data'
2018-05-11T22:02:58.344264+00:00 ip-172-31-129-45 kamailio[5253]: DEBUG: <core> [cfg/cfg_select.c:174]: select_cfg_var(): DEBUG: select_cfg_var(): select fixup is postponed: features.with_dynamodb_user_data
2018-05-11T22:05:03.467110+00:00 ip-172-31-129-45 kamailio[5551]: INFO: <core> [cfg/cfg_ctx.c:608]: cfg_set_now(): INFO: cfg_set_now(): features.with_dynamodb_user_data has been changed to 1
2018-05-11T22:15:51.695118+00:00 ip-172-31-129-45 kamailio[5574]: [2c03c333-1705-41b1-aa26-6f10492a5a6f] DEBUG: with_dynamodb_user_data is 1157627905
I've tried to debug using gdb using breaks and set a watch on the memory address of the var, but it looks like it keeps using different memory locations. The value in the address pointed at by group.handle seems to change for each invocation of set_cfg_now, so when I try and put a watch on the address of *group.handle + var.offset it never gets hit.
(gdb) print *group $36 = {num = 25, mapping = 0x7f56ee4474f0, vars = 0x0, add_var = 0x0, size = 164, meta_offset = 968, var_offset = 984, handle = 0x7f56edf92fd0, orig_handle = 0x0, dynamic = 1 '\001', next = 0x0, name_len = 8, name = "f"}
(gdb) print *var $62 = {def = 0x7f56ee471668, name_len = 23, pos = 24, offset = 164, flag = 0}
Any ideas?
Malcolm
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.comhttp://www.kamailioworld.com
-
Daniel-Constantin Mierla -
Malcolm O'Hare