Hi everyone,
we're trying to make work usrloc over dmq without a db on our nodes.
Our `kamailio.cfg` code to make it work on the nodes is like this: ``` ... loadmodule "usrloc.so" ... loadmodule "dmq.so" loadmodule "dmq_usrloc.so" ... modparam("dmq", "server_address", DMQ_ADDRESS) modparam("dmq", "notification_address", "sip:routers.ourcloud.com:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq_usrloc", "enable", 1) modparam("dmq_usrloc", "sync", 1) ```
Where `DMQ_ADDRESS="sip:<some_address>:5060"` is the public IP address of the running node.
`nslookup` of `routers.ourcloud.com`: ```➜ ~ nslookup routers.ourcloud.com Server: 8.8.8.8 Address: 8.8.8.8#53
Non-authoritative answer: Name: rrouters.ourcloud.com Address: <IP1> Name: routers.ourcloud.com Address: <IP2> ``` and obviously IP1 and IP2 are pubblic IPs of kamailio nodes that respond to my pings.
When i register a client to node IP1 it's ul dump is: ``` { "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [{ "Info": { "AoR": "0409828030", "HashID": 1970914268, "Contacts": [{ "Contact": { "Address": "sip:0409828030@172.19.0.255:56328;ob", "Expires": 289, "Q": -1, "Call-ID": "5mclwiX4oYAVK0z-iT7M0p9RBX5-02g6", "CSeq": 59427, "User-Agent": "Telephone 1.2.6", "Received": "sip:<PROXY_IP>:5060", "Path": "[not set]", "State": "CS_NEW", "Flags": 0, "CFlags": 192, "Socket": "udp:<IP1>:5060", "Methods": 8159, "Ruid": "uloc-599d8fa8-f-1", "Instance": "[not set]", "Reg-Id": 0, "Last-Keepalive": 1503498176, "Last-Modified": 1503498176 } }] } }], "Stats": { "Records": 1, "Max-Slots": 1 } }, "id": 1 } ```
On node with IP2 the ul dump is: ``` { "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [], "Stats": { "Records": 0, "Max-Slots": 0 } }, "id": 1 } ``` What are we doing wrong? Are we missing something? -- Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
Hello,
Which version of Kamailio?
What is the output of 'kamcmd dmq.list_nodes'?
Cheers,
Charles
On 24 August 2017 at 11:02, Aleksandar Sosic alex.sosic@timenet.it wrote:
Hi everyone,
we're trying to make work usrloc over dmq without a db on our nodes.
Our `kamailio.cfg` code to make it work on the nodes is like this:
... loadmodule "usrloc.so" ... loadmodule "dmq.so" loadmodule "dmq_usrloc.so" ... modparam("dmq", "server_address", DMQ_ADDRESS) modparam("dmq", "notification_address", "sip:routers.ourcloud.com:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq_usrloc", "enable", 1) modparam("dmq_usrloc", "sync", 1)Where `DMQ_ADDRESS="sip:<some_address>:5060"` is the public IP address of the running node.
`nslookup` of `routers.ourcloud.com`:
Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: rrouters.ourcloud.com Address: <IP1> Name: routers.ourcloud.com Address: <IP2>and obviously IP1 and IP2 are pubblic IPs of kamailio nodes that respond to my pings.
When i register a client to node IP1 it's ul dump is:
{ "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [{ "Info": { "AoR": "0409828030", "HashID": 1970914268, "Contacts": [{ "Contact": { "Address": "sip:0409828030@172.19.0.255:56328;ob", "Expires": 289, "Q": -1, "Call-ID": "5mclwiX4oYAVK0z-iT7M0p9RBX5-02g6", "CSeq": 59427, "User-Agent": "Telephone 1.2.6", "Received": "sip:<PROXY_IP>:5060", "Path": "[not set]", "State": "CS_NEW", "Flags": 0, "CFlags": 192, "Socket": "udp:<IP1>:5060", "Methods": 8159, "Ruid": "uloc-599d8fa8-f-1", "Instance": "[not set]", "Reg-Id": 0, "Last-Keepalive": 1503498176, "Last-Modified": 1503498176 } }] } }], "Stats": { "Records": 1, "Max-Slots": 1 } }, "id": 1 }On node with IP2 the ul dump is:
{ "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [], "Stats": { "Records": 0, "Max-Slots": 0 } }, "id": 1 }What are we doing wrong? Are we missing something?
Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
So the Kamailio version is: root@sr01:/# kamailio -v version: kamailio 5.0.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 5.3.1
the output of dmq.list_nodes is: { "jsonrpc": "2.0", "result": [{ "host": "<PUB_IP_2>", "port": "5060", "resolved_ip": "<PUB_IP_2>", "status": 2, "last_notification": 0, "local": 0 }, { "host": "<PUB_IP_1>", "port": "5060", "resolved_ip": "<PUB_IP_1>", "status": 2, "last_notification": 0, "local": 1 }], "id": 1 }
Thanks, -- Aleksandar Sošić alex.sosic(at)gmail<dot>com
On Fri, Aug 25, 2017 at 11:59 AM, Charles Chance charles.chance@sipcentric.com wrote:
Hello,
Which version of Kamailio?
What is the output of 'kamcmd dmq.list_nodes'?
Cheers,
Charles
On 24 August 2017 at 11:02, Aleksandar Sosic alex.sosic@timenet.it wrote:
Hi everyone,
we're trying to make work usrloc over dmq without a db on our nodes.
Our `kamailio.cfg` code to make it work on the nodes is like this:
... loadmodule "usrloc.so" ... loadmodule "dmq.so" loadmodule "dmq_usrloc.so" ... modparam("dmq", "server_address", DMQ_ADDRESS) modparam("dmq", "notification_address", "sip:routers.ourcloud.com:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq_usrloc", "enable", 1) modparam("dmq_usrloc", "sync", 1)Where `DMQ_ADDRESS="sip:<some_address>:5060"` is the public IP address of the running node.
`nslookup` of `routers.ourcloud.com`:
Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: rrouters.ourcloud.com Address: <IP1> Name: routers.ourcloud.com Address: <IP2>and obviously IP1 and IP2 are pubblic IPs of kamailio nodes that respond to my pings.
When i register a client to node IP1 it's ul dump is:
{ "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [{ "Info": { "AoR": "0409828030", "HashID": 1970914268, "Contacts": [{ "Contact": { "Address": "sip:0409828030@172.19.0.255:56328;ob", "Expires": 289, "Q": -1, "Call-ID": "5mclwiX4oYAVK0z-iT7M0p9RBX5-02g6", "CSeq": 59427, "User-Agent": "Telephone 1.2.6", "Received": "sip:<PROXY_IP>:5060", "Path": "[not set]", "State": "CS_NEW", "Flags": 0, "CFlags": 192, "Socket": "udp:<IP1>:5060", "Methods": 8159, "Ruid": "uloc-599d8fa8-f-1", "Instance": "[not set]", "Reg-Id": 0, "Last-Keepalive": 1503498176, "Last-Modified": 1503498176 } }] } }], "Stats": { "Records": 1, "Max-Slots": 1 } }, "id": 1 }On node with IP2 the ul dump is:
{ "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [], "Stats": { "Records": 0, "Max-Slots": 0 } }, "id": 1 }What are we doing wrong? Are we missing something?
Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Charles Chance Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello,
Sorry for slow reply.
Can you set debug=4 and send the output from startup through to register, for both nodes?
Also the full config would be useful, or at least the default request route up to your call to dmq_handle_message().
Cheers,
Charles
On 28 Aug 2017 12:29, "Aleksandar Sosic" alex.sosic@gmail.com wrote:
So the Kamailio version is: root@sr01:/# kamailio -v version: kamailio 5.0.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 5.3.1
the output of dmq.list_nodes is: { "jsonrpc": "2.0", "result": [{ "host": "<PUB_IP_2>", "port": "5060", "resolved_ip": "<PUB_IP_2>", "status": 2, "last_notification": 0, "local": 0 }, { "host": "<PUB_IP_1>", "port": "5060", "resolved_ip": "<PUB_IP_1>", "status": 2, "last_notification": 0, "local": 1 }], "id": 1 }
Thanks,
Aleksandar Sošić alex.sosic(at)gmail<dot>com
On Fri, Aug 25, 2017 at 11:59 AM, Charles Chance charles.chance@sipcentric.com wrote:
Hello,
Which version of Kamailio?
What is the output of 'kamcmd dmq.list_nodes'?
Cheers,
Charles
On 24 August 2017 at 11:02, Aleksandar Sosic alex.sosic@timenet.it
wrote:
Hi everyone,
we're trying to make work usrloc over dmq without a db on our nodes.
Our `kamailio.cfg` code to make it work on the nodes is like this:
... loadmodule "usrloc.so" ... loadmodule "dmq.so" loadmodule "dmq_usrloc.so" ... modparam("dmq", "server_address", DMQ_ADDRESS) modparam("dmq", "notification_address", "sip:routers.ourcloud.com:5060")
modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq_usrloc", "enable", 1) modparam("dmq_usrloc", "sync", 1)
Where `DMQ_ADDRESS="sip:<some_address>:5060"` is the public IP address of the running node. `nslookup` of `routers.ourcloud.com`: ```➜ ~ nslookup routers.ourcloud.com Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: rrouters.ourcloud.com Address: <IP1> Name: routers.ourcloud.com Address: <IP2>and obviously IP1 and IP2 are pubblic IPs of kamailio nodes that respond to my pings.
When i register a client to node IP1 it's ul dump is:
{ "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [{ "Info": { "AoR": "0409828030", "HashID": 1970914268, "Contacts": [{ "Contact": { "Address": "sip:0409828030@172.19.0.255:56328;ob", "Expires": 289, "Q": -1, "Call-ID": "5mclwiX4oYAVK0z-iT7M0p9RBX5-02g6", "CSeq": 59427, "User-Agent": "Telephone 1.2.6", "Received": "sip:<PROXY_IP>:5060", "Path": "[not set]", "State": "CS_NEW", "Flags": 0, "CFlags": 192, "Socket": "udp:<IP1>:5060", "Methods": 8159, "Ruid": "uloc-599d8fa8-f-1", "Instance": "[not set]", "Reg-Id": 0, "Last-Keepalive": 1503498176, "Last-Modified": 1503498176 } }] } }], "Stats": { "Records": 1, "Max-Slots": 1 } }, "id": 1 }On node with IP2 the ul dump is:
{ "jsonrpc": "2.0", "result": { "Domain": "location", "Size": 1024, "AoRs": [], "Stats": { "Records": 0, "Max-Slots": 0 } }, "id": 1 }What are we doing wrong? Are we missing something?
Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Charles Chance Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt
Street,
Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Charles,
I did solve the problem with
if(is_method("KDMQ")) { dmq_handle_message(); }
in the main route and the dmq_usrloc is working now.
I enabled dmq for the dialog also: modparam("dialog", "enable_dmq", 1)
and the dmq is working between kamailio instances but the call does not end and the kamailio instance where the call was made crashes.
sr01.evox.it | 30(38) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 24 sr01.evox.it | 0(8) ALERT: <core> [main.c:742]: handle_sigs(): child process 18 exited by a signal 11 sr01.evox.it | 0(8) ALERT: <core> [main.c:745]: handle_sigs(): core was generated sr01.evox.it | 0(8) INFO: <core> [main.c:768]: handle_sigs(): terminating due to SIGCHLD sr01.evox.it | 1(9) INFO: <core> [main.c:823]: sig_usr(): signal 15 received sr01.evox.it | 20(28) INFO: <core> [main.c:823]: sig_usr(): signal 15 received sr01.evox.it | 11(19) INFO: <core> [main.c:823]: sig_usr(): signal 15 received
I have also done a gdb on the core dump: root@sr01:/tmp/sr01# gdb /usr/sbin/kamailio ./core GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...(no debugging symbols found)...done. [New LWP 8] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -DD -E -e -w /tmp/sr01'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f9403a3f428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
But cannot manage to solve the issue now.
NB. I have this order of modules loades: ... loadmodule "dmq.so" loadmodule "dmq_usrloc.so"
loadmodule "htable.so"
loadmodule "dialog.so" ...
# ----- dialog params ----- # modparam("dialog", "db_url", DBURL) modparam("dialog", "default_timeout", 10800) modparam("dialog", "profiles_with_value", "calllimit") modparam("dialog", "enable_stats", 1) modparam("dialog", "db_mode", 0) modparam("dialog", "dlg_flag", FLT_DLG) modparam("dialog", "enable_dmq", 1) ...
Do you have any hints?
Thanks in advance, -- Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
On Mon, Sep 4, 2017 at 10:47 AM, Charles Chance charles.chance@sipcentric.com wrote:
Hello,
Sorry for slow reply.
Can you set debug=4 and send the output from startup through to register, for both nodes?
Also the full config would be useful, or at least the default request route up to your call to dmq_handle_message().
Cheers,
Charles
Hi,
Glad you solved it - omitting the call to dmq_handle_message() appears to be a common mistake. I may try to make it clearer in the docs that things will not work as expected if this is not included.
As for the dialog/dmq integration, I've not had a chance to test it myself. I will try to take a look later this week!
Cheers,
Charles
On 5 September 2017 at 16:09, Aleksandar Sosic alex.sosic@timenet.it wrote:
Hi Charles,
I did solve the problem with
if(is_method("KDMQ")) { dmq_handle_message(); }
in the main route and the dmq_usrloc is working now.
I enabled dmq for the dialog also: modparam("dialog", "enable_dmq", 1)
and the dmq is working between kamailio instances but the call does not end and the kamailio instance where the call was made crashes.
sr01.evox.it | 30(38) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 24 sr01.evox.it | 0(8) ALERT: <core> [main.c:742]: handle_sigs(): child process 18 exited by a signal 11 sr01.evox.it | 0(8) ALERT: <core> [main.c:745]: handle_sigs(): core was generated sr01.evox.it | 0(8) INFO: <core> [main.c:768]: handle_sigs(): terminating due to SIGCHLD sr01.evox.it | 1(9) INFO: <core> [main.c:823]: sig_usr(): signal 15 received sr01.evox.it | 20(28) INFO: <core> [main.c:823]: sig_usr(): signal 15 received sr01.evox.it | 11(19) INFO: <core> [main.c:823]: sig_usr(): signal 15 received
I have also done a gdb on the core dump: root@sr01:/tmp/sr01# gdb /usr/sbin/kamailio ./core GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl. html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...(no debugging symbols found)...done. [New LWP 8] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -DD -E -e -w /tmp/sr01'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f9403a3f428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
But cannot manage to solve the issue now.
NB. I have this order of modules loades: ... loadmodule "dmq.so" loadmodule "dmq_usrloc.so"
loadmodule "htable.so"
loadmodule "dialog.so" ...
# ----- dialog params ----- # modparam("dialog", "db_url", DBURL) modparam("dialog", "default_timeout", 10800) modparam("dialog", "profiles_with_value", "calllimit") modparam("dialog", "enable_stats", 1) modparam("dialog", "db_mode", 0) modparam("dialog", "dlg_flag", FLT_DLG) modparam("dialog", "enable_dmq", 1) ...
Do you have any hints?
Thanks in advance,
Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
On Mon, Sep 4, 2017 at 10:47 AM, Charles Chance charles.chance@sipcentric.com wrote:
Hello,
Sorry for slow reply.
Can you set debug=4 and send the output from startup through to register, for both nodes?
Also the full config would be useful, or at least the default request
route
up to your call to dmq_handle_message().
Cheers,
Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Any updates about this problem? -- Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
On Tue, Sep 5, 2017 at 5:32 PM, Charles Chance charles.chance@sipcentric.com wrote:
Hi,
Glad you solved it - omitting the call to dmq_handle_message() appears to be a common mistake. I may try to make it clearer in the docs that things will not work as expected if this is not included.
As for the dialog/dmq integration, I've not had a chance to test it myself. I will try to take a look later this week!
Cheers,
Charles
Hi,
still haven't managed to solve this problem. Kamailio crashes on call hangup so I do think this is a serious problem.
Kind regards,
-- Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
On Fri, Sep 8, 2017 at 10:08 PM, Aleksandar Sosic alex.sosic@timenet.it wrote:
Any updates about this problem?
Aleksandar Sosic mail: alex.sosic@timenet.it skype: alex.sosic cell: +385 91 2505 146
On Tue, Sep 5, 2017 at 5:32 PM, Charles Chance charles.chance@sipcentric.com wrote:
Hi,
Glad you solved it - omitting the call to dmq_handle_message() appears
to be
a common mistake. I may try to make it clearer in the docs that things
will
not work as expected if this is not included.
As for the dialog/dmq integration, I've not had a chance to test it
myself.
I will try to take a look later this week!
Cheers,
Charles
-
Aleksandar Sosic -
Aleksandar Sosic -
Charles Chance