Looking at the logs of last commits, I couldn't spot the change that
would add the leak.
What is the exact version you are running (kamailio -v)?
Are you using any of the functions exported by tcpops?
Cheers,
Daniel
On 17/11/15 15:24, Anthony Messina wrote:
I wish that were the case...
# kamcmd core.tcp_info
{
readers: 2
max_connections: 2048
max_tls_connections: 2048
opened_connections: 0
opened_tls_connections: 0
write_queued_bytes: 0
}
# kamcmd tls.info
{
max_connections: 2048
opened_connections: 0
clear_text_write_queued_bytes: 0
}
On Tuesday, November 17, 2015 03:08:59 PM Daniel-Constantin Mierla wrote:
Looks like a lot of connections being open, can
you get the output for:
kamcmd core.tcp_info
kamcmd tls.info
Cheers,
Daniel
On 17/11/15 14:59, Anthony Messina wrote:
> Attached. -A
>
> On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla wrote:
>> Can you run the following commands:
>>
>> kamcmd cfg.set_now_int core memlog 1
>> kamcmd corex.shm_summary
>>
>> Then grab the log messages from syslog related to shared memory summary
>> and send them over here.
>>
>> Cheers,
>> Daniel
>>
>> On 17/11/15 14:31, Anthony Messina wrote:
>>> After I reported last night, I restarted Kamailio and even though the 5
>>> UACs did nothing but ensure they had a registration overnight, this
>>> morning the issue has recurred. The following is the output you
>>> requested. Not sure how the memory is being used up by Kamailio.
>>>
>>> # kamctl stats shmem
>>> shmem:fragments = 181
>>> shmem:free_size = 8922584
>>> shmem:max_used_size = 58243792
>>> shmem:real_used_size = 58186280
>>> shmem:total_size = 67108864
>>> shmem:used_size = 54346088
>>>
>>> On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla
wrote:
>>>>> As you are using the master branch (development), do you run latest
>>>>> version?
>>>>>
>>>>> Can you look at available shared memory?
>>>>>
>>>>> kamctl stats shmem
>>>>>
>>>>> Check it over time and see if the free memory is decreasing.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> On 17/11/15 00:44, Anthony Messina wrote:
>>>>>> I have noticed the following issue which began with builds
somewhere
>>>>>> between git master commits bff0a08 and 6173ef7. I did not see
this
>>>>>> issue
>>>>>> with my previous builds and haven't been able to pin down the
problem,
>>>>>> which is why I haven't formally filed a bug.
>>>>>>
>>>>>> Any help or guidance is appreciated, because this has crippled my
use
>>>>>> of
>>>>>> Kamailio. Only a restart enables it to work again until the
issue
>>>>>> recurs.
>>>>>>
>>>>>> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug
#1491
>>>>>> workaround: not enough memory for safe operation: 8870536
>>>>>> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR:
tcp_read_req:
>>>>>> error
>>>>>> reading
>>>>>>
>>>>>> I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
>>>>>>
>>>>>> I have a very small operation and the only change on the
operational
>>>>>> side
>>>>>> is that all 5 of my mobile UACs (yes, that's all) have
switched from
>>>>>> CSipSimple/Android to Zoiper/Android, which doesn't yet have
support
>>>>>> for
>>>>>> client-side certificates so verify_certificate and
require_certificate
>>>>>> are
>>>>>> off for both the server and client config.
>>>>>>
>>>>>> The server is started with:
>>>>>> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
>>>>>>
>>>>>> I have tried modifying the shared mem to 128 but the issue still
>>>>>> occurs.
>>>>>>
>>>>>> Even right now, I am seeing the error when only one UAC has
>>>>>> established
>>>>>> a
>>>>>> TLS connection:
>>>>>>
>>>>>> # kamcmd tls.list
>>>>>> {
>>>>>>
>>>>>> id: 572
>>>>>> timeout: 3475
>>>>>> src_ip: 10.77.79.156
>>>>>> src_port: 58688
>>>>>> dst_ip: 10.77.79.3
>>>>>> dst_port: 5061
>>>>>> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA
Enc=RC4(128)
>>>>>> Mac=SHA1
>>>>>> ct_wq_size: 0
>>>>>> enc_rd_buf: 0
>>>>>> flags: 2
>>>>>> state: established
>>>>>>
>>>>>> }
>>>>>>
>>>>>> # kamailio.cfg
>>>>>> enable_tls=yes
>>>>>> loadmodule "tls.so"
>>>>>> modparam("tls", "connection_timeout", 60)
>>>>>> #modparam("tls", "tls_log", 1)
>>>>>> #modparam("tls", "tls_debug", 1)
>>>>>> #modparam("tls", "low_mem_threshold1", -1)
>>>>>> #modparam("tls", "low_mem_threshold2", 0)
>>>>>> modparam("tls", "session_cache", 1)
>>>>>>
>>>>>> # tls.cfg
>>>>>> [server:default]
>>>>>> method = TLSv1+
>>>>>> verify_certificate = no
>>>>>> require_certificate = no
>>>>>> private_key = /etc/kamailio/example.org.key.pem
>>>>>> certificate = /etc/kamailio/example.org.crt.pem
>>>>>> server_name =
example.org
>>>>>> cipher_list =
>>>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-
>>>>>>
AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-
>>>>>> AE
>>>>>> S
>>>>>> 256-
>>>>>>
SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM
>>>>>> -
>>>>>>
SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4
>>>>>> -
>>>>>>
SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128-
>>>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-
>>>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
>>>>>>
>>>>>> [client:default]
>>>>>> method = TLSv1+
>>>>>> verify_certificate = no
>>>>>> require_certificate = no
>>>>>> private_key = /etc/kamailio/example.org.key.pem
>>>>>> certificate = /etc/kamailio/example.org.crt.pem
>>>>>> cipher_list =
>>>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-
>>>>>>
AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-
>>>>>> AE
>>>>>> S
>>>>>> 256-
>>>>>>
SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM
>>>>>> -
>>>>>>
SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4
>>>>>> -
>>>>>>
SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128-
>>>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-
>>>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
>>>>>>
>>>>>>
>>>>>> Thanks. -Anthony