sr-users

sr-users@lists.kamailio.org

14 participants
37527 discussions

Registrar / Path
by Jawaid Bazyar 31 Oct '23

31 Oct '23

Hi, I have a scenario where I have an edge proxy (named RS), which locally stores registrations from Endpoints in its USRLOC, but which then also forward's REGISTER requests on to another proxy (named CN), which stores registrations in its USRLOC. (which then does DMQ with another CN like it). Both RS and CN are storing Path information, for example: kamctl ul show displays on both RS and CN: "Path": "sip:vs-rs01.blah.foo.bar;lr;received=sip:3.4.5.6:33577", So I know the path info is in there. I want CN , when presented with an INVITE, to reply with a 302 Redirect. Right now I have this: if (!lookup("location",sip:$tU@nodomain)) { } reg_fetch_contacts("location", "$tu", "called"); xlog("location info $ulc(called=>path)"); sl_send_reply("302","Redirect"); and the xlog does display the path. Good, so I have it being sent all the way through RS and CN and into CN’s USRLOC. but this isn’t complete. I am now faced with two issues. One, I need to mangle the Path into a Contact header for the Redirect. Two, I need to delete the Contact header that sl_send_reply is putting on there, which is the Contact as RS sees it from the endpoint. This needs to be removed and replaced with a Contact of only the RS server. It feels like there ought to be a simpler way to do this than parsing the Path string and reassembling a Contact from it, but if there is it's not readily apparent. Both Google and chatgpt have failed me :) Thanks in advance, Jawaid

3 7

Kamailio 5.7 still segfaulting on TLS with Ubuntu Jammy
by Benoît Panizzon 31 Oct '23

31 Oct '23

Hi Gang I just updated our dev plattform from 5.6 to 5.7 5.5 was working fine with TLS under Ubuntu Focal. 5.6 under Jammy started to dump cores. 5.7 still shows the same behaviour, usually shortly after a restart. Oct 20 09:45:01 dev-cpereg01 kamailio[11925]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f1bb0dd4bd8), called from tls: tls_init.c: ser_free(405), first free tls: tls_init.c: ser_free(405) - ignoring Oct 20 09:45:11 dev-cpereg01 kamailio[11905]: ALERT: <core> [main.c:776]: handle_sigs(): child process 11924 exited by a signal 11 Oct 20 09:45:11 dev-cpereg01 kamailio[11905]: ALERT: <core> [main.c:779]: handle_sigs(): core was generated Could anyone point me the correct direction on how to fix this issue? Am I the only one seeing this issue? -- Mit freundlichen Grüssen -Benoît Panizzon- @ HomeOffice und normal erreichbar -- I m p r o W a r e A G - Leiter Commerce Kunden ______________________________________________________ Zurlindenstrasse 29 Tel +41 61 826 93 00 CH-4133 Pratteln Fax +41 61 826 93 01 Schweiz Web http://www.imp.ch ______________________________________________________

2 4

RPM Repos Maintenance Needed
by tyler moore 30 Oct '23

30 Oct '23

Hi All, I started a discussion on the matrix channel about some 404 RPM repos and wanted to bring it here to further discuss. We typically configure our various RPM-based package managers to allow patch updates from the repos. As an example, if trying to install 5.7.x, can be done as follows: |yum -y install yum-utils yum-config-manager --add-repo https://rpm.kamailio.org/centos/kamailio.repo yum-config-manager --disable \* yum-config-manager --enable kamailio-5.7 yum install kamailio | This will fail though because https://rpm.kamailio.org/centos/7/5.7/5.7/x86_64/ does not exist. Throughout the repos for centos/rhel/fedora I found the existence of packages is inconsistent. We can tell the intention was to allow the above behavior as the kamailio.repo comes with the above configuration. The above example will update /etc/yum.repos.d/kamailio.repo as follows: |[kamailio-5.7] name=Kamailio - 5.7 - Packages for the latest Kamailio 5.7 release baseurl=https://rpm.kamailio.org/centos/$releasever/5.7/5.7/$basearch/ enabled=1 metadata_expire=30d repo_gpgcheck=0 gpgkey=https://rpm.kamailio.org/rpm-pub.key type=rpm skip_if_unavailable=True | That is for centos though, and I found that some of the other distros have incomplete kamailio.repo files as well. In example, the rhel kamailio.repo is missing separate entries for 5.7, but it does exist in the centos version. Looks like some small cleanup is in order to make the end user experience more consistent. I believe sergey was maintaining those repos last, please chime in here if you can. If needed I can pick up some of those responsibilities as well. -- Regards, *Tyler Moore* <https://devopsec.net/> Full Stack Software Engineer <http://www.google.com/search?q=full+stack+software+engineer+job+description> Flyball Labs <http://goflyball.com/> Office: 888-907-2085, ext: 34 <tel:8889072085;ext=34> Cell: 248-909-2769 <tel:2489092769> Email: tmoore(a)goflyball.com FLYBALL TECHNICAL SOLUTIONS, LLC <http://flyballlabs.com/> -- Tyler Moore Regards, *Tyler Moore* <https://devopsec.net/> Full Stack Software Engineer <http://www.google.com/search?q=full+stack+software+engineer+job+description> Flyball Labs <http://goflyball.com/> Office: 888-907-2085, ext: 34 <tel:8889072085;ext=34> Cell: 248-909-2769 <tel:2489092769> Email: tmoore(a)goflyball.com FLYBALL TECHNICAL SOLUTIONS, LLC <http://flyballlabs.com/>

3 2

RPM Repos Maintenance Needed
by tyler moore 28 Oct '23

28 Oct '23

Hi All, I started a discussion on the matrix channel about some 404 RPM repos and wanted to bring it here to further discuss. We typically configure our various RPM-based package managers to allow patch updates from the repos. As an example, if trying to install 5.7.x, can be done as follows: yum -y install yum-utils yum-config-manager --add-repo https://rpm.kamailio.org/centos/kamailio.repo yum-config-manager --disable \* yum-config-manager --enable kamailio-5.7 yum install kamailio This will fail though because https://rpm.kamailio.org/centos/7/5.7/5.7/x86_64/ does not exist. Throughout the repos for centos/rhel/fedora I found the existence of packages is inconsistent. We can tell the intention was to allow the above behavior as the kamailio.repo comes with the above configuration. The above example will update /etc/yum.repos.d/kamailio.repo as follows: [kamailio-5.7] name=Kamailio - 5.7 - Packages for the latest Kamailio 5.7 release baseurl=https://rpm.kamailio.org/centos/$releasever/5.7/5.7/$basearch/ enabled=1 metadata_expire=30d repo_gpgcheck=0 gpgkey=https://rpm.kamailio.org/rpm-pub.key type=rpm skip_if_unavailable=True That is for centos though, and I found that some of the other distros have incomplete kamailio.repo files as well. In example, the rhel kamailio.repo is missing separate entries for 5.7, but it does exist in the centos version. Looks like some small cleanup is in order to make the end user experience more consistent. I believe sergey was maintaining those repos last, please chime in here if you can. If needed I can pick up some of those responsibilities as well. Regards, Tyler Moore Full Stack Software Engineer Flyball Labs Office: 888-907-2085, ext: 34 Cell: 248-909-2769 Email: tmoore(a)goflyball.com

2 2

Kamailio 5.4.3 Troubleshooting Traffic Blocking Issue After Docker and OpenSSL Update
by Pintu Lohar 27 Oct '23

27 Oct '23

Hi All, We have recently encountered a challenging issue in our environment that I wanted to bring to your attention. A few months ago, we performed updates to our Docker and OpenSSL installations, upgrading OpenSSL to version 1.1.1u. Since then, we have been experiencing intermittent traffic blocking (TLS and UDP both for around 10 minutes and getting recovered) on some of our instances. Prior to the OpenSSL upgrade, we had not observed any such issues. However, the situation has become more complex due to a substantial increase in traffic, with each instance now handling 4K-5K concurrent calls and a daily total of approximately 200,000 calls per instance. Our initial suspicion was related to a Docker iptable issue. To investigate further, we conducted a GDB trace using "kamctl pstrap" and discovered specific log entries indicating potential issues within the "pthread_rwlock_wrlock" function. This has prompted us to explore whether the problems are linked to the OpenSSL upgrade. We are considering upgrading Kamailio to the latest version, although this process typically requires a significant amount of time for proper planning and testing. Given the high volume of calls we are currently handling, we are also exploring the possibility of a temporary solution involving a rollback of the OpenSSL version to see if it resolves the issue. Curiously, during a recent load test using SIPP, which simulated 10,000 concurrent calls with a rate of 100 calls per second, we did not encounter the same problem. any hints or insights you may have that could help us pinpoint the root cause of the problem . We do have a complete GDB trace available, but its size necessitates sharing a truncated version for now. version: kamailio 5.4.3 (x86_64/linux) 0178a5 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 0178a5 compiled on 12:36:56 Oct 22 2023 with gcc 4.8.5 Operating System Host : LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: Rocky Description: Rocky Linux release 8.8 (Green Obsidian) Release: 8.8 Codename: GreenObsidian container os (Centos7) Kamctl pstraip output (truncated to last few lines) ---start 154 ----------------------------------------------------- kamailio: No such file or directory. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007fbf6f38a0c3 in __epoll_wait_nocancel () from /lib64/libc.so.6 $1 = 82 $2 = {pid = 154, unix_sock = 182, idx = 27, status = 1, rank = 48, desc = "tcp receiver (tcp:xxx.xxx.xxx.xx:5060 <http://223.62.234.57:5060/>) child=27", '\000' <repeats 81 times>} #0 0x00007fbf6f38a0c3 in __epoll_wait_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x0000000000684858 in io_wait_loop_epoll (h=0xb41720 <io_w>, t=2, repeat=0) at core/io_wait.h:1039 n = 84 r = 0 fm = 0x2 revents = 8201656 __FUNCTION__ = "io_wait_loop_epoll" #2 0x0000000000699e7a in tcp_receive_loop (unix_sock=186) at core/tcp_read.c:1997 __FUNCTION__ = "tcp_receive_loop" #3 0x00000000005279e7 in tcp_init_children () at core/tcp_main.c:5134 r = 27 i = -1 reader_fd_1 = 186 pid = 0 si_desc = "tcp receiver (tcp:xxx.xx.xxx.xxx:5060 <http://223.62.234.57:5060/> )\000\000\000\355\217c\000\000\000\000\000\240\213{\001\001\000\000\000\017\000\000\000\000\000\000\000\250\a2o\276\177\000\000\200\b\231o\000\000\000\000\260\006\346\261\376\177\000\000\230\210w\000\000\000\000\000=\000\000\000>\000\000\000\270\016z\350\274\177\000\000\060;8o\002\000\000\000P\224\346\352\001\000\000" si = 0x0 __FUNCTION__ = "tcp_init_children" #4 0x000000000042ae66 in main_loop () at main.c:1771 i = 16 pid = 94 si = 0x0 si_desc = "udp receiver child=15 sock=xxxx.xxx.xxx.xx:5060\000\000\000\v\235\200", '\000' <repeats 13 times>, "P\r\346\261\376\177\000\000\270%}\000\000\000\000\000T\000\000\000\000\000\000\000\060;8o\277\177\000\000\343\207\201\000\000\000\000\000\277;8o\277\177\000\000 \271A\000\000\000\000\000@\374Ko\276\177\000" nrprocs = 16 woneinit = 1 __FUNCTION__ = "main_loop" #5 0x0000000000433ad6 in main (argc=9, argv=0x7ffeb1e60e38) at main.c:2856 cfg_stream = 0x16c9010 c = -1 r = 0 tmp = 0x7ffeb1e61f0c "" tmp_len = 2496 port = 2496 proto = 1472 ahost = 0x0 aport = 0 options = 0x7d5238 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2208520674 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7fbf6f40f3a0 <intel_02_known> p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 76, st_ino = 120506675, st_nlink = 2, st_mode = 16877, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1698340416, tv_nsec = 618631009}, st_mtim = {tv_sec = 1698340416, tv_nsec = 618631009}, st_ctim = {tv_sec = 1698340416, tv_nsec = 618631009}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 376 times>... option_index = 0 long_options = {{name = 0x7d742f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d2a34 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7d7434 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7d743a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7d7440 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7d7449 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7d7453 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7d745d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7d7468 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7d7471 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7d747c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" [Inferior 1 (process 154) detached] ---end 154 ------------------------------------------------------- ---start 155 ----------------------------------------------------- kamailio: No such file or directory. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007fbe6e5bd39e in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 $1 = 83 $2 = {pid = 155, unix_sock = -1, idx = -1, status = 1, rank = -4, desc = "tcp main process", '\000' <repeats 111 times>} #0 0x00007fbe6e5bd39e in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fbe68b745c9 in CRYPTO_THREAD_write_lock () from /lib64/libcrypto.so.1.1 No symbol table info available. #2 0x00007fbe68afe07e in CRYPTO_free_ex_data () from /lib64/libcrypto.so.1.1 No symbol table info available. #3 0x00007fbe68ea52c2 in SSL_free () from /lib64/libssl.so.1.1 No symbol table info available. #4 0x00007fbe69149480 in tls_h_tcpconn_clean_f (c=0x7fbceeae00d0) at tls_server.c:655 extra = 0x7fbceec3b898 __FUNCTION__ = "tls_h_tcpconn_clean_f" #5 0x00000000004fbf2d in _tcpconn_free (c=0x7fbceeae00d0) at core/tcp_main.c:1529 __FUNCTION__ = "_tcpconn_free" #6 0x0000000000510912 in tcpconn_put_destroy (tcpconn=0x7fbceeae00d0) at core/tcp_main.c:3284 __FUNCTION__ = "tcpconn_put_destroy" #7 0x00000000005156a5 in handle_tcp_child (tcp_c=0x7fbe6f813440, fd_i=-1) at core/tcp_main.c:3723 tcpconn = 0x7fbceeae00d0 response = {140449434960080, -2} cmd = -2 bytes = 16 n = 1865551792 t = 887186228 crt_timeout = 410 con_lifetime = 57680 __FUNCTION__ = "handle_tcp_child" #8 0x00000000005208ed in handle_io (fm=0x7fbe6f9d7c78, ev=1, idx=-1) at core/tcp_main.c:4542 ret = 1 __FUNCTION__ = "handle_io" #9 0x00000000004e93ed in io_wait_loop_epoll (h=0xad5740 <io_h>, t=5, repeat=0) at core/io_wait.h:1070 n = 1 r = 0 fm = 0x7fbe6f9d7c78 revents = 1 __FUNCTION__ = "io_wait_loop_epoll" #10 0x0000000000523c99 in tcp_main_loop () at core/tcp_main.c:4827 si = 0x0 r = 28 __FUNCTION__ = "tcp_main_loop" #11 0x000000000042b1be in main_loop () at main.c:1779 i = 16 pid = 0 si = 0x0 si_desc = "udp receiver child=15 sock=xxx.xx.xxx.xxx:5060 <http://223.62.234.76:5060/>\000\000\000\v\235\200", '\000' <repeats 13 times>, "P\r\346\261\376\177\000\000\270%}\000\000\000\000\000T\000\000\000\000\000\000\000\060;8o\277\177\000\000\343\207\201\000\000\000\000\000\277;8o\277\177\000\000 \271A\000\000\000\000\000@\374Ko\276\177\000" nrprocs = 16 woneinit = 1 __FUNCTION__ = "main_loop" #12 0x0000000000433ad6 in main (argc=9, argv=0x7ffeb1e60e38) at main.c:2856 cfg_stream = 0x16c9010 c = -1 r = 0 tmp = 0x7ffeb1e61f0c "" tmp_len = 2496 port = 2496 proto = 1472 ahost = 0x0 aport = 0 options = 0x7d5238 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2208520674 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7fbf6f40f3a0 <intel_02_known> p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 76, st_ino = 120506675, st_nlink = 2, st_mode = 16877, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1698340416, tv_nsec = 618631009}, st_mtim = {tv_sec = 1698340416, tv_nsec = 618631009}, st_ctim = {tv_sec = 1698340416, tv_nsec = 618631009}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 376 times>... option_index = 0 long_options = {{name = 0x7d742f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d2a34 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7d7434 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7d743a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7d7440 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7d7449 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7d7453 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7d745d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7d7468 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7d7471 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7d747c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" [Inferior 1 (process 155) detached] ---end 155 ------------------------------------------------------- Thanks & Regards Pintu Lohar

1 0

Inquiry Regarding Kamailio Behavior and Drop Requests
by satyaprakash ch 26 Oct '23

26 Oct '23

Hi, We possess a code within Kamailio, which specifically handles malformed SIP responses. Below is the code snippet: *reply_routeCopy code if(!sanity_check("17604", "6")) { xlog("Malformed SIP response from $si:$sp\n"); drop;* Additionally, we have Kamailio drop request statistics as follows: Command: *kamctl stats | grep core:drop_requests* Output of the command: *core:drop_requests = 5* My queries are: 1. Is the aforementioned "reply_route" related to the drop requests we are experiencing? 2. If it is not related, what could be the cause of the drop requests?

3 8

Information about Kamailio operating as an SBC
by Alvaro Escobar 24 Oct '23

24 Oct '23

Hello, I am looking information about Kamailio operating as an SBC. I saw this video of Kamailio World 2021: Kamailio As An SBC For Network Segregation: https://www.youtube.com/watch?v=UW6l3R4OnsY&t=1381s But where I can find more detailed technical information ? Regards

3 3

How to filter out xhttp request in event_route[siptrace:msg]
by Benoit Panizzon 24 Oct '23

24 Oct '23

Hi Further trying to eliminate every possible cause of UDP drops... We use Homer as HEP server in conjunction with the siptrace module. So to prevent DQM traffic to be sent to Homer I added: event_route[siptrace:msg] { if(is_method("KDMQ")) { drop(); } } And had a closer look to the traffic sent to homer. I noticed, also xhttp:requests, as far as I see the replies, not the requests, get mirrored to homer. xhttp is not a sip method I could match I guess. The event_route[siptrace:msg] docs state only method can be filtered. So I can not filter requests on the separate http port I listen on for xhttp json-rpc requests. How can I prevent xhttp traffic to be mirrored to homer by the siptrace module? Mit freundlichen Grüssen -Benoît Panizzon- -- I m p r o W a r e A G - Leiter Commerce Kunden ______________________________________________________ Zurlindenstrasse 29 Tel +41 61 826 93 00 CH-4133 Pratteln Fax +41 61 826 93 01 Schweiz Web http://www.imp.ch ______________________________________________________

4 5

STIR/SHAKEN with Kamailio
by Olle E. Johansson 23 Oct '23

23 Oct '23

Hello Kamailians! As STIR/SHAKEN seems to cross the ocean and arrive on the European shores, I’m curious on how you’ve implemented it with Kamailio. I asked on our Matrix chat and got two responses that seems to not use the Kamailio STIR/SHAKEN support but rather a 3rd party service that they’ve integrated using restful APIs. Please help me and share how you’re implementing STIR/SHAKEN with Kamailio. I’m not shaken, nor stirred. yet. Regards, /O

7 9

How to configure ndb_redis sentinel
by mayamatakeshi 21 Oct '23

21 Oct '23

Hi, regarding ndb_redis, I have 3 sentinels like this: "192.168.88.155:26379" "192.168.88.156:26379" "192.168.88.157:26379" So was thinking that I would configure ndb_redis this way: modparam("ndb_redis", "server", "name=srvZ;sentinel_group=myredis;sentinel_master=1;sentinel= 192.168.88.155:26379") modparam("ndb_redis", "server", "name=srvZ;sentinel_group=myredis;sentinel_master=0;sentinel= 192.168.88.156:26379") modparam("ndb_redis", "server", "name=srvZ;sentinel_group=myredis;sentinel_master=0;sentinel= 192.168.88.157:26379") But the samples in the documentation show an extra sentinel key in the config lines: # sentinel (for a redis slave) modparam("ndb_redis", "server", "name=srvZ;sentinel_group=group_name;sentinel_master=0;sentinel=1.2.3.4:26379 ;sentinel=1.2.3.5:26379") # sentinel (for a redis master) modparam("ndb_redis", "server", "name=srvZ;sentinel_group=group_name;sentinel_master=1;sentinel=1.2.3.4:26379 ;sentinel=1.2.3.5:26379") So I am not sure if my configuration is correct. Can someone clarify?

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

sr-users