29 Mar
2011
29 Mar
'11
12:31 p.m.
Hi All,
I'm trying to setup an active/passive proxy instance with kamailio
3.1.2. I'm using ucarp (www.ucarp.org) to provide the floating ip
addresses. I have installed and configured ucarp and it appears to be
working as expected.
I have configured kamailio on both systems to be online all the time
(using the net.ipv4.ip_nonlocal_bind kernel parameter on the "standby"
server). I have configured kamailio with the listen parameter to be
listening on the floating ip addresses only.
The problem now is that kamailio uses the inbound interface as the
outbound socket for messages (I think this is the default, expected,
behaviour).
If I enable the "mhomed" parameter then I get the following error:
ERROR: <core> [forward.c:218]: ERROR: get_out_socket: no socket found
ERROR: <core> [forward.c:572]: forward_req: ERROR: cannot forward to af
2, proto 1 no corresponding listening socket ERROR: sl [sl_funcs.c:282]:
ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (7/SL)
I had a look around on the docuwiki and I found $fs. Would I need to
force the outbound socket for each message in this case, or do I have a
simple comfiguration issue?
Any thoughts/comments would be greatly appreciated.
Thanks
29 Mar
29 Mar
12:39 p.m.
I have kamailio working with ucarp in an active/standby mode. The way I did
it was to bind kamailio to a fixed ip address and use ip tables to forward
between the ucarp address and the fixed address. Since this is effectively a
NAT traversal as far as Kamailio is concerned, you have to setup appropriate
record routes and the advertised address within kamailio.
Paul
-----Original Message-----
From: Asgaroth
Sent: Tuesday, March 29, 2011 10:31 AM
To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
UsersMailing List
Subject: [SR-Users] Outbound socket selection (HA implementation)
Hi All,
I'm trying to setup an active/passive proxy instance with kamailio
3.1.2. I'm using ucarp (www.ucarp.org) to provide the floating ip
addresses. I have installed and configured ucarp and it appears to be
working as expected.
I have configured kamailio on both systems to be online all the time
(using the net.ipv4.ip_nonlocal_bind kernel parameter on the "standby"
server). I have configured kamailio with the listen parameter to be
listening on the floating ip addresses only.
The problem now is that kamailio uses the inbound interface as the
outbound socket for messages (I think this is the default, expected,
behaviour).
If I enable the "mhomed" parameter then I get the following error:
ERROR: <core> [forward.c:218]: ERROR: get_out_socket: no socket found
ERROR: <core> [forward.c:572]: forward_req: ERROR: cannot forward to af
2, proto 1 no corresponding listening socket ERROR: sl [sl_funcs.c:282]:
ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (7/SL)
I had a look around on the docuwiki and I found $fs. Would I need to
force the outbound socket for each message in this case, or do I have a
simple comfiguration issue?
Any thoughts/comments would be greatly appreciated.
Thanks
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
12:43 p.m.
Forgot to mention - the reason it is done this way is that Kamailio cannot
bind to the virtual IP that ucarp is managing.
It has to bind to an IP address that is available the entire time.
-----Original Message-----
From: Paul Pankhurst
Sent: Tuesday, March 29, 2011 10:39 AM
To: SIP Router - Kamailio (OpenSER) and SIP Express Router
(SER) -UsersMailing List
Subject: Re: [SR-Users] Outbound socket selection (HA implementation)
I have kamailio working with ucarp in an active/standby mode. The way I did
it was to bind kamailio to a fixed ip address and use ip tables to forward
between the ucarp address and the fixed address. Since this is effectively a
NAT traversal as far as Kamailio is concerned, you have to setup appropriate
record routes and the advertised address within kamailio.
Paul
-----Original Message-----
From: Asgaroth
Sent: Tuesday, March 29, 2011 10:31 AM
To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
UsersMailing List
Subject: [SR-Users] Outbound socket selection (HA implementation)
Hi All,
I'm trying to setup an active/passive proxy instance with kamailio
3.1.2. I'm using ucarp (www.ucarp.org) to provide the floating ip
addresses. I have installed and configured ucarp and it appears to be
working as expected.
I have configured kamailio on both systems to be online all the time
(using the net.ipv4.ip_nonlocal_bind kernel parameter on the "standby"
server). I have configured kamailio with the listen parameter to be
listening on the floating ip addresses only.
The problem now is that kamailio uses the inbound interface as the
outbound socket for messages (I think this is the default, expected,
behaviour).
If I enable the "mhomed" parameter then I get the following error:
ERROR: <core> [forward.c:218]: ERROR: get_out_socket: no socket found
ERROR: <core> [forward.c:572]: forward_req: ERROR: cannot forward to af
2, proto 1 no corresponding listening socket ERROR: sl [sl_funcs.c:282]:
ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (7/SL)
I had a look around on the docuwiki and I found $fs. Would I need to
force the outbound socket for each message in this case, or do I have a
simple comfiguration issue?
Any thoughts/comments would be greatly appreciated.
Thanks
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
12:46 p.m.
I'm using a different setup.
Both servers have their own fixed ip-address and both have a virtual
ip-address.
pacemaker makes sure that when one server is down, its virtual ip is
moved to the other node.
So this is more a active-active setup.
Enabled ip_nonlocal_bind, set listening to both virtual ip-addresses and
forcing the socket based on the incoming ip-address.
I don't know if this is a good solution, didn't find much usefull info
on the web,
but it works.
mhomed=0
listen=1.2.3.1:5060 # virtual ip-address 1
listen=1.2.3.2:5060 # virtual ip-address 2
route{
if ($Ri == "1.2.3.1") {
force_send_socket(1.2.3.1:5060);
} else if ($Ri == "1.2.3.2") {
force_send_socket(1.2.3.2:5060);
}
With kind regards,
Robert Verspuy
--
*Exa-Omicron*
Patroonsweg 10
3892 DB Zeewolde
Tel.: 088-OMICRON (66 427 66)
http://www.exa-omicron.nl
2:42 p.m.
Thank you all for your replies.
Our requirement, currently is fairly simple, and, I'm thinking that
pacemaker/heartbeat/corosync is a little more overhead than required.
Ucarp seems to fit the profile fairly well for what we are trying to
achieve.
I'd like to have kamailio decide on the outbound interface if at all
possible, I suspect I would need to delve into all the routing branches
if I were to force the outbound socket manually.
If I listen on all interfaces and enable mhomed then kamailio transmits
on the correct outbound interface, but then uses, what I suppose, is
called the primary address associated with the interface. If I were to
stay with this configuration, then, would I need to enable the
advertised address/port parameters? What about devices behind NAT
firewalls, if my advertised address was 1.2.3.4 and the actual interface
address was 1.2.3.3 then the NAT'ed UA would see traffic arriving to
them as originating from 1.2.3.3 which is not what I am trying to
achieve (it needs to arrive in as 1.2.3.4, which is the same address
that they registered against. In either case, I would need to set the
advertised address based on the destination ip address (i assume
obtained by $dd or $rd) so that I can set the appropriate advertised
address/port or forced outbound socket.
I saw a previous post saying that for mhomed to work correctly kamailio
needs to be listening on the interface that is used for the routing of
the destination address, this makes sense to me. However, with the carp
configuration I have, the "virtual" address is indeed sitting on the
interface that has the destination network configured. I guess Kamailio
is unable to decide on which ip address to use on the outbound
interface, would it not be possible to have the address used to send, be
that of the "listen_address" parameter even if it is an associated
"virtual" address on the same physical NIC.
Or maybe this may be possible with a couple "ip rule" commands to
associate a destination net with a virtual ip address?
With corosync/pacemaker/heartbeat, are you able to have kamailio operate
on the floating address without haveing to perform the routing decisions
for kamailio?
Any comments/suggestions would be greatly appreciated.
Thanks
3:03 p.m.
On 03/29/2011 02:42 PM, Asgaroth wrote:
Thank you all for your replies.
Hello again,
I have sent you another email in the thread but no reply. Can you
confirm that you have a listening UDP socket on the interface that is
selected by the kernel for sending the datagram in a mhomed environment.
If you are not listening on that interface Kamailio will complain with
the error message you received in mhomed=1 config
Marius
Our requirement, currently is fairly simple, and,
I'm thinking that
pacemaker/heartbeat/corosync is a little more overhead than required.
Ucarp seems to fit the profile fairly well for what we are trying to
achieve.
I'd like to have kamailio decide on the outbound interface if at all
possible, I suspect I would need to delve into all the routing branches
if I were to force the outbound socket manually.
If I listen on all interfaces and enable mhomed then kamailio transmits
on the correct outbound interface, but then uses, what I suppose, is
called the primary address associated with the interface. If I were to
stay with this configuration, then, would I need to enable the
advertised address/port parameters? What about devices behind NAT
firewalls, if my advertised address was 1.2.3.4 and the actual interface
address was 1.2.3.3 then the NAT'ed UA would see traffic arriving to
them as originating from 1.2.3.3 which is not what I am trying to
achieve (it needs to arrive in as 1.2.3.4, which is the same address
that they registered against. In either case, I would need to set the
advertised address based on the destination ip address (i assume
obtained by $dd or $rd) so that I can set the appropriate advertised
address/port or forced outbound socket.
I saw a previous post saying that for mhomed to work correctly kamailio
needs to be listening on the interface that is used for the routing of
the destination address, this makes sense to me. However, with the carp
configuration I have, the "virtual" address is indeed sitting on the
interface that has the destination network configured. I guess Kamailio
is unable to decide on which ip address to use on the outbound
interface, would it not be possible to have the address used to send, be
that of the "listen_address" parameter even if it is an associated
"virtual" address on the same physical NIC.
Or maybe this may be possible with a couple "ip rule" commands to
associate a destination net with a virtual ip address?
With corosync/pacemaker/heartbeat, are you able to have kamailio operate
on the floating address without haveing to perform the routing decisions
for kamailio?
Any comments/suggestions would be greatly appreciated.
Thanks
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
3:32 p.m.
Hi Marius,
I have sent you another email in the thread but no
reply. Can you
confirm that you have a listening UDP socket on the interface that is
selected by the kernel for sending the datagram in a mhomed
environment. If you are not listening on that interface Kamailio will
complain with the error message you received in mhomed=1 config
I have my kamailio configuration as follows:
listen=192.168.1.100:5060
listen=192.168.2.100:5060
And my linux server ip configuration is as follows:
Where 192.168.1.99/192.168.2.99 is the ip address configured for the
respective interfaces, and the 192.168.1.100/192.168.2.100 are the
"virtual" interfaces created by ucarp.
[root@testprxy #] ip addr show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
link/ether 00:0c:29:9d:0d:c5 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.99/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.100/32 scope global eth0
inet6 fe80::20c:29ff:fe9d:dc5/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
link/ether 00:0c:29:9d:0d:cf brd ff:ff:ff:ff:ff:ff
inet 192.168.2.99/24 brd 192.168.2.255 scope global eth1
inet 192.168.2.100/32 scope global eth1
inet6 fe80::20c:29ff:fe9d:dcf/64 scope link
valid_lft forever preferred_lft forever
Do I need to have kamailio listen on the 192.168.1.99/192.168.2.99
addresses as well?
I think currently the kernel my select 192.168.1.99 to route out to
192.168.1.x/24, and 192.168.2.99 to route out to 192.168.2.x. See
routeing table below:
[root@testprxy #] ip route show
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.99
192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.99
default via 192.168.1.1 dev eth0
Now if I did something like:
ip rule add from 192.168.1.100/32 table 100
ip rule add to 192.168.1.100/32 table 100
ip rule add to 192.168.1.0/24 table 100
ip route add default via 192.168.1.1 table 100
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.100 table 100
ip rule add from 192.168.2.100 table 101
ip rule add to 192.168.2.100 table 101
ip rule add to 192.168.2.0/24 table 101
ip route add 192.168.2.0/24 dev eth1 src 192.168.2.100 table 101
Would kamailio be able to use those interfaces then?
Thanks
5:14 p.m.
On 03/29/2011 03:32 PM, Asgaroth wrote:
Yes , for kamailio to send datagrams on those interfaces it has to
explicitely listen on them.
Hi Marius,
Hello Asgaroth,
Comments inline
I have sent
you another email in the thread but no reply. Can you
confirm that you have a listening UDP socket on the interface that is
selected by the kernel for sending the datagram in a mhomed
environment. If you are not listening on that interface Kamailio will
complain with the error message you received in mhomed=1 config
I have my kamailio configuration as follows:
listen=192.168.1.100:5060
listen=192.168.2.100:5060
And my linux server ip configuration is as follows:
Where 192.168.1.99/192.168.2.99 is the ip address configured for the
respective interfaces, and the 192.168.1.100/192.168.2.100 are the
"virtual" interfaces created by ucarp.
[root@testprxy #] ip addr show
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
link/ether 00:0c:29:9d:0d:c5 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.99/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.100/32 scope global eth0
inet6 fe80::20c:29ff:fe9d:dc5/64 scope link
valid_lft forever preferred_lft forever
3: eth1:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
link/ether 00:0c:29:9d:0d:cf brd ff:ff:ff:ff:ff:ff
inet 192.168.2.99/24 brd 192.168.2.255 scope global eth1
inet 192.168.2.100/32 scope global eth1
inet6 fe80::20c:29ff:fe9d:dcf/64 scope link
valid_lft forever preferred_lft forever
Do I need to have kamailio listen on the 192.168.1.99/192.168.2.99
addresses as well?
I think currently the kernel my select 192.168.1.99 to route out to
192.168.1.x/24, and 192.168.2.99 to route out to 192.168.2.x. See
routeing table below:
[root@testprxy #] ip route show
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.99
192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.99
default via 192.168.1.1 dev eth0
Now if I did something like:
ip rule add from 192.168.1.100/32 table 100
ip rule add to 192.168.1.100/32 table 100
ip rule add to 192.168.1.0/24 table 100
ip route add default via 192.168.1.1 table 100
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.100 table 100
ip rule add from 192.168.2.100 table 101
ip rule add to 192.168.2.100 table 101
ip rule add to 192.168.2.0/24 table 101
ip route add 192.168.2.0/24 dev eth1 src 192.168.2.100 table 101
Would kamailio be able to use those interfaces then?
Thanks
The restriction is from a design/functionality p.o.v. Kamailio must be
able to receive a reply (response) on the interface it sends to. So if
you can ensure that the kernel will choose one interface that kamailio
is bound to, then you are ok, but if it choses one iface that kamailio
does not listen, then you have no way of sending the datagram(and that
error will appear).
I can't really say that the route table will work. One way I used for
testing is via Ping or netcat without any local parameters. The choosing
of the local address and port is left to the kernel, the same as
kamailio with mhomed, so if you can find a suitable routeing table where
only the ucarp interfaces are selected then you are ok. Why don't you
listen also on 192.168.1.99/192.168.2.99 ?
Marius.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
5:47 p.m.
Hi Marius,
I can't really say that the route table will work.
One way I used for
testing is via Ping or netcat without any local parameters. The
choosing of the local address and port is left to the kernel, the same
as kamailio with mhomed, so if you can find a suitable routeing table
where only the ucarp interfaces are selected then you are ok. Why
don't you listen also on 192.168.1.99/192.168.2.99 ?
When kamailio transmits the messages it will substitute (by default) the
outbound interface ip address for its route/via headers. If I enable
kamailio to listen on all addresses then kamailio sends the messages
with source address of .99. I need the source address and routing
headers to contain the floating address (.100), this is also for NAT
traversal on the "public" address of the proxy.
6:18 p.m.
On 03/29/2011 05:47 PM, Asgaroth wrote:
Hi Marius,
Hello,
If i understand, then you will need to force sending of outgoing
requests (not in-dialog or replies) to be sent via the floating address,
so reachable for them. You might want to look force_send_sock as well
(don't know if already this appeared on this thread)
Marius
I can't really say that the route table will
work. One way I used for
testing is via Ping or netcat without any local parameters. The
choosing of the local address and port is left to the kernel, the same
as kamailio with mhomed, so if you can find a suitable routeing table
where only the ucarp interfaces are selected then you are ok. Why
don't you listen also on 192.168.1.99/192.168.2.99 ?
When kamailio transmits the messages it will substitute (by default) the
outbound interface ip address for its route/via headers. If I enable
kamailio to listen on all addresses then kamailio sends the messages
with source address of .99. I need the source address and routing
headers to contain the floating address (.100), this is also for NAT
traversal on the "public" address of the proxy.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
12:39 p.m.
On 03/29/2011 12:31 PM, Asgaroth wrote:
Hi All,
I'm trying to setup an active/passive proxy instance with kamailio
3.1.2. I'm using ucarp (www.ucarp.org) to provide the floating ip
addresses. I have installed and configured ucarp and it appears to be
working as expected.
I have configured kamailio on both systems to be online all the time
(using the net.ipv4.ip_nonlocal_bind kernel parameter on the "standby"
server). I have configured kamailio with the listen parameter to be
listening on the floating ip addresses only.
The problem now is that kamailio uses the inbound interface as the
outbound socket for messages (I think this is the default, expected,
behaviour).
If I enable the "mhomed" parameter then I get the following error:
ERROR:<core> [forward.c:218]: ERROR: get_out_socket: no socket found
ERROR:<core> [forward.c:572]: forward_req: ERROR: cannot forward to af
2, proto 1 no corresponding listening socket ERROR: sl [sl_funcs.c:282]:
ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (7/SL)
Hello,
Some observations: mhomed only works for UDP, but I think it's enough
for your setup. Second, for it to select the outbound interface,
Kamailio must be started and listening on that interface (or the
interface the kernel selects to route traffic to the specific destination).
I would suggest to start Kamailio to listen on all available interfaces.
Marius
I had a look around on the docuwiki and I found $fs.
Would I need to
force the outbound socket for each message in this case, or do I have a
simple comfiguration issue?
Any thoughts/comments would be greatly appreciated.
Thanks
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
30 Mar
30 Mar
5:10 p.m.
All,
Thanks for the tips and suggestions, I have now managed to get this
working using mhomed=1 and setting $fs, for some reason I could not get
"force_send_socket" to work with avp's (but thats a topic for another
thread).
Thanks again for your time.
31 Mar
31 Mar
11:31 a.m.
FYI, I have the following setup without any problems (at least no
problems on the server side, just buggy clients ;-)
2 Kamailio in active-active setup. Each Kamailio listens to the
"dedicated" IP address and the "virtual" IP by using ip_nonlocal_bind.
Virtual IP is managed by Pacemaker (I'm not using an OCF resource file
for kamailio, only the init-script).
Communication between Kamailio and back-end servers also uses the
virtual IP address.
Kamailio uses the dedicated IP as default IP (first "listen" directive).
mhomed=0
REGISTERs are replicated between the Kamailios. Is is the only part
where the proxies use their dedicated IP address (so I can easily detect
replicated REGISTERs). Therefore I call
force_send_socket(IPADDRESS_MYSELF:5060); before
forward("IPADDRESS_OTHERNODE");
regards
Klaus
On 29.03.2011 11:31, Asgaroth wrote:
Hi All,
I'm trying to setup an active/passive proxy instance with kamailio
3.1.2. I'm using ucarp (www.ucarp.org) to provide the floating ip
addresses. I have installed and configured ucarp and it appears to be
working as expected.
I have configured kamailio on both systems to be online all the time
(using the net.ipv4.ip_nonlocal_bind kernel parameter on the "standby"
server). I have configured kamailio with the listen parameter to be
listening on the floating ip addresses only.
The problem now is that kamailio uses the inbound interface as the
outbound socket for messages (I think this is the default, expected,
behaviour).
If I enable the "mhomed" parameter then I get the following error:
ERROR:<core> [forward.c:218]: ERROR: get_out_socket: no socket found
ERROR:<core> [forward.c:572]: forward_req: ERROR: cannot forward to af
2, proto 1 no corresponding listening socket ERROR: sl [sl_funcs.c:282]:
ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (7/SL)
I had a look around on the docuwiki and I found $fs. Would I need to
force the outbound socket for each message in this case, or do I have a
simple comfiguration issue?
Any thoughts/comments would be greatly appreciated.
Thanks
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi All,
I'm trying to setup an active/passive proxy instance with kamailio
3.1.2. I'm using ucarp (www.ucarp.org) to provide the floating ip
addresses. I have installed and configured ucarp and it appears to be
working as expected.
I have configured kamailio on both systems to be online all the time
(using the net.ipv4.ip_nonlocal_bind kernel parameter on the "standby"
server). I have configured kamailio with the listen parameter to be
listening on the floating ip addresses only.
The problem now is that kamailio uses the inbound interface as the
outbound socket for messages (I think this is the default, expected,
behaviour).
If I enable the "mhomed" parameter then I get the following error:
ERROR:<core> [forward.c:218]: ERROR: get_out_socket: no socket found
ERROR:<core> [forward.c:572]: forward_req: ERROR: cannot forward to af
2, proto 1 no corresponding listening socket ERROR: sl [sl_funcs.c:282]:
ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (7/SL)
I had a look around on the docuwiki and I found $fs. Would I need to
force the outbound socket for each message in this case, or do I have a
simple comfiguration issue?
Any thoughts/comments would be greatly appreciated.
Thanks
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
4988
days inactive
4990
days old
12 comments
6 participants
participants (6)
-
Asgaroth -
Bruce McAlister -
Klaus Darilion -
marius zbihlei -
Paul Pankhurst -
Robert Verspuy