Re: [Serusers] Running multiple instances of rtpproxy? How to set up in SER? (detailed answer here)

A follow-up on using multiple instances of rtpproxy. I finally got this to work, although I had to look at a lot of source code and contradictory (and terse) bits of documentation and finally experiment a while. So here is what I would consider "real" documentation on how to use these functions and features of both SER and rtpproxy in one place. Although not a requirement to make this work, I'll mention my hardware platform configuration is a Dell 2950 III 48V DC-powered unit with a single four-core processor. The second CPU socket is empty. FreeBSD reports: CPU: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (2665.33-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE, MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xce3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR, PDCM,DCA,<b19>> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> Cores per package: 4 I run rtpproxy in the two-interface mode, which from previous messages here I have found that I must be the only person who does, but I do have several good reasons for doing so. I also recently reported a coding bug in rtpproxy that at least prevents the two-interface mode from working properly. The fix I came up for rtpproxy is: (diff -u this time, since apparently some OSes "patch" program can't cope with diff -c) --- main.c.STOCK 2008-02-20 18:51:44.000000000 +0000 +++ main.c 2009-03-25 20:31:34.000000000 +0000 @@ -930,7 +940,7 @@ * cannot be trusted and address is different from one * that we recorded update it. */ - if (spa->untrusted_addr == 0 && !(spa->addr[pidx] != NULL && + if (spa->untrusted_addr[pidx] == 0 && !(spa->addr[pidx] != NULL && SA_LEN(ia[0]) == SA_LEN(spa->addr[pidx]) && memcmp(ia[0], spa->addr[pidx], SA_LEN(ia[0])) == 0)) { rtpp_log_write(RTPP_LOG_INFO, spa->log, "pre-filling %s's address " @@ -940,7 +950,7 @@ spa->addr[pidx] = ia[0]; ia[0] = NULL; } - if (spa->rtcp->untrusted_addr == 0 && !(spa->rtcp->addr[pidx] != NULL && + if (spa->rtcp->untrusted_addr[pidx] == 0 && !(spa->rtcp->addr[pidx] != NULL && SA_LEN(ia[1]) == SA_LEN(spa->rtcp->addr[pidx]) && memcmp(ia[1], spa->rtcp->addr[pidx], SA_LEN(ia[1])) == 0)) { if (spa->rtcp->addr[pidx] != NULL) My setup is a bit more complicated than probably typical, so if you don't use two interfaces, simply use the one-interface invocation of rtpproxy ("-l oneipaddr") and build up your additional IP addresses as needed. You will see more on this in a moment. Each rtpproxy process needs its own IP address(es) to bind to for RTP traffic, so you will need to alias multiple IP addresses on the same ethernet interface or use multiple interfaces. In my case, I put multiple IP address for each side of the rtpproxy instances on the same physical interface using "ifconfig interface-name alias" commands. I chose to use multiple IP addresses on the same physical interface because I find that on this system I can't move more than about 40Mbit/sec per interface on a single rtpproxy instance before the processor core is at 85% utilization, so four should max-out around 160Mbit/sec, still well below the capacity of a single gigabit ethernet interface. Depending on your CPU speeds and maker as well as the system bus and ethernet card type you are using, your numbers will vary. That may mean buying plug-in cards rather than using the interfaces that happen to be on the motherboard (which frequently are slower, interrupt intensive and generally inefficient compared to others you can install). The commands to put multiple IP addresses on the same interface would be like this on BSD and similar systems: (IP addresses are altered) ifconfig em4 10.5.6.7 netmask 255.255.255.192 ifconfig em4 alias 10.5.6.8 netmask 255.255.255.255 ifconfig em4 alias 10.5.6.9 netmask 255.255.255.255 ifconfig em4 alias 10.5.6.10 netmask 255.255.255.255 and so on. Aliases in the same netblock as the non-aliased IP address normally have a 255.255.255.255 netmask, at least in the BSD world. FreeBSD lets you do this in /etc/rc.conf like this: ifconfig_em4="inet 10.5.6.7 netmask 255.255.128.0" ifconfig_em4_alias0="inet 10.5.6.8 netmask 255.255.255.255" ifconfig_em4_alias1="inet 10.5.6.9 netmask 255.255.255.255" ifconfig_em4_alias2="inet 10.5.6.10 netmask 255.255.255.255" (If you are running Linux, the way you do aliased IP addresses is probably different.) On FreeBSD, the result looks like: (our inside network) ifconfig -a ... em4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING> inet 10.5.6.7 netmask 0xffffffc0 broadcast 10.5.6.63 inet 10.5.6.8 netmask 0xffffffff broadcast 10.5.6.8 inet 10.5.6.9 netmask 0xffffffff broadcast 10.5.6.9 inet 10.5.6.10 netmask 0xffffffff broadcast 10.5.6.10 ether 00:15:17:5b:xx:xx media: Ethernet autoselect (1000baseTX <full-duplex>) status: active (public Internet side, built with similar commands) em5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING> inet 66.1.2.3 netmask 0xffffffc0 broadcast 66.1.2.63 inet 66.1.2.4 netmask 0xffffffff broadcast 66.1.2.4 inet 66.1.2.5 netmask 0xffffffff broadcast 66.1.2.5 inet 66.1.2.6 netmask 0xffffffff broadcast 66.1.2.6 ether 00:15:17:5b:xx:xx media: Ethernet autoselect (1000baseTX <full-duplex>) status: active ... Now that you have some IP addresses defined, start several instances of rtpproxy each listening for instructions on at least a different port than the other instances, as in: (As found in /etc/rc.local - non BSD systems may need this elsewhere, but this needs to be ahead of where ser is started and after the interfaces get their IP addresses.) /usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8001 -l 66.1.2.3/10.5.6.7 /usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8002 -l 66.1.2.4/10.5.6.8 /usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8003 -l 66.1.2.5/10.5.6.9 /usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8004 -l 66.1.2.6/10.5.6.10 CAUTION: There are probable security issues with using public IP addresses for the rtpproxy control address, so avoid doing that. (rtpproxy will take instructions from anybody who can get to the port and knows how to talk to it, or might just send it junk and crash that process. If you must use a public IP, block external access to the port range that is used to control rtpproxy.) Once running, reference those rtpproxy processes via a line like this in ser.cfg: modparam("nathelper", "rtpproxy_sock", "udp:127.0.0.1:8001=1 udp:127.0.0.1:8002=1 udp:127.0.0.1:8003=1 udp:127.0.0.1:8004=1") That's all on one line. You will need to comment-out the existing nathelper line that uses the AF_UNIX socket to communicate with just the one copy of rtpproxy. If you still have the old AF_UNIX rtpproxy running, kill it. The "=1" part of the modparam is the weighting, and can be handy if some of the rtpproxy processes are running on a different computer that is faster or slower than the local one or others. If all on the same computer, they should all be equal. The documentation on the weighting control is unusually weak (even by SER documentation standards), so I can't tell you exactly how the weighting control works, what ranges of values are valid (or if they can be floating point), or the total number of weights should add up to a total value, or if it is simply a round-robin distribution, where a "1" says the next call goes to this rtpproxy before advancing to the next rtpproxy instance, while a "2" says the next two calls go here before advancing, and so on. I'm using all "1" values for the local system and it seems to be balancing more or less evenly, so for other choices you will need to experiment. Once running for a while, you should see something like this in "top" PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 2207 xxxxxxxx 1 8 0 4180K 1056K nanslp 1 17.5H 27.05% rtpproxy 2208 xxxxxxxx 1 8 0 4192K 1068K nanslp 2 17.5H 26.66% rtpproxy 2206 xxxxxxxx 1 102 0 4204K 1080K RUN 3 17.4H 25.29% rtpproxy 2205 xxxxxxxx 1 8 0 4228K 1104K nanslp 0 17.3H 19.92% rtpproxy You should now also see in tcpdump/ngrep output that the SDP payloads have different IP addresses in them for the rtpproxy that is handling the given call. Based on limited experience so far, starting more rtpproxy instances on a machine than there are total processors/cores available doesn't seem to do anything bad, but I would think it would be less efficient than having the numbers of rtpproxy equal to or less than available CPUs. I did not mess with any of the mechanisms SER offers that can force a given call to a given instance. Such a thing might be useful if you want to guarantee a higher level of stability in packet forwarding or if some of the instances were functionally different. In my case, all instances of rtpproxy have the same capabilities. Good luck! Frank Durda IV http://nemesis.lonestar.org Copyright 2009, ask before reprinting.