While I have a lot of sympathies for your disappointment, I'm not really sure you are tying it with the proper causes. Let me explain my viewpoint:

- I think the Email you have received from Andrei explains even using references why Via(INVITE)==Via(BYE) requirement violates standards and actual functionality as well. Why do you think you have not been getting a good advice? What's wrong with the CANCEL suggestion?

- the MD5 performance problem you are worried about has been addressed. I admit we have answered by classifying this as a non-problem, still this is our best-knowledge of the matter based on quite details profiling. Do you have any numbers supporting this is a real problem?

Generally we don't think that for any given hardware performance is a problem with SER. Of course it can degrade for example by use of database, or any other expensive operations, but I'm quite confident that SER's thoughput is excellent and if service logic consumes more resources hardly anything can be done. At a point of time, more throughput takes more boxes but I think that this threshold is actually very high with SER.

With certainty I know that SER has been used in the big and in the *biggest* deployments. I'm worried that this may sound a bit unfriendly towards the effort you guys developed or purchased as professional service, but I think that the presence of such deployments demonstrates scale is a non-issue in reasonably dimensioned environment. SER doesn't come up with dimensioning plans, one of the reasons being that it is non-trivial to provide generally valid assertions (traffic, hardware, confgiruation, dependencies on database, network architecture, all of these differences in actual deployments make it hard to provide general rules of thumb.)

- I cannot possibly comment on interoperability of "high-dollar SBCs and switches" to the general extent you are implying. I'm worried you can be dramattically disappointed if you tie all your expectations to dollars. I only know with certainty that in the specific cases we have encountered I can impossibly assert that "high-dollar" and " brands" means knowing how INVITE shall look like. In fact, we have been using SER to fix INVITEs from high-dollar brands to look like they are supposed to look like. Which is a double-edged sword, as frequently turning a message into something that A likes makes it hard-to-swallow for B. Unless you are in a single-vendor environment, the likelihood of necessity to address interop issues is, say, higher than noticeable.

- I agree that lack of parameter passing is a shortcoming. I agree the documentation is suboptimal. I'm very thankful to all participants who spend their valuable time and return SER's value by their contributions, but there is no "central contribution control" that would allow someone to cause the participants to address your particular wishes.

-jiri

Frank Durda IV wrote:

Hello,

You are a bit out of date on this. The problem was specific to CANCELs being ignored because the tag SER added to the CANCEL did not match the tag that SER added to the INVITE for the same call.

This flaw is in at least 2.0.0RC1 (we could not use 0.x versions due to even more bugs), and adding syn_branch=0 merely changed the failure from CANCELs being ignored 100% of the time by commercial switches to being ignored perhaps 90% of the time (but at least now the messages looked more like what the RFC says they should look like).

This bug, where in many cases for calls sent from major name commercial equipment*, the CANCELs would not have the exact same tag as SER created for the INVITE (which two brands of commercial switch correctly ignored), was fixed by changing SER so that it creates its branch tag from header lines that do not legally change between INVITE and CANCEL. Exactly what I stated in later posts in this thread. I had to come up with the fix for SER myself.

• Strangely, calls from simple systems like Asterisk,

always produced the header lines in the CANCEL that were identical to those in the INVITE, so the same tag was generated and the CANCEL from such sources worked. The fancier equipment opted to add legal information to header lines between INVITE and CANCEL and so doomed SER to compute a different branch tag. SER could have solved this by using data for the computation only up to the ";" in those headers lines where data could be added or not using header lines that have a risk of being different, OR by saving a copy of the tag computed earlier. Yeah, yeah, I know about the stateless dream. Whatever.

My code change has been running for several million calls now at all our sites and solved the problem. CANCELs are now accepted and acted-on by both Sonus and Alacatel-Lucent switches where previously both brands of switch would ignore most CANCEL messages passed through SER. The calls were coming to us via/from ACME packet, Sonus, Nortel, Cisco and other switch brands, eg equipment who should know what an INVITE and CANCEL should look like and would have been accepted if SER hadn't put a mismatched tag in there.

Despite solving this issue in-house, the annoyance threshold has been reached and we will be removing SER from the production call path. SER seems fine for smaller setups, but shove 100-500Mbit/sec or more of calls through it around the clock (times several similar sites and all running on a number of high-high-end servers that are barely ticking over) and strange things start happening.

Most of that incoming traffic is coming from high-dollar SBCs and switches, built by people who probably have the standards implemented mostly right, and with enough volume, the "infrequent and rare" problems in SER that are known but no one wants to talk about or fix decide to appear frequently, and so SER becomes a pain.

No doubt, we set ourselves up for extra pain by electing to use NAT (for additional Internet isolation and for overlapping customer private network addressing) and rtpproxy, and worse, the two-interface mode of rtpproxy, which we chose for physical network isolation as well as to expand RTP bandwidth capacity. The two-interface mode part was extremely painful, required many code fixes in both SER and rtpproxy to make that usable. as well as new functions in SER, yet bugs that were there in SER and rtpproxy all along remain. On the good side, SER did protect a particularly unhardened and vulnerable commercial switch from the stupider SIP messages some of our clients generated, garbled or forwarded "as is" to us, as well as the garbage packets one has to expect coming from the Internet.

To, some the concept of using SER and rtpproxy only as cannon-fodder to take and if needed crash in response to the junk we were sometimes sent seems like a demeaning assignment for a piece of software that really is meant to do so much more, but we didn't need a lot of that, and the things we hoped we could do with SER were gradually whittled to a few because of the huge .cfg language restriction of not being able to pass variables or the contents of variables to built-in functions. (Developers, you have got to implement a way to pass dynamic data to functions, not just constants.)

In our case, Everything coming in from the outside had only one place to go, a PSTN gateway switch sitting nearby. This meant our database was empty, and if there had been a way to disconnect mysql and eliminate two or three thousand mysql connections on each server doing little or nothing (other than whining that I needed to enable more connections), we would have done so. All SER configuration was done via the .cfg file, of which we had to run dozens of instances (one per SIP "trunk") due to limitations in SER that had to be gotten-around by brute force. See the lack of variable passing mentioned above. In a NAT environment where you have to substitute public and private addresses in various headers of each message, the lack of variable passing means you have to hard-code constants for the old and substituted IP addresses everywhere and had to have giant if-else trees to decide which pair to substitute and in which direction this message was going so as to get the substiutions right. We finally elected to run multiple separate copies SER on each server so as to not disturb other customers when adding or changing another.

For almost two years SER did a reasonable job doing the task of blocking traffic from the bad/malformed packets, and undesired sources, even though a few got through that killed the PSTN switch. However, in the end, it was the good packets from desired sources that SER then turned into bad packets before passing them on (such as concoting different tags between INVITE and CANCEL messages with the same Cseq) that contributed to the forced retirement of SER.

A probably-final suggestion for SER. Fixing/updating the documentation for SER or its successor(s) would be an immense help to those using or considering its use. The web site(s) I have tried to use are dreadful, and even a simple index that lists functions names and a one-liner of what they are good for IN THE CURRENT VERSION is tough to come by. On one I got the impression that more time was spent implementing some cutesy HTML/javascript feature than on making the content usable.

There is too much expectation that someone using SER will troll all the source files, .c's and .h's looking for any comments on how functions work or what parameters they accept that might have gotten left in there, or better still, updated to match how the code actually works today. Be sure to match up dates because you may find three description of the same function and what parameters it takes, and only or none of them is how it actually works today. I realize few people like to write documentation, but it desperately needs to be done.

Finally, mere mortals can't send e-mails to the SER developers lists! It appears to be a closed club with no doorman. I gave up trying that ages ago. That being the case, we "users" have to rely on the developers coming out once in a while to read what their users are saying out here on the "mortals" serusers list, OR maybe they sould open access to some list both parties read and write to. Having worked on another open-source project (FreeBSD, back when I had spare time long ago), these things are handled quite differently there and so communications between developers and users, particularly advanced or heavy-volume users, is pretty easy.

Good luck to all and thanks for the fish.

Frank Durda IV - send mail to this address and remove the "LOSE" and adjust the month/year password accordingly: <uhclemLOSE.dec09%nemesis.lonestar.org> http://nemesis.lonestar.org "If you have a point, I suggest you sharpen it." - Frank Durda IV 2003 Copyright 2009, ask before reprinting.

---

Serusers mailing list Serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers

Frank Durda IV wrote:

Jiri Kuthan wrote:

...

While I have a lot of sympathies for your disappointment, I'm not really sure you are tying it with the proper causes. Let me explain my viewpoint:

• I think the Email you have received from Andrei explains even using

references why Via(INVITE)==Via(BYE) requirement violates standards and actual functionality as well. Why do you think you have not been getting a good advice? What's wrong with the CANCEL suggestion?

The problem here is that the discussion of BYE being involved went away some time ago. The branch tag in CANCELs not matching the branch tag for the INVITEs for the same call was the problem and what needed a fix. (I incorrectly assumed initially that BYE would be wrong as well, but as it is not a two method message sequence, BYE can't get into trouble with the branch tag computation going awry. Just INVITE & CANCEL for a call must always have the same branch tag.

That's the problem with a discussion that bounces back and forth for weeks or months before you finally just climb under the hood, add hundreds of printfs/xlogs and figure out what it is doing wrong yourself and then have to figure out a way to make it right or at least closer to what the RFC and real-life equipment claim is right. Once that was done, the muddle of earlier theories went away.

Anyway, I have fixed the problem myself from all appearances, and the INVITE/CANCEL mismatch issue is closed here.

why was syn_branch set to 0?

...

• the MD5 performance problem you are worried about has been addressed.

I admit we have answered by classifying this as a non-problem, still this is our best-knowledge of the matter based on quite details profiling. Do you have any numbers supporting this is a real problem?

The root problem was that the branch value that was computed for the INVITE wasn't the answer that was computed for the CANCEL. How slowly or how quickly that was done was not the main problem, although I do see enormous waste in using md5 for generating a hash that could have been devised from a hundred simpler algorithms. It's just a branch tag! It doesn't have to defeat NSA crypotgraphic analysis. time_t in hex down to the usec would easily exceed the uniqueness requirement and be a lot smaller and is usually right there in an integer, ready for saving a copy of and/or using.

...

Generally we don't think that for any given hardware performance is a problem with SER. Of course it can degrade for example by use of database, or any other expensive operations, but I'm quite confident that SER's thoughput is excellent and if service logic consumes more resources hardly anything can be done. At a point of time, more throughput takes more boxes but I think that this threshold is actually very high with SER.

For this particular task, MD5 is just a needless waste of computational power. Inefficiencies in this and other tasks took CPU away that could have been used for other things, never mind what. I could have run more rtpproxy sessions on the computer if it wasn't so busy doing MD5 and other unnecessary math.

While I don't disagree with that, let me reiterate that the performance penalty doesn't show in profiling and seems therefore a very little compelling problem. In other words, I don't think it would help you to boost throughput of your system at all.

to the topic of compliance with standards the following quoation from RFC3261 is probably of interest to you:

" The algorithm used to compute the hash is implementation-dependent, but MD5 (RFC 1321 [35]), expressed in hexadecimal, is a reasonable choice. "

(Understand that for years of my career I had to count T-states on individual instructions while writing assembly language drivers so that things could happen in the alloted time, so unnecessary math == bloat and I point out such poor practice when I see it.)

I don't dispute that's unnecessary despite the standard recommendation. The point is that as long as it has no impact on overall performance, this is unlikely to change for you, especially when the typical SER sentiment is that there is too much performance optimization.

And that is one of the things that baffle me about SER as a whole. SER goes out of its way in some places to do things in a way that someone thought would make the code run really really fast, like using inline code macros, or building the 32-bit integer representations of strings you were expecting to find and comparing those, all clearly done in the name of speed. The latter probably doesn't help much on modern compilers with aggressive optimizers, but such coding practices makes the memory footprint bigger and risks more paging. Meanwhile, the second technique created a hardcoding that broke the ability for SER to handle SIP-T/SIP-I,

Can you share more on that with me? I mean we are passing massive amounts of SIP-? and haven't yet run into such problems....

something SER could have passed transparently otherwise, ao that was something of a foolish thing to do for the perceived speed gains. I even suspected the lack of a way to pass variables to functions was because of an obession with speed, not because someone didn't know how to add four or five lines of rules to the lex file.

So all those things were obviously done in the name of speed, but then what does SER do on every message to generate a measly branch tag? Why, it uses an intentionally slow and complex computational algorithm (MD5) when such intensive number crunching isn't needed. Something of a dual-personality there when it comes to wanting to be fast or not caring about speed.

...

With certainty I know that SER has been used in the big and in the *biggest* deployments. I'm worried that this may sound a bit unfriendly towards the effort you guys developed or purchased as professional service, but I think that the presence of such deployments demonstrates scale is a non-issue in reasonably dimensioned environment. SER doesn't come up with dimensioning plans, one of the reasons being that it is non-trivial to provide generally valid assertions (traffic, hardware, confgiruation, dependencies on database, network architecture, all of these differences in actual deployments make it hard to provide general rules of thumb.)

And I know people at other telecom companies or companies that have a telecom presence or product, ones with really well known names and a zillion dollars. Some of these use SER, and you know what they tell me? They say, yeah we had to hire programmers to fix the problems in SER and write the missing bits, but it was the closest starting point to what we wanted that we could find.

I think I hear similar stories. Is there something you are finding surprising or unexpected in it? I also occur to think if any of these companies was to share one of the zillions or a fraction of it, it would find noble folks on the mailing list to reprogram SER for them to control power grids :-)

So, yes companies far bigger than mine are using it, but they are having to hire people to panel-beat it into an usable shape and document it. These outfits are on this list or can see its contents and are seeing my words (I know because they have commented on my messages I have posted on this list before), but some of these companies have a rule to not post anything back to lists like this because then their competition might think they were doing something in this or that area or their enemies might know where weaknesses and vulnerabilities exist that could be exploited. It seems that this is one of those things that happen when your company gets big enough or has people at the top that are paranoid enough.

Yes. It is pitiful but unfortunately I don't know what to change to that this improves.

By the way, with maybe one exception, I plan to post all the improvements/fixes I have made to SER (most of which are in and around NAThelper and rtpproxy), and maybe they will rolled into the main tree or maybe not, but at least they will be available to others and might help them avoid some headaches we have had to endure.

That's a man's word :-) Thank you very much indeed!

...

• I cannot possibly comment on interoperability of "high-dollar SBCs and

switches" to the general extent you are implying. I'm worried you can be dramattically disappointed if you tie all your expectations to dollars. I only know with certainty that in the specific cases we have encountered I can impossibly assert that "high-dollar" and " brands" means knowing how INVITE shall look like. In fact, we have been using SER to fix INVITEs from high-dollar brands to look like they are supposed to look like. Which is a double-edged sword, as frequently turning a message into something that A likes makes it hard-to-swallow for B. Unless you are in a single-vendor environment, the likelihood of necessity to address interop issues is, say, higher than noticeable.

My point here is when the SER maintainers or active respondents get feedback on these lists that these well-respected devices do not behave well with SER on specific points and that these devices rebel against coding shortcuts (like INVITE!=CANCEL or syn_branch=1), the response here should to be to address the problem and devise a fix, not to tell me or some other messenger how something else, ANYTHING else should change but not SER.

There is couple of things: First of all it is impossible to give you an SLA for solving your problems via the mailing list. folks on this mailing list are volunteers and have their obkligations too. I think there is a general sense of willingness to help, but as with most other volunteering activities there is simply no service level agreement.

The other thing is that I don't think that the problem has been understood yet (despite a lot of info you have supplied). Yet other thing is that many think (myself among those) that there is a certain bar which is not wise to be crossed in aligning to non-compliant implementations. There are actually some features trying to address broken implementations, but we have experienced when we went to far by accomodating some broken implementations that the resulting behaviour broke yet other ones.

I think it unlikely that I or anybody else on this list could get the maker of a SBC that costs $250,000 per box to change their device to overlook a point in the RFC that uses the word MUST three times. When you get caught not being compatible, undertake the work to be compatible.

I don't think there is a compliance problem here. I thikn you confirmed that the BYE problem is not a problem, and I don't think we have material showing what's wrong with CANCEL and syn_branch=1. For syn_branch=0 you have shown errors in upstream devices which SER amplifies but then I would choose not to make use of this config option.

...

• I agree that lack of parameter passing is a shortcoming. I agree the

documentation is suboptimal. I'm very thankful to all participants who spend their valuable time and return SER's value by their contributions, but there is no "central contribution control" that would allow someone to cause the participants to address your particular wishes.

I agree and to those participants who provided constructive suggestions over the past two years, I thanked them publicly and privately, and do again now in case they missed it. To the developers et all, well to be honest I haven't seen much of them. I mean, I count 70 times that Jiri has posted here in this group from Feb 2008 thru Oct 2009. (I posted a higher number over the same period.)

Well, I feel honored someone has considered it relevant to count number of my emails :-) Anyhow for those who are interested in that, I hope to keep contributing my modest ways but probably not by increasing my mailing-list rate. That's almost a full time job and I already have one.

Anyway, I know that at some point each developer put a lot of effort into writing this piece or that part of SER at some point in the past. I also realize that people get other jobs, get families, run out of the spare time needed to stuff like this, and so software and documentation fall into the marginally maintained category. Maybe that is what I'm seeing here, but I don't know.

At least for me with a job and family you are quite right.

There are also cases where the continued development is in a pay-for version and the free version languishes, with the carrot that if we pay for it, that version will be better. I expected the latter was the case for SER. Great, except that offering to pay for help and fixes didn't work either.

It is fair questions to ask ourselves.

There are companies that offer services (I'm not very experienced with these thought) and products. A company I occur to be intimately familiar with delivers a SER-based product in a rack on a truck with TL9000 certificate. I'm not aware of something inbetween, resembling a shareware in that it is lowly priced and highly productized. (Probably easier to be found for consumer software with large quantities and absolutely no customization.)

Believe me, two years ago my company tried five different ways to get someone at the listed "pay support" entities listed on the SER web site to pay attention to us, tell us how much and we were prepared to put them on a plane and have them configure our lab setup and make it work cleanly and efficiently. However, we couldn't even get a reply to the voice-mails and e-mails left. So even the pay-for support didn't seem too promising, and after three weeks of being ignored with cash in hand and deadlines looming, we just resigned ourselves to the fact that if it didn't work right or didn't do something we wanted, we would have to fix it or write it ourselves, and here we are.

just to add my 2 cents -- frequently I have been given some offers in the style "come and fix our bugs and pieces with which we are stalling with and we get you an hourly wage". Please don't think I'm trying to imply that's what you did -- I'm merely explaining why I got sort of mistrusting against "come and help us" offers, and probably many others did as well. I remember companies to whom we offered to fix a compelling problem before we settle on terms, and haven't heard of compensation until their system began crashing again. I remember several integrators who wanted to be paid for poor equivalent of what we had on the store, and were asking us to standby and fix their errors. That didn't appear very appealing to us.

For many like myself, work satisfaction is more important than compensation, and improving the baseline more attractive than customization. Frequently too some come and ask for things we have created in our commercial offerings built on top of SER. These things are available on the market at market price which DIYer companies wish to compete with, not realizing the amount of work (algorithm, testing, interop, maturing with other customers, field feedback). Frequently they end up with unexpected DIYer cost.

It all goes back to the ambitions of SER and a business model of it. SER is a really standalone server and not a whole system. Except ai1, it is not coming as a package with web and database, and it is not coming with a scalability and redundancy concept. Also I'm aware of proprietary modules that are not publicly available and can go beyond the GPLed codebase (particularly with routing and manageability). I suspect that with SER's performance, it is particularly an operator vehicle and the "few big customers" model makes the economy harder for an operator-grade package to be affordably available. Not that we haven't build such, but it has not been typically on par with pricing expectations of many whose expectations were more moderated by notion of open-source than the solution quality.

Anyhow if I had an ambition to operate a reasonably large critical setup, I would probably try to identify companies that have put such together as a package, deployed it with other customers, and align pricing expectations to that.

Well -- just a long excuse for myself should your message be on my voicemail :-) ... which I don't actually think so hopefuly.

-jiri