Transcript

Nenad Corbic: My name is Nenad Corbic, I am the chief software engineer at Sangoma . Today we are going to discuss the Sangoma drivers – the drivers and concepts and building tools that we have created so far and how we integrated with Asterisk and as well with FreeSwitch .

I have quite a bit of information and I am going to go pretty fast. But it’s in order for you to appreciate all the components that we build. I will go through a little history lesson of what the current TDM marketing picture looks like and what are the problems that one sees when working with TDM and what we have done about it.

So as I said the API Building Blocks we will see the Zaptel Architecture and Zaptel Model. I will also introduce the libsangoma which is the unified API from Sangoma. Libsigboost we will quickly introduce the Sangoma PSBN stacks that are now available.

Sangoma FreeSwitch we will dive into the OpenZap architecture and what so special about it. At the end I will mention Womera and how we work with it and build scalable gateways with Asterisks and FreeSwitch.

So TDM Architecture and limitations to bottlenecks. In current design of Asterisk and FreeSwitch and pretty much all the other architecture we have seen is that they are monolithic. Usually you have a card and you have a driver and then there is a system that is connected to that driver via a yacucal (sp) so everything has to reside in that same system. It becomes a problem when you actually try to scale this system to a large number of TDM devises. What ends up happening is that you are limited by the size o f the system.

Another big thing with the TDM is that interrupt frequency. What is so special about TDM is that data comes in at 1 mil. Second and it is very much a real time system. We are expecting Linux mostly to behave like a real time operating system while it is not. There are some real challenges in that in terms of intermplatancy (SP) we have actually found that with new systems and especially with some older PCI X systems that the latency can vary all over the place. Sometimes good and then all of a sudden you get 10 mil seconds or 20 mil seconds worth of frequency on your interrupt which can completely screw up your cues and timing. So how do we battle that?

The very big ones in any kind of variant scale is a kernel context penalty. Now you have heard Ole talk about the context penalty running lots of sib calls. The same thing is with TDM. You have a lot of channels that context penalty counts very high but systems driven performance drops off exponentially.

We will get some clustering solutions. There currently Asterisk and pretty much all the architecture right now. Their kind of hard to install and configure. From a TDM point of view they are hard to work with. Lucation (SP) is a codec there has always been limitations unless you do it in hardware, but that can get very expensive. However with the CPUs running all the cores and tomorrow we will have eight cores, sixteen cores, so codex is not so much of a problem as the echo cancellation is where it needs to be a real time system. So we will dive into these.

This might look a little complicated at first. This is the current Asterisks or TDM model or DAHDI When ever I say zaptas, think of it as DAHDI’s.
The red arrow are potential problems in this model and in this architecture. First of all Zaptel (SP) requires any hardware manufacturer to give it 1 millisecond interrupts. Zaptel was 8 bits for every channel. Because of its echo canceller arrangement. Further more, Zaptel uses software that you will see and it does a lot of voice processing and has a whole bunch of functionality inside the Kernel. What is the problem with this model? The problem with this model is if you force a card to give you one millisecond interrupt that gives you 1000 interrupts per second. The whole Zaptel runs into hardware interrupt. That completely, puts a lot of stress on the system because nothing is really meant to run for a long time inside a hardware interrupt. But that is the only way that we get real time performance out of the Linux kernel. The funny thing is that if we had to run zaptel on windows, it would not work. Because you cannot run stuff in the interrupt, but Linux allows us to do it and we are able to do it and it works very well.
The problem is the interrupt per seconds, especially when you start scaling and putting 8 port cards 4 8 port cards you start getting into very high CPU interrupt. And why do we really need 1 millisecond because if you see Asterisks it doesn’t work in 1 mil second level, it works at 20 mil second level. So in its cashing every twenty times it calls up Asterisks and says, ok here is your time it takes.

The third one, up in the slide- the potential problem is the fact that the kernel context value. We try to start a scale at asterisks system and we go over, lets say 200 channels, and the performance degrades considerably. Why? Well the biggest reason is from 200 channels we have 200 kernel devices. For 200 kernel devices you have to run a hole on each one, you have to select and read and write and read events for every single device so while it is a very nice system to program in, it is very easy, you know you open get “ ran one chan one” you can just go ahead and grab that device and open it and you have that data just for that span – that ease of use costs you the scalability, so nothing is free.

We talked about infrar (SP) Frequency and the latency. Now I just want to very quickly talk about latency. As I said, this is something we found and have known about it, we actually measured it on the solsiliscipe SP with these new machines and 3 cores and 8 cores right now are actually scare how they are performing. And the way the treat interrupts is like the priority of the interrupt is uh, it usually works fine, but every twenty minutes or half an hour you can get a very large [unintelligible] Frequency. The only way to combat this is to have a large number of [unintelligible] in the card to actually save 20 mil seconds worth of data or actually do some sort of a fancy DMA scheme that we end up doing in the driver to allow the hardware and push air into the driver even though we don’t have actual interrupts.

So what we have done while looking at Zaptel – We love Zaptel, we love Asterisks, we think it is great. It is performing very well, fine parameters. People seem to forget that it hasn’t been designed to put thousands of cards in, so you have to plan for that. You have to work inside the limits the system has been designed for. So too actually design a system to run more and more calls, our hardware is able to do higher chunk size DMA. Even though we cannot fix Zaptel and the architecture and stuff like that we can optimize the hardware and the drivers to work more efficiently. So if you install one of these drivers you can quickly say Sangoma chunk sizes or DAHDI chunk sizes data for you for 5 mil or 80 mil second, which is equal to 10 mil seconds, that will decrease the number of interrupts by a factor of 10. So all of a sudden we are going to be pushing 80 or 10 mil seconds into Zaptel and we would be generating 100 mil per second. This is huge. You probably won’t see the benefit of one card, but running 16 E1’s, you will see a huge benefit on it. If you run it embedded you will see a huge benefit, especially from a single core processors.
One other thing is that the multi cores do not help you because a kernel cannot share in interrupts between CPUs . So let’s say you have A 4s, all the A4s run on a single CPU on a single interrupt. You know once you have swung of the pass, afterwards they can run on different CPU’s, but the interrupt itself, all the bar runs on that same interrupt. That is why pushing more boards on the system actually does not help you.

Man: Question (cannot hear)

Nenad: No, it does everything inside the interrupt. The pass, yes, but the core, the media pass pushing is all done through the interrupt. Especially the echo canceling and all that stuff. (SP) Passes are not fast enough. They do not have the periodic … you cannot get 20 mil seconds, especially if its an iga system SP yes, but as soon as you load it just a bit the pass ways just fold right away.
So that is pretty much what we have done for Zaptel and how much we have gone through zaptel. We had customers coming to us saying we want to build customs with your part. We want to do higher lows/ higher scales so for that matter and for us to fix these other problems we had to interrupt.uce our own kernel module. It is called TDM API. It does very limited functionality of what Zaptel is. Zaptel has a lot of stuff inside the kernel which we kind of don’t agree with. You know the more you do inside the space you can make use of multiple cores. User space applications can distribute better across systems but as little as possible in the interrupt pass the data through get it up to the user space and then let the user space handle the data and other processors. In this case this actually solves the problem of having a lot of interrupts, but it still does not address the context penalty of having many devises. So right now with our systems we run 16E1’s without any issues with Asterisks. Mind you this is as SIFF to data input. If you are going to run a lot of devices ADR’s or HDR you may have to consider that number, but it works very well as a SIFF input. Very stable.
One way to solve the context penalty is a thing called Span Mode. Now in Span mode what we are doing is we are going to give creative devise her E1 Span so what that means is that we are going to pass all the data from one single span to out into the user space. Now this adds some complexity to the application because the application has to go in and multiplex and demultiplex and data for every channel but its well worth it because the context penalty goes away.
We had 28 E1’s. That gave us 28 devises that we had to select so we are actually pushing much more data through less number of devises.

At this point we can start really scaling our gateways. This scales up to 32 E1’s without any issues on quad functions. We have unified this two systems that we have talked about the most – this thing where you have a device per channel. We call that a channelized mode. And we have a devise for span called span mode and we have unified both into one unified DPI in libsangoma. Now this Libsangoma provides a unified experienced so that this would be the device here. This would be the channelized devise. This would be one to two hundred lets say. If you are running in span mode you are running span devises that run from one to twenty. You have timing devise that gives you timing ticks. You have control devises where you get events and there are all channels and you can fully manage the card from the user space.

The biggest development that we have done through a major development cycle is too unified in windows and Linux. You write identical applications using windows and Linux using the same data. That is our big change.

Another thing that we have just added since about two years ago this path we have our own stacks now. We have a Net Briggs BRI Stack (sp), a telesoft PLI Stack, and we have our own SS7 Stack. As well as our own R2 Stack – now if you were in South America somewhere something like this wouldn’t matter. But what we have done extracted all these stacks into one unified EPI. For us we cannot support multiple source scopes – multiple branches of code, so we have to do everything once and just support one source code.

So what we have done here, as you can see, is, we call it boost, for no particular reason. Singling EPI. We wrap the stacks into its own domain and provide a solid base interface singularly, so now a custom application, whether it be Asterisks can use this and talk to the voice channels and talk to our stacks using a very simple unified API. Now whether you are talking to the SS7 or talking to the BRI or PRI it does not matter. The application can actually work in both modes – channelized or span mode.

These stacks provide Sangoma media gateway. It is all the work that we have done over the years to speed up the drivers and the media pack and to produce a system that almost logic oriented. There is a media path, a signalling path, the core that works between them and then the libraries that maybe needs a caller ID or whatever, and we provide a unified API for whatever system you are talking too. This architecture has been our goal. We want to support one gate way. We want to support Asterisk using this, we want to support FreeSwitch or any other that is using API. Someone wants to use this with one goal we found this works very very well.

Where does FreeSwitch fit in? Three years ago when we had a need, when FreeSwitch first started, we continued to working and developing Asterisks and making it better and scaling Asterisks to larger systems. FreeSwitch on their side have been working on their SIP Stack and their Core and you will hear that they have a very impressive SIP performance – 5000 calls running on one machine. But there TDM site has not been developed as much. There is a reason for that because we were working with parallel packs.

You will see that is quite similar the picture of our upgrade. There is a reason for this. Early on we had agreed on this kind of architecture and we were going to work on our layers – on the kernel and our interface and they were going to build a open stack that wrapped our API and our Signalling.

You will see the difference between sigma media but it supports any zap held card actually. But again, it opens up again and is a user space application or library that provides a unified API through media and signalling.

You will see that there are no red marks in Zaptel. OpenZap uses Zaptel in a very dumb down mode – meaning they do not use any processing inside to kernel. This works just fine in the small systems. Zaptel and OpenZap from architecture has a core and that core talks to the media engines. The media or the engines or channels, whatever you want to call them. They can be Zaptel or Sangoma. Sangoma for example talks through a UPI architecture and Zaptel talks through the Zap ZP channel to the Zapta Channels. They provide a unified channel through the Zapfio (SP). It has been designed in such away that if you don’t like one stack you can put in another stack. It is very easy to add a new stack to this kind of system because everything again is very object oriented and in there very well defined interface. Libraries, Codex could be there any IO processing that is needed is provided to a custom voice API using the OpenZap API architecture. Our quest is to be like our dialogic friends there that had all this stuff done 50 years ago. They have a full API that you don’t have to actually go down and place a call. Sangoma magically goes ahead and places a call on any kind of signalling and gives you the media stream. That is the whole point of this so that it is very easy for people to build their own applications.

Just a summary. * Unified user space CAPI for both voice and signalling. * object oriented voice architecture, * every component is very well defined. * Clearly define paths – media signalling, voice processing, hardware and the unification of hardware. It was very important for FreeSwitch. This is not a Sangoma product. Sangoma has been a very big contributor of FreeSwitch from day one and we have helped it become what it is today. But again this is not a Sangoma product. It has to be an open platform for everyone to participate in so that is the reason for OpenZap. *easy efficient new stacks and the really important thing about Free Search is that OS agnostic meaning that it works on Linux and windows and on Solaris. That was the motivation for Sangoma – our drivers were going to be agnostic, our drivers were going to be used in Linux, Windows and identical IPS.

So where is the research in all of this? As you will see FreeSwitch is not in the picture here. FreeSwitch is just a custom application using OpenZap. There is no TDM processing inside FreeSwitch like PDM is like another channel, one channel, for all PDM processing. At Zapta we had a chance at DAHDI, at SS7 we had a chan chan chan for every protocol. You know there is a huge amount of gold implication going on and bug duplication and because all the channels do the same thing. So why do we have to write all the channels differently just because the protocols are different? Like Chan Zap and Chan SS7 doesn’t have hard IMF processing just because someone add the IMF protocol.

So again free switch is one channel inside the top OpenStack and opens up channels, all the signalling and all the media and provides FreeSwitch with just the a very simple, you know – Here is a call, here is the voice channel on span 1, chan 1 and then FreeSwitch goes ahead and processes it and sends it to which ever processing engine? Makes Sense?

As you can see, with this architecture it is very easy for us to change Zap, for example media processing engine. If we want to run FreeSwitch in a high performance mode, where we have Span based API, meaning that we don’t have the huge turnover context penalty – we just run our high performance PDI which is the Span mode of the Sangoma driver where one devise is per each one. But FreeSwitch itself doesn’t even know about it. It doesn’t actually know that is running in this type of performance mode. All it knows is that all of a sudden, the system is much freer and has much more processing power. Again, only one part of the system has changed. It has nothing to do with signalling, nothing to do with libraries – it is a sign of a well defined system.

As you will see up here one alkalis heal in FreeSwitch right now in the TDM world is its Stack are lacking. The stacks are ok, but they haven’t been tested as much as Asterisks. Their analogue stacks are actually pretty good, but they don’t have a BRX stack. So how do we solve that? Well, you will see in the previous graph that Sangoma has already built unified API Stacks.

The idea all along was that we were going to integrate all of our stacks into FreeSwitch. That is exactly what we have done. Just to sum what we did with OpenZap we added all our boosts stacks into OpenZap and it looks like this.
Look at the part that we have changed right now. We have not changed the media path. We have just added another signalling channel to OpenZap. This single channel talks to all of our stacks at once. You debug these stacks once and these stacks are already production working with Asterisks. We have just added one component to OpenZap to support all of our stacks.

In combination and working in Parallel, all of a sudden we get this whole PDM section of the code, literally over night and this code is all production green code that has been running for years now with Asterisks. So we are pretty excited about it.

What does FreeSwitch look like in terms of a configuration point of view? Well, since everyone knows how OpenZap works, I figure we should draw parallels. We start with OpenZap. (That first line is actually wrong, they changed it in their latest release. It is all sitting in local user FreeSwitch update) They had them separated before, all the configuration files, but now they think it is the right idea to have them all in the same place. There is a OpenZap.com and it is very much similar to Zaptel or DAHDI. All it does is tell you how many v channels you have and this is where the v channel is and that is it. In the maps the v channels and b channels and that is called OpenZap.com. In that file we say that sense we are going to use our boost protocol, we can specify boost spans etc just by a span ID. In here we can add a different information about what is the context – pretty much the equivalent of the paral.com SP . The extension.com we have it in the FreeSwitch config by default xml. Xml as you know, when you make a mistake it tells you, you made a mistake. It is a little harder to configure, but once you get use to XML you appreciate it.
This all looks complicated to me, so what we have done is we have created a lansian GFS (SP). It is a configuration tool that figures it out for you and that gives you a working system. We do this for Asterisks as well. We have done the same thing with FreeSwitch so we believe that we have enough information that when you actually go through the card we can do this for you.

I just want to show you. This is our flow chart of OpenZap. This is an application. Let’s say that you want to write an application on OpenZap to do something. Literally anything you want. It is about five lines of code with OpenZap which configures in default to specify Zap global in it. Zap goes to Open Zap config file – pulls out the information and continues the structures for you and you pretty much say run. The run spun a thread and goes in and starts working with a signalling stack, when a call comes it, it tells you have an incoming call. The call is for this and this channel and this is the file channel for the channel which you can start using. So very powerful you get a working skeleton system.

So this is my interrupt.uction to OpenZap. It is meant to be an effectual overview.
Going back to our architecture. This system for me still has one flaw. Again it is very monolithic. Everything is bound together like function calls and you cannot rip this system apart easily. Just because the TDM cards are so expensive from the CPU point of view, it would be nice if we could somehow have the parts just working on another system to have Asterisks or FreeSwitch to have the full power over the machine – that would be really cool.
So this is another key, another building block to our puzzle that we are assembling. The TDM
(Very rapid here did not get the product 38.41 into the speech.)
Craig started this because when you run open or his library, people always screwed it up by improperly using a function and crashing the library. The customer says it is crashing it is actually on you to update what they screwed up using. So using a socket to talk to something is very powerful because it cannot crash you. It can crash your own machine, but the socket cannot generally crash the system. So that is kind of a cool way to produce an API. It is away to extract signalling in the media. It uses for control and UPD for media. You might say ‘don’t tell me this is another protocol like SIFF’ – not even close it’s a very simple, no RDP, just pure ulog. You are able to decouple the TDM software from the system and that is the goal.

The server is a server client relationship just like any server sits on the PCP board waiting for connections and a client makes a connection and says, ok place a call. And the even send word back call answered and here is the upc board if you should be listening for your media phone.

So we have created a Chan at Asterisks as well as FreeSwitch that can do this. Now OpenZap has not gone anywhere as you can see here. All that we talked about is still here. But what we have done is we have decoupled that by a ___________ connection (very rapid and underbreath). So now we can start pushing a lot more parts here and let Asterisks and FreeSwitch be free to work in another system or scale, to a much higher level.
With SS7 you get one signalling channel and hundreds of voice channels and this is where boomer SP comes in. Open dot same thing, boomer server using OpenZap talking through FreeSwitch in this channel. Again it is same concept that FreeSwitch has. FreeSwitch has a channel to OpenZap and only one channel and again, one channel to hold all the signalling and all the hardware – all the TDM.

Rather Quickly how does it look like? It text based protocol. So TCP you send up a hello message, hang up message – accept – all standard messages that are in our control protocols, whether it is PRI or SIF it really doesn’t matter. So set up event flow, the call goes out and you get SIF to send out the media.

What does this look like in an woomera (SP) server. A big mess *laugh. * This whole picture was there before. We have already went through this picture. The only thing we added was woomera server on top of OpenZap API and now we are able to decouple the TDM hardware from the actual engine room of the SIF. The reason for this is Asterisks can run a thousand SIF to SIF calls. Why, because it does not have the TDM component. It does not have this hardware interrupt running at every one millisecond. The reasoning is that If Astreck is just running woomera from sockets, it’s the sockets it is equivalent to SIF to SIF. Actually is less because there is no RTP on the woomera site, so it’s able to handle much more calls and it is much more stable.

For example this is our SS7 in the picture right here. Where we have our SS7 Stack Welcome to our open media gateway which is pretty much equivalent to our OpenZap daman talking to the actual Asterisks. We want to define one object that does one thing and one thing well. I want to debug that object once. And I don’t want to debug it twice. It increases stability and Asterisks stability quite a bit.

So again this is an example of our SS7 where we have the SS7 stacked on our wormer SP server on top of OpenZap or single Media Gate Way – they are really interchangeable. OpenZap talks tour our TDM Kernel devise and using Wovmer DCI it sends calls to the FreeSwitch or Asterisks server. From then on it can go to SIF. One SS7 mind handles many voice lines unlike VRI (sp) in this particular case our SS7 solution we can have one SS7 gate way central to our machines. This SS7 x 2 16 handling one E1 gateway. All pushing data through a single FreeSwitch or SIF server.

Your asking do I really want to push 500 calls or a 1000 calls through a single Asterisk machine? No. Again, it is not designed for that so let’s not push the limits of Asterisks. I would rather do a clustering model of a McLeod McCubing where every machine does a will in minutes and this is my favourite feature of wormer (sp) is that inherent flow balancing. I call it real time low balancing. Why? Because it is just a server so two Astreck boxes connects to the same server and says, ok I’m connecting, I’m a listener, send calls please. So what happens is, let’s say this is a 500 voice channel server with TDM. So we would have 16 E1’s here 500 calls, What I can do here is that I have two Asterisks systems connecting to my woomera server saying OK I am ready to listen for incoming calls. The Wovmer server will pass the calls to both machines at the same time. The first one that answers gets the call. Asterisks does not see any of this. All Asterisks sees is oh, I have an incoming call. So all that logic of two calls came in and the first one that answers gets it – if Asterisks didn’t get that call it would actually see it. It is real transparent and real hotwired because I get an immediate feed back of who is busy and who is not. Why? because the two …. yes?

Man: Are you keeping the two Astreck boxes set sync or are you in two separate instances.

Nenad: in this case it is in two separate instances, but if you wanted it to stay single you could. It depends on what your objective is here. If this suggest a SIF 16 gateway, there is no need right. The point is the SIF lives on top of the blue boxes and we all know how to scale a SIF. We have open serve, different technology to work with multiple SIF boxes. The cool thing is that both of these systems are now low balanced and if we loose an Asterisks box the other takes over alone. Now I could put another box. Let’s say two boxes are running and I see that my servers are full, I could in real time add another box and just connect to the woomera server and all of a sudden the Wovmer server just says, ah three listeners now. Send a call to all three of them, and the first one that gets it gets the call. So it is very powerful and works very well in the field and that is the basis of our clustered SS7 architecture.
With one SS7 server handles multiple voice gateways and each voice gateway can talk with multiple Asterisks gateways. So we can have a star architecture where every Asterisks talks with two wormer servers and every wormer server talks with two or four Asterisks servers. From the way down, we are distributing up and on the way up each Asterisks can keep talking to this one or that one. So you can have a variable in a place call this and a place call there. That is how simple it is to provide, you know, just another path. Place a call here, if he is busy, will place a call to another one. And this is production ready. You can use this, this is our cluster architecture. We use this in AT&T and all over the world – France Cellcom. The SS7 is certified by over 50 couples.
Question what is open sourced.

Man: Everything is opened sources except for the SS7. The woomera, the single media gateway, all the parallels – everything is open sourced, except the stacks, we had to buy the stacks so we are not allowed to open them. That is the only reason. It is one of those things. We need a solution now. Sangoma for example, we got a stack …. I am not sure you heard, but Asterisks just got an open R2 committed into Asterisks source and they responded 100% by Sangoma. Sangoma started an Open R2 project and that Open R2 project is now in Asterisks, so we play on both sides, but we have customers who need this solution now. And when we were building the solution, there was no SS7 Stack that actually worked. Just because it is a complicated system we do give support and that is where you have to pay there. It’s like a fraction of what the real solution.

Man: [inaudible]

Nenad: is they use a different ISA Protocol and it was those two are actually on our waiting list. It will take us a little while to get those two up and running. And our customers that needed to go there, they choose a different solution. They stayed with us with other instillations, but with those two instillations,

Man: [inaudible]

Nenad: as long as they use a normal SIF were good, you know. But they use a variant of ISA. As long as we use a similar ISA, we are good. But they have a variant that is completely different.

All the stuff in our multiple machines that you have seen or all the stuff can be run back on a single machine and we are running a 16 E1 gateway with Asterisks and running about 40% system load, 500 channels and we run it for like months and months and it is stable. So if someone says that Asterisks cannot handle 500 calls of SIF media, it is simply not true. Again, I am not running major IGI scripts, I am not running a PBX here. I am running 16 [inaudible] voice trails. Again FreeSwitch, similar performance characteristics. Someone asked if it runs a little faster, I don’t have that information. The reason is we have just started to officially release a support for FreeSwitch so all our drivers will be regression tested against Asterisks and FreeSwitch when we set it up so that is the main thing. We continue to support the open source gateways. We think it is the future. We think Open Source is the future and if we get better Stacks from Open Source we will support those. Distributing, in my opinion is the best way to go. It provides clustering and reliability and no single point of failure.

We have a SIF TDM gateway based on Span Mode. Just to show you how Span mode works. We are running a 1000 Sims PDM calls on a quad core running with a 50% load.

[voice drops very difficult to hear]

Gate is based on the Stack mode. Well that is about it. Any questions?
Question but very difficult to hear as was the answer.
Thank you for your time.

[applause]