Openflow for Wi-Fi offload

Software defined networking is separating the "Brains" of the switches and routers from their "Limbs". So, the routers and switches become plain "Packet pushers" (Limbs only) and a separate entity that sits in the "Cloud" (Brain) will instruct them how and where to push the packets. This is a disruptive networking paradigm which creates space for a lot of innovation. There is also a protocol called Openflow defined that specifies how the cloud entity (Called as an Openflow controller) should talk to the routers and switches.

So, I have been trying to think about how this new networking paradigm could help in the mobile data offloading area. I feel by making the Wi-Fi access points "Openflow enabled" we can do mobile data offloading in a more efficient way.

Currently the Wi-Fi gateways like PDG/TTG/ePDG have an interface with the PCRF entity in the mobile core network. The PCRF contains a rich set of time based, subscriber based and application based policies that can specify the offloading rules. So, mobile data offloading can be done at these Wi-Fi gateways by using these offloading rules downloaded from the PCRF. These gateways can also apply QoS policies to the individual flows based on the policies in the PCRF. Currently these PCRF policies specified by the operator could be used to apply QoS rules and offload internet traffic as shown below -


One of the issues with the above model is that the PDG/TTG boxes today which also play the role of a security gateway are hosted in the centralized data centers of the operator. So, if say 80% of the traffic is to be "Offloaded" at these gateways, all of this traffic will have to be back-hauled unnecessarily to these centralized data centers before being offloaded to the internet. This is inefficient from both an end user experience perspective as well as a routing perspective, as the content is getting more distributed and is moving closer to the end users.

By making Wi-Fi access points Openflow enabled, we have a way to download all the operator's QoS and offloading rules right down to the access points. This will enable the access points to offload data and save the operator's back-haul cost as well as improve end user experience specially for over the top video traffic. The below diagram illustrates Openflow enabled offloading -


What other benefits can we get from putting an Openflow client on the Wi-Fi access points? Well, Openflow is a protocol that enables "Software defined networking" which is a revolution that has begun to happen all over the industry. Soon we will be seeing a bunch of Openflow compliant services available in the cloud. Since these will be pure applications, we can expect several innovative free and open source applications that can add value to the access points available. So, by enabling the access points with Openflow, a lot of this work can be leveraged in the future to enrich the capabilities of the Wi-Fi access points. Also, there's an implementation of Openflow client available that runs on top of OpenWrt today. I wonder if any Wi-Fi access point vendor is working on making their access points Openflow supported today..

Wi-Fi offload and onload at the same time

The operators want to have their cake and eat it too, and it's possible with Wi-Fi offload!

So, Wi-Fi offload is a no-brainer for the operators because of the following reasons -

1> They save spectrum
2> They avoid congestion of their core network elements
3> They keep their customers happy by given them cheaper internet access with better speeds

Of course the operators would love to offload their subscriber's to Wi-Fi. But, at the same time they would like to retain control over their subscribers even while they are on Wi-Fi. The Wi-Fi offload architectures are evolving to meet this requirement.

In the beginning, there was "Pure" Wi-Fi offload:


Then IWLAN based offload, where everything was first offloaded and then on-loaded to the operator's core network, so the operator's could retain "Control" of the user even when he was on Wi-Fi

Then came "Selective" offload. In this model, everything is offloaded to Wi-Fi, however a few selected flows are on-loaded back to the operator's core network and the rest is offloaded off to the internet at the Wi-Fi gateway. That way the operator could retain "Control" over their subscribers even when they were on Wi-Fi, and at the same time save their core network elements from congestion. They would on-load only flows that mattered to them, for example for access to their walled garden services, and for certain services for which "Seamless mobility" between 3G/4G and Wi-Fi mattered etc. They would offload the remaining traffic to the internet before it could enter their core network elements.


While it may seem that the Wi-Fi gateway located at the edge of the packet core is the natural place to do this "Selective offloading", it is not a very efficient solution from the routing perspective. This is because all of the internet traffic has to be first back-hauled to the operator's core network, before decided to offload them back into the internet.

A more optimal model would be to do the selective offloading at the Wi-Fi access points itself -

The above model will be very efficient from a routing perspective specially for traffic that is latency and jitter sensitive. Pushing the offloading rules from the operator's core network down to the Wi-Fi access points may be a bit tricky, but nothing that cannot be solved. I wonder if any Wi-Fi access point vendor is attempting this today...

Towards an "Evolved Packet Edge"

There is a lot of discussion about the "Evolved Packet Core" in the telecom world these days. However, there is a lot of 'Evolution' happening on the edge of the mobile internet which gets lesser attention. In this article I shine some light on the evolution happening at the edge. There are 2 major themes driving this evolution of the network edge -

1. Small Cells (Include both the lower capacity base stations (3G/4G femtocells and picocells) and Wi-Fi access points in this category) - Given that spectrum is limited and too expensive, operators have realized they have to include a large number of these small cells as a part of their network. The inclusion of small cells in the network leads to a huge increase in the number of "RAN nodes" that the mobile core network needs to talk to, thus disrupting the scaling equations of their core network elements
   
2. A large part of the mobile data traffic is video and this traffic is very latency and jitter sensitive. So, the Content Delivery Networks (CDNs) are becoming more and more distributed to deliver content from a point nearest to the end user

Both of the above points call for a distributed architecture for the evolved packet core, in order to scale better as well as to provide a better end user experience. The SAE-EPC architecture allows for distributing the evolved packet core network elements and locating the S-GWs and P-GWs closer to the end users. However, most of these network elements are built on top of the same hardware platform used for SGSNs and GGSNs (3G core network elements). Also, in most cases multiple logical elements are physically co-located on the same box in order to reduce the amount of signaling messages between the different logical elements in the EPC network. For these reasons, most of the available EPC elements today are better suited for a centralized deployment model.

While the EPC elements may eventually become more distributed, it may not happen in a hurry. So, until then what are the options to deliver a better end user experience and scaling of the network? The answer seems to be to move towards a more distributed architecture using "Edge gateways" like the Femtocell Gateways and WiFi access gateways that are defined by 3GPP, as shown in the below diagram -

As shown in the above diagram, these 'Evolved packet edge' devices like the Femtocell Gateways and the WiFi access gateways can be more distributed, and could support "Selective" offloading of the internet traffic to the internet POP directly without routing this traffic to the packet core.

These edge devices will need to talk to the EPC elements for all the control plane signaling, for authenticating the users, for downloading the offload related policies etc. So, the EPC elements will still be fully in charge of the sessions, but then they could decide that a certain set of "Flows" need not be routed all the way to the core network, and decide to offload them away to the internet POP at the edge itself. To enable this selective offloading there are certain standard (SIPTO comes to mind) and a few non standard mechanisms.

So, in a nutshell, by employing these "Evolved packet edge" network elements and distributing them strategically, the operators can gain the the following benefits -

1. Reduce the load on their core network elements
   
2. Offer a better end user experience
   
3. Reduce cost by getting rid of routing inefficiencies caused by the centralized approach

Given the strategic position these edge devices are located in the network, they should be able to participate in the network in many innovative ways in the near future.

Handover problem with LTE Femtocells

The 3GPP standards (36.413, the S1-AP specification in particular) lack support for inbound handovers to an LTE Femtocell (a.k.a HeNB), which is behind an HeNB Gateway. The problem comes while trying to route an S1 Handover Request from the MME to an HeNB, via the HeNB gateway.

The diagram below illustrates the scenario -

• To initiate the S1 Handover procedure, the source eNB sends an "S1 Handover required" message to the MME. This message has the TAI (Tracking Area Identity) and the target eNB Id in it. Note that the target eNB ID in this case will be unknown to the MME, since that belongs to an HeNB that is "Hidden" from the MME by the HeNB gateway.
• If there was no HeNB gateway hiding the target eNB, the MME would simply send an S1 Handover Request to the HeNB directly as indicated by the 'target eNB Id'. But since in this case the target eNB is an HeNB which is hidden behind the HeNB gw, the MME will have to send the S1 Handover request to the HeNB gateway instead
• The MME could send the S1 Handover request to the HeNB gateway by using TAI (Which should map directly to the target HeNB gateway) OR alternatively, by using the target eNB Id and a "Subnet" concept. The subnet concept is one in which HeNB Ids contain the HeNB gateway ID also as a part of their ID, so that MME could find out the HeNB gateway ID corresponding to a particular HeNB using the high order bits and masking out the low order bits. So, there are a couple of ways for the MME to find out to which HeNB gateway the S1 Handover Request message should be routed to. So, there are a couple of ways by which the MME finds out which HeNB gateway to route the S1 Handover request to.
  
• The problem arises when the "S1 Handover Request" reaches the HeNB gateway... This message doesn't have any target eNB ID in it!!!
  
• So, how would the HeNB gateway route the S1 Handover Request message further down to the actual target HeNB? Unfortunately this is not handled by the 3gpp specifications as of today
• The HeNB gateways will be able to route this message to the HeNB via a hack.
• The HeNB gateways will have to peep inside the "Source to target transparent container" IE, to get the target cell id from inside it. This breaks layering and renders the "Transparency" of this IE ineffective.
  

To solve this problem, the 3gpp S1-AP specifications will need to be updated to include the target eNB Id as part of the "S1 Handover Request" message. Until then the HeNB gateways will need to do with the above workaround.

Trouble over-riding libc calloc/malloc/free etc

Ever tried over-riding the libc allocator functions? I did and the journey was not as smooth as I'd anticipated it to be. My reasoning was that if I implemented the overriding functions malloc/realloc/calloc/memalign/valloc/free etc in my application, then they would get picked ahead of the libc functions by the dynamic linker, since they lie in the main application code itself and the whole program would use these versions instead of the libc functions. My thinking was almost right (The dynamic linker put my versions AHEAD of the libc versions and all dynamic symbol resolution led to my functions getting called - in fact even the libc strdup function that called malloc, ended up calling my version of malloc instead of the libc malloc itself).

However, there was an exception. One particular function _dl_tls_setup (Implemented in the libdl library) ended up calling the libc-calloc!!! This led to a crash at process shutdown time. At process shutdown, _dl_deallocate_tls tried to free up that memory, but then it called my version of free!!! However, my version of free expected a different header than what libc had put in that chunk and thus crashed the code.

Why did this happen? Why didn't the libdl library not consult the dynamic linker for resolving the calloc/free symbols? When faced with such questions, its best to study the running code using GDB. I attached GDB to the running process and did the following -

(You could refer to my GDB log file as you read the steps below)

1> dis-assembled the function _dl_tls_setup
2> Figured that the call to calloc was consulting the PLT (Procedure linkage table) entries for the libdl library
3> I then dis-assembled the PLT entries at the address mentioned in the previous step
4> That led me to the GOT (Global offset table) by the libdl library
5> Note that each shared library has its own PLT and GOT entries that has mapping information for each and every dynamic symbol that it uses that is defined by some other library
6> The GOT contained the address of the calloc function that was being referenced by _dl_tls_setup
7> I then did a 'info sharedlibraries' call in GDB which showed the memory mapping information of the entire program
8> From that I could see that the calloc address fell within the address bounds of the LIBC function
9> Doing a similar exercise (Steps 1-8) for the _dl_deallocate_tls function that was calling the 'free', I realized it was calling the free defined by my application code

Upon posting some questions to 'Linux Dynamic linker' gurus, I finally got the following answer -

"The libdl library is very special. Since it has to do some allocations before all of its dynamic linker structures are setup, it maps to libc-calloc at startup time without really going through the dynamic symbol resolution step. However, only calloc is called during the phase when the dynamic linker symbol resolution mechanisms are NOT YET set up. By the time it calls 'free' all the dynamic symbol resolution mechanisms are setup already and it ends up picking the 'free' from your application"

Aaha! So, how did I get around this issue? For now, I have gone for a simple hack - If the process is 'Shutting down' then if 'free' finds some header mismatch, simply return from the function ignoring this chunk. Well, never mind if this memory leaks, since anyways, the whole heap used by the process will be released upon process exit in any case. I'm almost certain that except during the process shutdown period (When the _dl_deallocate_tls gets called) this scenario will never occur.

Does anyone have a better idea? Or some more gyaan about this topic? It would be interesting to hear from you.

ESC India 2007

The 'Embedded Systems Conference' was held for the first time in Bangalore this year. It was a great success considering that it was being held here for the first time. Industry stalwart Jack Ganssle kicked off a series of interesting technical presentations. I hosted 2 classes this year - 1> C++ Templates and STL and 2> Concurrency design patterns.

In my past experience I have observed that engineers in the embedded domain shy away from templates for many reasons. So, in my class I explored the many myths surrounding C++ templates and also talked about some real issues with templates and the workarounds for those issues. Then I went on to extol the virtues of generic programming by taking the example of the design of the STL.

In the other talk, I spoke about the various design patterns in the domain of concurrent software. Much of the material was collected from my past experience in developing highly concurrent networked software for telecom/datacomm products. I talked about the C++ free and open source toolkit called ACE that implemented many concurrency design patterns with some examples of the patterns taken from the toolkit.

Going by the amount of questions I got from the audience, both during the talk and what followed via e-mail conversations, I think both talks were highly useful to the audience. It was a very satisfying experience for me. Looking forward to ESC India 2008.

C++ Pointers and Memory Management

Pointers and memory management form the most error prone area in C++. Thorough knowledge of pointer related pitfalls and traps is a valuable asset for any programmer. The fundamental issue with pointers is that there is no strong coupling between the pointer and what it points to. Pointers could be freely assigned to random memory areas, copied etc, without doing anything about the objects these point to. So, its very easy to end up with memory leaks and corruption. One way we can make pointers safer is to build a strong coupling between pointers and the objects they point to. A popular technique for doing that is called the 'Smart pointer' idiom in C++. Smart pointer based techniques such as scoped pointers, reference counting and copy on write, go a long way in making pointer related code more robust and efficient. I recently gave a talk on this very topic. You can find the slides of this talk at this slideshare link.