In my last blog I have covered the ways the physical infrastructures of large-scale cloud service providers are built. In this blog I intend to explore the network virtualization aspects of present and future infrastructure builds. My focus will be mainly on the datacenter oriented networks.

As a part of this exploration, I plan to cover the following:

1. Why incumbent technologies like VLAN fall short?
2. What are the new network virtualization requirements?
3. What’s missing in current approaches?

Why incumbent technologies like VLAN fall short?

Even though new approaches are emerging, a huge number of existing datacenters and enterprise networks are still being built out of VLAN technologies that expose operators to substantial scaling and operational risks. These have been beaten to death and most people have a good understanding of the issues. However, for people looking for a refresher I am going to cover them in more detail at the end of this blog post in the section “What’s wrong with VLAN based approaches?”.

What are the new network virtualization requirements?

Given the inability of incumbent VLAN technologies to support machine segmentation at-scale, it is imperative to have a closer look at the problem space itself. Is there a true need for machine segmentation at lower level (L2/L3) in a cloud-based infrastructure? I think the need for segmentation has become ever more pressing and prominent for common multi-tenant and enterprise use cases.

Here are some of the network virtualization requirements of the new infrastructure builds.

•  Machine compartmentalization

The compute and storage resources of modern datacenters are being shared by numerous public and private entities that have variable needs and security practices. For example, some entities store and process mission-critical data, which cannot be compromised, whereas others might process data whose loss will have minimal business impact. Depending on the sensitivity of the data, the level of security hygiene varies. For example, someone could run an OS with known security vulnerabilities, exposing other machines in the same domain to hackers.

• Application of value-added services for machine pools

Network Virtualization can also be leveraged to impose service-chaining capabilities. For example, let’s consider a set of virtual machines that carry sensitive data and other virtual machines that have different roles and are exposed to different depth of security exploits. In such cases one might want to scrub all traffic going towards the first group of machines from the second group using DPI (Deep Packet Inspection) capabilities. In a complicated and distributed setup that carries a variety of machine pools (Virtual Networks), having a well-orchestrated service chaining and policy-based mechanism is important.

• Dynamic spanning of machine pools

In the older paradigm, groups of machines had to be physically co-located as mandated by VLAN technology. However, currently these machines can reside anywhere within a multi-datacenter (and possibly multi-cloud) build out. So, there needs to be a capability by which these virtual networks can span dynamically based on the application needs.

• Extension of Enterprise networks into Cloud

Enterprise networks are actively looking at public or private clouds to extend their networks as a mechanism to flexibly access more resources. In many cases, however, an enterprise network running a block of RFC1918 space may want to extend seamlessly into a private or public cloud environment without altering its addressing scheme. The address block used is typically ingrained into the security policies that are used by these enterprise networks. In some case the enterprise might be heavily using L3VPN (RFC2547/4364) from the providers and they just want the cloud services to resemble a set of VLANs connected to their VPN cloud. Hence, mechanisms are required that can enable automated and seamless extension of these networks. Some large providers have used IPSEC-based point-to-point tunnels to connect these address spaces, but as we have learned from the CPE (customer premises equipment) based VPN space, provisioning and operating scalable L3-based mechanisms (i.e. provider L3VPNs) is a lot easier than operating an IPSEC tunnel mesh.

What’s missing in current approaches?

Recently some proprietary overlay approaches that have tried to tackle the above problem spaces. These approaches remind me of the VL2 work that I did at Microsoft in 2008-2009 timeframe – It was supposed to introduce network virtualization within a single datacenter by encapsulating the payload after doing some smart lookups in a central machine where all the information was derived from a central database. NVGRE was used for overlay in the data-plane in VL2, while other newer approaches have proposed VXLAN. Each of these data-plane methods has been proposed in IETF towards standardization. Every Cloud Operator and SDN vendor has its own proprietary machinery that cannot interoperate with another system – even though a standards based overlay (VXLAN or NVGRE) may be able to exchange packets, lack of control plane will prevent disparate systems to comprehend them in any significant manner.

Over the last year, however, some approaches have shown up at IETF, which leverages constructs that have proven to scale well within Internet. Proper control plane interactions using standardized mechanism fosters a multi-vendor environment that keeps the vendors honest as well as enables the virtualization models to scale.  I see promise in the L3VPN-End-System draft as well as some on the E-VPN work that is happening. However, I have never been a fan of extending the L2 services beyond a rack or so. So lesser the L2 capabilities bleed in better it is, to my mind. Applications should shred any dependency of L2 services like broadcast etc.

In addition to a lack of fully standardized interoperable control plane, I have observed that each of these newer overlay based systems implement their own independent northbound API for provisioning systems and applications/services. This works well for a cloud provider with significant internal resources to build the entire system end-to-end (think – Google, Amazon, etc.). However, this can present a major challenge to enterprise customers as well as emerging cloud providers as they get locked into a vendor or a provider. .

To summarize, what is missing is the capability to orchestrate the topology, segmentation and policy in an automated and standards-based way within a data-center and on a multi-datacenter/multi-cloud basis. This becomes quite a challenging and complex problem as different cloud operators have their own variants of the compute and storage orchestration systems. However, since network is all-pervasive and fundamentally critical glue, standardization is extremely important in this area.

This field is not yet matured and there will be many constructive and innovative efforts to shape out this area. Initial innovations around network virtualization and software-defined networking suggest a much larger inflection in scale-out system design and architectural thinking in coming days.

What’s wrong with VLAN based approaches?

• Physical Span of VLANs within Datacenters:

VLANs are built on spanning-tree based protocols that are quite temperamental and fundamentally weak in nature as a control plane protocol. A switch domain implementation with more than two layers of hierarchical switches and more than very few topology loops is prone to witness operational issues.

In general, two or more disparate L2 switching domains are never inter-connected within a datacenter. So a VLAN built in a particular co-location area within datacenter is seldom spanned beyond that. Even if a VLAN needs to host more number of machines and the co-location area has run out of space, VLANs rarely get extended. This in turn results in the VLAN based network segment getting fragmented into multiple VLANs, which complicates the packet filtering rules on the routers exponentially.

• Scale of VLANs:

The best practice is not to carry more than 500 hosts in a VLAN. This is due excessive cumulative broadcast traffic that gets generated as well as the unicast flooding that hit each box in case the switching system loses a MAC address. Additionally, the maximum number of VLANs that is supported by Ethernet frame format is 4K. But operators can’t even approach 4K VLANs in most cases, since gateway routers run out of HSRP/VRRP scaling long before the 4K limits. I have seen well-established and popular datacenter aggregation platforms unable to scale beyond 500-600 HSRP/VRRP sessions.

• Scale and complexity of Packet filters on routers:

In many cases, the packet filters (ACLs) on routers are built through simplistic scripts etc. Many environments are faced with situations where the ACLs are completely out of control. People can neither clean them nor consolidate them due to the uncertainty they would induce to the operational environment. Inability to manipulate the router ACLs in an automated and nimble, yet reliable way has turned out to be a nightmare for many operators.

• Spanning-tree in CLOS network:

Most modern cloud datacenters run networks with CLOS based architectures that inherently don’t work well with spanning-tree types of protocols. Spanning-tree operates by putting links in blocking state (for one or more VLANs) to prevent frame loops, but CLOS architectures operate with the notion of equally distributing flows over multiple parallel paths in a semi-mesh topology. People have looked into other L2 control plane protocols (TRILL etc.) to replace spanning in a CLOS type environment, but those approaches have demanded new hardware and vendor- specific customizations that have forced the operators to shy away. Multi-chassis based port-channel approaches have introduced some sanity in spanning-tree world, but they wouldn’t be as applicable in CLOS architectures.

By Parantap Lahiri – VP Solution Engineering, Contrail Systems

 Fig. 1 shows a large-scale cloud provider infrastructure. Typically these networks are global, richly connected to the Internet and use technologies that are optimized for each subset (shown as areas A – G). While architecting these infrastructures, it is critical to understand how different areas would scale and mutually interact with each other to create a dynamic but cost controlled service.

Each area has distinct characteristics as discussed below.

A. Intra-DC Network:

This is where most of the servers reside. The traffic profile has changed over the last few years and the applications demand a lot of lateral east-west bandwidth in addition to the usual north-south bandwidth. Multi-tier CLOS is a popular style of network topology for such areas and mature IP control planes have proven to be excellent ways to interconnect the network elements.

IP is ubiquitous, but VLANs have a dependency on L2 technologies. For VLANs, there is also an inherent expectation that the servers are in close proximity and that server-to-server latencies are low. Traditional L2 protocols are weak, and they don’t scale beyond very limited physical proximities. So for cloud applications, L2 dependencies present a handicap. Hence, barring a few legacy applications, most modern applications operate in a pure L3 world beyond rack level. Of course the applications have to instrument ways to understand server affinity/proximity and have to minimize on inter-DC traffic loads. But on the other hand, L2 dependencies confine the servers running the applications to a smaller physical segment of the datacenters (co-location areas) and pose significant scaling challenges. For these reasons, most modern applications have eliminated the dependencies on L2 networks.

In summary, the following are key characteristics of intra-DC networks:

• Abundant network capacity within datacenters, low over-subscription and dense but topologically symmetrical networks
• Services like Firewalls, Load balancing, IPS/IDS that had traditional L2 attach points are being upgraded, often as VMs or virtual appliance. Services are being architected to operate on native L3 networks.
• Dependence on L2-centric topologies and protocols are being minimized
• IP networks heavily leveraging ECMP and operating on robust L3 control plane protocols are gaining traction.

B. Local Cluster of DCs:

This setup is only used for really large-scale cloud providers. Since applications breathe well with available network bandwidth, having richly connected datacenters in close proximity has been a common approach. In these cases, the datacenters are either connected by high number of single mode fiber cables or mux-ponder technology is used to carry multiple wavelengths (lambdas) over lower number of fibers. Available technologies can easily push eighty (80) 10 Gbps bandwidth carrying wavelengths over a single fiber pair.

The main challenges in this segment are the network topology and termination cost on the routers. If typical WAN routers are used to terminate the lambdas, then given the number of ports needed, the overall cost can be prohibitive. The unit cost of colored optics needed to feed the mux-ponders also comes into play. Network topology wise, one must be thoughtful about where to put the different components of a distributed network, since clustered datacenters typically go live in phases. So if the first datacenter carries all the aggregate “spine” routers, all the subsequent datacenters will need to connect back to the first one with an appropriate fiber length. The distance between the datacenters are important since the optical technology cost can go up significantly after a certain distance.

C. Metro cluster of clusters:

In this segment either the capacity (e.g. 10Gbps links) have to be leased or metro dark fibers have to be leased. For people who have heavy traffic needs between the clusters, dark fibers have been the choice for cost reasons. Mux-ponders light up dark fibers in many cases, and colored optics are used on router termination points. In other cases, transponders have been used. Each of these optical technologies has its cost points and benefits, often subject to unique architectural and business considerations.

D. Internet Core:

This network exists for large-scale cloud providers who connect to Internet globally at multiple points with different type of fiscal arrangements. Delivering traffic to the right hand-off points is extremely important to reduce the egress cost as well as to maintain the quality of user experience.

This layer typically carries the whole Internet routing table along with multiple alternate BGP paths. Traffic engineering mechanisms are common in this layer. Some people have used RSVP-TE with MPLS, some others have used out-of-band reservation mechanisms. Typically this layer gets run similar to the core of Tier 1 Internet Service Providers. Some people have long-haul dark fibers and some have used a mix of dark fiber and leased lines. For smaller provider a default route or subset of Internet prefixes can be carried in this core.

This layer has both a L3 routing core along with an optical backbone in many cases. Some people are working on converging the two layers with packet-optical core, by formation of a super-core that doesn’t carry Internet prefixes. Networks have also instrumented QoS and L3VPN capabilities in this segment to instrument WAN virtualization and honor service level agreements. As this capability has matured, it has also become table stakes for smaller cloud operators

E. Inter-DC bypass network:

People using this network have high-capacity requirements between the datacenters that are not located in the same metro. Since the number of prefixes carried in this network is limited to internal prefixes, packet-optical integration has worked out naturally in many of these cases. Dark fiber has been the choice for most people. The ultimate goal is to connect all datacenters with abundant bandwidth so that applications can scale-out with fewer constraints.

This is one area where Open Flow based mechanisms have seen some traction. Traffic Engineering based on bandwidth reservation/calendaring and smart traffic placement has been worked on. Since the termination cost has to be low, people have tried to avoid more complex mechanisms (i.e. MPLS RSVP-TE), despite their pervasive deployment by large providers.

F. Satellite Networks at Edge:

These networks are used similar to the edge nodes of CDN providers. The service delivery points are taken closer to the end-user to improve the user experience. Caching is used heavily along with many other techniques. Connectivity to Internet is rich at these locations. Some people terminate the user connections here and then direct the user traffic within the cloud infrastructure to provide the best experience. In many cases these locations can be even just few servers that are embedded in the Telco premises.

G. Access to Public and Private Networks:

Large cloud infrastructures connect to Internet at almost all areas expect the Inter-DC bypass network. The number of connections to the Internet depends on the scale of the provider as well as the amount of traffic originated/consumed. Many providers have settlement free peering relationships, ensuring zero-cost traffic exchange with other larger networks. Policy based network ingress/egress is an area that has seen good amount of work. Policy-based network optimization ensures that selective traffic can be flexibly delivered or attracted via appropriate Internet connections.

Large providers can also have private access arrangements with Enterprise customers. These access points are typically used by the Enterprises to access dedicated services hosted by the providers. In some of these services the Enterprise customers want the hosting providers to use the same address space as used by the Enterprises within their own networks. This creates a situation where multiple overlapping addresses have to be used by the cloud service providers between the access points and the datacenters. Typically L3VPN technology has been used to provide scalable VPN address segmentation in the backbone network. This is one of the areas that I will cover in more details in future posts.

Virtual Network Implications:

The lessons learned from automation of distributed control systems such as the Internet are invaluable as cloud operators scale overlay virtual networks.
For cloud providers, supporting seamless virtual networks across these diverse segments is critical. Tenants can extend their existing enterprise networks across such a cloud infrastructure, extending their available capacity and ensuring a significantly higher level of global business agility.

In future blogs I intend to explore the relevance of network virtualization with respect to the physical networks. More importantly, I will explore ways to evolve existing architectural paradigms to enable interoperable, scalable and resilient software platforms that protect and align with the current and future physical infrastructure investments.

Parantap Lahiri – VP Solution Engineering, Contrail Systems

Many of us are truly excited about the resurgence of networking as cool and happening given the popularity of SDN. However, irrespective of the ability to programmatically define the network topology or network behavior, to my mind the datacenter network architecture and platforms have gone through a profound change in last 5-7 years.

Most of this commentary draws from my experience in architecting very large-scale network infrastructure for Microsoft Online Services and UUNet.

Even mid last decade the standard network topologies used to be multi-tiered architectures as shown in typical Cisco books. The aggregation, distribution and core layers used to be typical large L2-L3 chassis devices that not only were complex but also resulted in a severely oversubscribed topology due to the extremely high cost of each port. In most deployments the boxes were deployed in pairs and an additional layer of complexity was imposed just to ensure that there is no outage in case of any platform or component failure. Frequently many of these layers were burdened with long packet filters (ACLs) to institute segmentations of networks, which has been one of the primary ways to implement security policies.

In a typical enterprise application environment, scale required by individual application is small and as a result the web-servers, application servers, databases/storage all share a simple network that covers only few racks. In such cases, having heavily oversubscribed upper layers of the hierarchy might not be as impactful.

However, for large web-scale environments the application needs had been drastically changing in the background and the paradigm of over complicated network layers with limited expensive bandwidth was being challenged severely. The typical north-south bandwidth demand was challenged by enormous amount of east-west bandwidth demand from emerging cloud and web services. The only way for the cloud services to succeed has been a horizontal scaling model that results in heavy workload between the servers. Enterprises are starting to experience similar needs with emerging applications like Big-data, etc.

Fortunately enough, the merchant silicon based platform evolution had already started to get foothold and it has been an excellent opportunity to put them to test. At present day, people have built large-scale network fabric within data centers that would allow near 1:1 over-subscription end to end anywhere within a datacenter. There are few aspects that have been tightly controlled to execute on such efforts.

1. The identification of topology is extremely important. For example, if the topology is well thought out, connecting 100K servers with each other over some number of 64X10G 1RU commodity routers, is not that difficult. Of course variations of CLOS architecture and innovative approaches come out handy here.

2. Management and operations aspects have to be thought through. It is one of the most crucial but misunderstood parts of the whole equation as managing and troubleshooting large CLOS networks can be non-trivial.

3. The right set of control plane protocol has to be chosen keeping in mind the present and future scaling needs.

4. There would be tons of ports, optics and cables needed here; so good understanding of cost structure of platforms is beneficial.

Inside datacenters large capacity symmetrical networks covering the whole datacenter or individual cluster can be easily built using routers and optics at commodity price points. In such networks ECMP based flow hashing works well if one assumes a good mix of the sizes of the flows generated by the applications. There have been concerns about elephant flows getting clubbed and creating hot spots, but those concerns are reduced if the server interface speed and infrastructure interface speed are 4X to 10X apart.

Here is the link on the work that addresses the control plane solutions on commodity platforms in a large real-world deployment. In order to build large-scale networks within datacenters, one need not introduce complicated logic into the control or data plane and try to drive every bit of link utilization possible. The fact is that bandwidth costs have been going down drastically and topologies can be easily constructed so that bandwidth is abundant. Adding more complexity can prove detrimental to operational stability and reliability.

Implications of datacenter evolution:

The datacenters built by large-scale cloud service providers have evolved in profound ways. Network is the fundamental enabler to cloud services and plays a pivotal role in following aspects:

1. Enabling the compute and storage resources to interact with each other in an unconstrained way based on application needs. Of course, in many cases the compute and storage have some affinity but they are spread around globally. So many serious players are building high-bandwidth global backbone.
2. Connecting the cloud to Internet for web based access. Many cloud providers will connect to Internet at multitude of egress points with commercial arrangements with settlement-free peers, local and global transit providers.
3. Segmenting users of cloud resources from one another by creating virtual network space and allowing appropriate controls between Virtual Networks.
4. Connecting the virtual network space to appropriate enterprise network domains, or to other virtual networks/domains as a part of a federated model.

The first two areas are clearly going to be addressed by the physical networking domain. Going forward the industry is going to see enormous innovation as the speeds and feeds are going to evolve from 40G to 100G to 400G and so on. Aspects like port density, interface mix, power draw, optical component integration and optical interface support etc. are going to define the coming generation datacenter switches. The switches will be more optimized with higher non-blocking forwarding capacity as opposed to proportionately higher control plane scaling capacity. Analytics, diagnostics and smart traffic engineering will be redefined as a result of the innovations in the physical networking space.

Vendors making the right platform choices for their customers are going to give their customers significant edge over their competitors. Getting locked with a single vendor would significantly reduce customers’ chances to leverage the innovation prowess of the industry.

The areas covered by third and fourth points will move virtual network infrastructure more naturally into the server space. The servers with their sizeable memory and flexible compute resources are well positioned to absorb the network edge and provide required scale. Industry standards are critical for cloud entities to interoperate with each other. The global cloud infrastructure will carry tons of business data that will be open for consumption in a streamlined way. A virtual network infrastructure with well orchestrated segmentation and access logic can potentially prove as an enabler to such a market place.

Conclusion …

The datacenter space running cloud infrastructure has already gone through some significant shifts in last several years, even before the arrival of the SDN approaches. The shift happened due to solid business needs driven by the emerging applications whose traffic demands had morphed from traditional requirements.

In the coming years, the datacenter space is going to see some very interesting progress in both physical networking as well as the virtual networking world. For the physical networking piece the Top of Rack switch could become the boundary of the network, but for the virtual networking piece the servers will become the new network edge.

The physical network evolution will be encouraged by adoption of multi-vendor approaches by cloud service providers. That will push the vendors to innovate aggressively and continue to reduce the cost and/or increase capacity as they move units of data around the datacenter. On the other hand, the network virtualization area would need well-vetted standards and scale-out approaches to become an enabler for cloud evolution.

Customized and non-standard solutions for virtualization have definitely done a good job in bringing awareness to the cloud space network virtualization. However, enterprise cloud infrastructures spanning over multiple public and private clouds will become the norm going forward. It is hard to see a world without standardization in the virtual networking space leveraging the well-known distributed system principles that we have learned through Internet evolution.

By Parantap Lahiri – VP Solution Engineering, Contrail Systems

A news-worthy week, for sure.

With the news of VMware’s $1.2B acquisition of Nicira and Contrail Systems’ Series A funding announcement, it’s been an exciting time in the SDN space.  The whole industry is suddenly jumping in on a passionate dialog of how to achieve customers goals of open standards-based, scalable SDN architectures, and how to optimize operational efficiency.

Thanks to Om Malik for a nice writeup on our Series A funding:

SDN Startup Contrail Systems gets $10M led by Khosla Ventures

http://gigaom.com/cloud/software-defined-networking-startup-contrail-systems-gets-10m-led-by-khosla-ventures/

———————————

It was picked up by many, including Dow Jones VentureWire:

Khosla Ventures Puts $10M into Stealthy Contrail Systems