Delivering new, transformational capabilities increasingly requires that we develop for ourselves competencies which we’d previously turned to our suppliers for. Our experience building Azure public cloud services over the last several years bears this out. We’re now developing our own server infrastructure and investing in innovative silicon technologies, such as field-programmable gate arrays (FPGAs), to accelerate our workloads — things that would have been unheard of years ago.
At the Open Compute Project 2019 conference, we unveiled Project Zipline. Project Zipline is comprised of a new, cutting-edge compression and encryption pipeline and the contribution to open source of the register transfer language (RTL), which is used to implement it in hardware. Azure's dramatic growth has rapidly increased our need for optimized data processing and storage systems. Zipline enables compression without compromise by simultaneously delivering higher compression ratios, higher throughput, and lower latency than other solutions available today.
Today, we announce the companion technology to Project Zipline — Project Corsica. Over two years in the making, Corsica is our ASIC implementation of Zipline technology. It delivers Zipline’s first rate performance in compression, encryption, and data authentication, all accelerated in a special purpose ASIC developed by Microsoft in collaboration with Broadcom.
Internal innovations in data storage and encryption
A massive surge in demand for data is taking place, straining the industry’s storage capacities and data management capabilities. At the same time, security and privacy concerns have increased, making large-scale encryption an essential piece of the equation.
A few years ago, we started looking at scale challenges in the cloud centered on data growth and the future of storage needs. In a digitized world where artificial intelligence drives business processes, customer engagements, and autonomous infrastructure, we need to create and store more information than ever before.
Enter Project Corsica, which has the potential to be as transformative to our data storage capabilities as any singular technology we’ve deployed to date. Fundamentally, data is at the heart of digital transformation, leveraged by companies to improve customer experiences and differentiate their products and services.
Project Corsica was designed specifically with Azure’s data storage needs and service quality guarantees in mind. Corsica can help to remove data processing bottlenecks by shifting compression and encryption away from the CPU and onto a dedicated hardware accelerator that’s purpose built for these functions.
Corsica’s performance is stellar. It enables our cloud platform to:
- Compress and encrypt data streams at SSD transfer rates, delivering extremely low latency.
- Combine multiple Zipline pipelines to deliver data throughput of 100Gbps, mitigating system bottlenecks.
- Perform encryption in-line with compression, enabling pervasive encryption with zero system impact.
The Zipline compression algorithm used by Corsica yields dramatically better results than other algorithms for cloud data sets, more than 20 percent higher compression ratios versus the commonly used Zlib-L4 64KB model.
At its core, Corsica is a cutting-edge compression technology optimized for a large variety of datasets. Our release of RTL allows hardware vendors to use the reference design to produce chips that harness the algorithm’s compression, cost, and power benefits. Corsica is available to the Open Compute Project (OCP) ecosystem, enabling future contributors to further benefit the entire ecosystem, including Azure and our customers.
Project Zipline partners and ecosystem
As a leader in the cloud storage space, I'm particularly proud that we're able to take all the investment and innovation we've created and share it through OCP so our partners can provide better solutions for their customers. The following industry collaborators have expressed their support for the Project Zipline technology that underpins Corsica.
I look forward to seeing more of the industry joining OCP and collaborating so their customers can also see the benefit.
You can follow these links to learn more about our open source hardware development or Microsoft's Project Zipline contribution on GitHub.