Cloudflare has unveiled a new serverless event streaming product named K2, and the company acknowledges that its own infrastructure proved too unusual for standard off-the-shelf software such as Apache Kafka.
Micah Wylde and Marc Selwan’s announcement states that Cloudflare required a lasting buffer at the edge for its Basin Pipelines project. Pipelines functions as a stream processing engine built on a pull-based design, which means another system must hold events until they get read, altered, and then placed into R2. The company promises it will never discard events once they’re accepted into the Pipelines Stream, that storage has to be durable — it can’t, even across extended stretches of time.
Most firms turn to Apache Kafka when they need that sort of setup. Cloudflare, though, sits at the edge, spread across a vast number of machines across over 335 cities. Because of its design, it frequently cannot use conventional distributed systems software like Kafka, and instead must redesign how these systems get put together and kept running.
Building a Log on R2
Instead of building a new distributed database from scratch, Cloudflare chose to depend on the state primitive it already possessed: R2, its object storage system. This kind of storage combines extremely durable storage (11 9s!) with strongly consistent APIs. By moving replication and consensus down to the storage layer, the application layer — K2 in this case — is made radically simpler, cheaper, and higher performance. It also splits apart compute and storage, allowing each to be scaled independently.
R2 lacks support for appends, which would normally be used on a log. K2 handles this by collecting writes in-memory on an edge service, waiting a brief interval for data to show up, and then writing all events as a segment file at once. The ordering and strictly rising offsets come from R2’s atomic operations, removing the need for a separate coordination service.
Queues, Streams, and Pipelines
Cloudflare currently provides Queues and Basin Pipelines. Queues focus on keeping track of individual items of costly or time-consuming work that require asynchronous completion. They handle complex logic on each work item, with retries, delays, and dead-letter queues for failed attempts included among the features.
K2’s design focuses on high-scale data movement, long-term retention, and fan-out consumption. It handles messages as batches when they are produced and consumed, a feature that allows for efficient processing but sacrifices message-level retries in the process. This batching approach is what causes higher producer latency compared to queues.
When the final output lands in object storage or Iceberg tables, the company points to Pipelines. For custom processing or any other destination, the recommendation shifts to K2.
What the Latency Costs
The main drawback of R2 is increased produce latency. Because writing to object storage takes longer than using a local disk, K2 must hold off until the local batch builds up before it can begin the write. At launch, this produces roughly 1 second of produce latency at the 99th percentile of response times.
An upcoming technical deep dive from Cloudflare will reveal further specifics about how K2 is designed.
Getting Started
The public beta for K2 is now live, and you can set up your first stream in seconds using the guide. There are several ways to build streams: cf, Wrangler, the dashboard, or other tools.
| Primitive | Purpose |
|---|---|
| Queues | Track individual items of expensive or time-consuming work |
| K2 Streams | High-scale data movement, long-term retention, fan-out consumption |
| Basin Pipelines | Serverless ingestion for transforming and writing events to R2 or Iceberg tables |
K2 was born from Cloudflare’s own architecture, which rendered conventional distributed systems impractical to build. The announcement describes the outcome: a durable event streaming primitive that relies on R2’s durable storage.
Source material: “Cloudflare K2: serverless event streams,” cloudflare.com.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.