 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/roadmap_95ac4e1c73.png) 
Two months ago, we open-sourced RisingWave, a cloud-native streaming database. RisingWave is developed on the mission to democratize stream processing — to make stream processing simple, affordable, and accessible. You may check out our [recent blog](https://singularity-data.com/blog/risingwave-a-cloud-native-streaming-database/), [document](https://www.risingwave.dev/docs/latest/intro/), and [source code](https://github.com/singularity-data/risingwave) for more information about RisingWave.

Rome was not built in a day, and neither are database systems. We started developing RisingWave (from scratch!) in early 2021; we did a major [code refactor](https://singularity-data.com/blog/building-a-cloud-database-from-scratch-why-we-moved-from-cpp-to-rust) in the summer of 2021; in April 2022, we open-sourced RisingWave under Apache License 2.0 to make it accessible to everyone. So, why and how did we build the system from scratch? Where are we now? And what’s next?

## Building the system from scratch

Many successful database products are forked from existing battle-tested systems. While there already exist several fully-fledged streaming systems, such as Apache Storm, Apache Flink, and Apache Spark Streaming, we made the decision to build RisingWave from scratch. The key reason is that most existing streaming systems are not designed as a database where storage is a primary concern. They are also not built for cloud infrastructures. Thus, building a cloud database from a big-data system can bring heavy technical debt.

However, building RisingWave from scratch doesn’t mean that we build every single component on our own. In fact, we adopt several building blocks from the open-source community, including [etcd](https://etcd.io/), [Apache Calcite](https://calcite.apache.org/) (we recently [deprecated Calcite-based SQL frontend](https://github.com/singularity-data/risingwave/pull/2523)), and [sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs). We also use [Grafana](https://grafana.com/) and [Prometheus](https://prometheus.io/) to implement the observability framework of RisingWave.

## The current status

RisingWave is under rapid development. We have over 70 contributors from across the globe, and we review and merge 100+ Pull Requests weekly. However, RisingWave is still far away from being recognized as a “fully-fledged” system. So, what’s the current status of RisingWave? In this section, we answer this question from several perspectives.

### SQL interface

RisingWave aims to support the complete set of PostgreSQL commands. It has supported the core [SQL commands](https://www.risingwave.dev/docs/latest/sql-commands/) that revolve around these four categories of operations so far:

- **Creating tables and sources.** The fundamental functionality of a DBMS is to store data. RisingWave supports PostgreSQL-compatible relational model, and includes most PostgreSQL basic [data types](https://www.risingwave.dev/docs/latest/sql-data-types/). In addition, RisingWave is able to connect to a large range of external [sources](https://www.risingwave.dev/docs/latest/create-source/) to consume streaming data. Currently, RisingWave supports many popular data formats such as JSON, Protobuf, and Avro on its source. When creating the source, users can manually specify the mapping from the nested structure to a plain table.
- **Creating materialized views.** To deliver continuous analytics for users, RisingWave allows users to define materialized views over both tables and sources. Materialized views can capture the accumulative analytical result, and RisingWave will continuously and incrementally maintain materialized views with a strong snapshot consistency guarantee. Currently, RisingWave supports a large set of PostgreSQL-compatible SQL queries, which covers a large set of [TPC-H queries](https://github.com/singularity-data/risingwave/issues/1380), in materialized views. Different from traditional databases, as a streaming database, RisingWave also (1) allows joins between tables and streaming sources; (2) natively supports streaming window functions, including tumbling windows, hopping windows, and sliding windows; and (3) allows materialized views to be defined on top of existing views, with potential joins on other streams.
- **Querying the database.** RisingWave allows users to issue queries over both tables and materialized views directly, without additional systems as data sinks. Currently, RisingWave supports a large set of PostgreSQL standard user queries, including [20 out of 22 TPC-H queries](https://github.com/singularity-data/risingwave/issues/1380).

### Cloud-native

We have implemented some of our key designs to support a cloud-native streaming database.

- **Disaggregated architecture.** RisingWave can be decoupled into different [components](https://github.com/singularity-data/risingwave/blob/main/docs/architecture-design.md), including SQL frontend service, [compute nodes](https://github.com/singularity-data/risingwave/blob/main/docs/streaming-overview.md), [storage service](https://github.com/singularity-data/risingwave/blob/main/docs/state-store-overview.md), and [meta service](https://github.com/singularity-data/risingwave/blob/main/docs/meta-service.md). Under such design, each component can scale out (or in) independently according to its own usage, enabling fine-grained control over the resource usage, without the cost overhead of the wasted resources.
- **Distributed engine.** At the core of RisingWave is its distributed streaming engine. We have built a high-performance distributed [streaming engine](https://github.com/singularity-data/risingwave/blob/main/docs/streaming-overview.md) completely from scratch, such that we take advantage of recent tech advances, like Rust async runtime. A decoupled storage layer on top of cloud service guarantees the durability and infinite capacity of state storage. We have built a cloud-native storage service [Hummock](https://github.com/singularity-data/risingwave/blob/main/docs/state-store-overview.md), which is based on LSM-tree to optimize its write performance and extensively uses tiered caches to bridge the performance gap of the latency between accessing memory and cloud storage.
- **High availability.** RisingWave maintains its high availability via a periodic checkpointing procedure. Whenever there is a fail detected in the system, RisingWave can automatically recover from the latest consistent checkpoint and resume the stream processing.

### Observability

Observability is a vital part of building a reliable and debuggable system. We spend quite an effort in the observability of RisingWave. In particular, we add tracing to the whole pipeline inside the computing engine. We use [Prometheus](https://prometheus.io/) to collect various metrics of the system status, and we provide a [dashboard](https://github.com/singularity-data/risingwave/tree/main/dashboard) using Grafana to visualize both real-time system metrics and the streaming plan for maintaining materialized views.

### Deployment

We have open-sourced the [Kubernetes operator](https://github.com/singularity-data/risingwave-operator) for RisingWave, and the project is under heavy development. With Kubernetes operator, RisingWave can be deployed as either a cloud-hosted service or an on-premise deployment in self-hosted clusters on top of Kubernetes.

### Development

We also allow users and developers to try RisingWave on their local desktops. We developed [Risedev](https://github.com/singularity-data/risingwave/blob/main/risedev), a developer tool to automatically config all services and manage the dependency. Any developer can deploy a local testing playground of RisingWave within a [simple line of command](https://www.risingwave.dev/docs/latest/get-started/#build-from-source-linux--macos) in the terminal.

### Ecosystem

RisingWave is a part of the **modern real-time data stack**. We spend a huge amount of effort integrating RisingWave with other systems. Currently, RisingWave supports connecting to Apache Kafka, Redpanda, Apache Pulsar, etc. It can also ingest data from Debezium CDC. We plan to collaborate with many other open-source systems, including Redis, ElasticSearch, Apache Hudi, and Apache Iceberg, to build a thriving community together.

## Our roadmap

Building a robust database system can take more than a decade, and we will be continuously focusing on improving the system's performance and robustness in the future. While the design and implementation can evolve over time to catch up with the latest technology trends, we do have a clear roadmap of the key features for the next two years.

### Short-term objectives (within the next 3 months)

Our recent target in the upcoming three months is to deliver a fully functional streaming database. Hence, we will focus on the following objectives:

- **Richer SQL support.** Our primary goal is to further support more SQL standards, for example, more generic nested subqueries, more expressions, and more functions.
- **Creating Index.** Indexes are a major feature to speed up query processing in all databases. Chances are, such indexes can also be shared across the streaming engine to reduce the cost of state maintenance. RisingWave will allow users to issue `create index` commands. These index implementations will be further optimized to enhance both the batch query engine and the streaming engine.
- **Access control.** The next focus will be on access control. We will fully support fine-grained access control by adding basic functionalities including system catalog, user authentication, etc.
- **Connecting with downstream systems.** While RisingWave can serve the users’ query requests over the streaming data directly, it also allows users to export the streaming data into existing data infrastructures in the data pipeline. We are working on adding a create sink command in RisingWave, such that we keep the simplicity of the SQL interface without losing the functionality of traditional stream processing.
- **Highly concurrent serving.** So far, we mainly focus on stream processing. In the mid-term, we will also turn our focus back to the query engine of the database. We have a plan which includes a series of optimizations tailored to high concurrent serving workloads. A major refactor on the architecture of our query engine is on the schedule. In particular, we will introduce the local execution mode to avoid expensive coordination among compute nodes during the query runtime.
- **Benchmark.** People constantly ask about our performance benchmarking results. We haven’t done it so far, because RisingWave is still under rapid development, and the performance numbers will be out of date anytime. Nevertheless, this is definitely in our next plan! We will also carry out extensive tests on the cost and cost-performance ratio, which will drive the technical design of our future optimization.

### Mid-term objectives (within the next 6 months)

Aside from performance enhancement and bug fixing, we will add more functionalities to the system within the next six months.

- **Elasticity.** In contrast to the traditional approach for scaling out (or in), RisingWave stands on the shoulder of the state-of-the-art in both academia and industry. We have designed a novel and lightweight online scaling mechanism to cope with the streaming traffic spike. The idea is that, instead of shutting down the cluster entirely, we take a fine-grained approach inside the database kernel. This feature is under heavy development, and we expect it to be finished in six months.
- **UDF support.** We plan to support user-defined functions (UDF) to enable our users to express a larger spectrum of applications.
- **Schemaless.** One key observation is that users still need to define how source data is mapped from nested format to flat tables upon ingestion. This requires users to have knowledge of the source data in advance. To further simplify the users' usage, we will support schemaless sources in RisingWave. Users can ingest raw JSON data into RisingWave, and RisingWave can automatically infer the source schema, such that users can speculate the data schema and define views after the data arrives.
- **Web UI 2.0.** We will upgrade our web UI to the next generation. This will cover: (1) a comprehensive observability platform monitoring all detailed metrics in a user-friendly interface; (2) an administrative web console for DBAs to easily manage the configuration of both the cluster and database instances.

### Long-term objectives (within the next 18 months)

It is always hard to predict the future. One thing we know for sure is that there are many unknown factors. Nevertheless, we are also clear that our mission is to make stream processing simple, affordable, and accessible, to bring stream processing to the next level. To achieve that goal, we will be continuously exploring the following topics:

- **Multi-tenancy.** We are witnessing the trend that multi-tenant architecture dominates cloud infrastructures, as it significantly reduces the cost for both cloud vendors and end users. The idea is to share resources among multiple users within a single database instance. To enable such sharing, we have to do a series of future work in tenant isolation, security control, schema management, and many other aspects.
- **Machine learning support.** A major application for all data engineering pipelines is to support machine learning. Current machine learning tools are often decoupled with databases. Although such architecture introduces the flexibility of using different machine learning tools, it often comes at a price of cost and latency. There is a huge waste of resource usage and poor data freshness for online machine learning. One of our future objectives is to explore how the modern real-time data stack can be co-designed and seamlessly integrated with existing machine learning tools, such that both the freshness of data and the efficiency of model training and inferencing can be well preserved.
- **Adaptive stream processing.** All database systems support query optimization. The query optimizer collects the statistics of the data and searches for the best query plan when users issue their queries. However, for streaming databases, the story is different. A streaming pipeline is pre-defined when creating materialized views before the data are actually ingested into the system. The technical challenge here is how to automatically adapt the streaming pipeline on-the-fly, and let the streaming engine learn how to fit the fluctuating streaming workloads from the data directly without human intervention.
- **Smart cloud scheduler.** RisingWave is designed to take advantage of diverse and infinite resources on the cloud, or even across multiple clouds! Chances are, we can build a smart cloud scheduler that monitors the status of the streaming tasks, and automatically scales RisingWave to meet users' needs in SLA in the best configuration. Our long-term objective is to use state-of-the-art machine learning models to learn the best and most cost-efficient scaling policy from the workloads instead of some pre-fixed manual policy. Hence we can provide an experience of serverless computation for streaming.

## Summary

RisingWave is a next-generation cloud-native streaming database designed to democratize stream processing. In this blog, we discussed how we plan to approach our objectives. This is not only our internal plan, we make the project’s objectives transparent to hear the feedback from the community.

The next wave of stream processing is coming. To learn more, remember to check the RisingWave [GitHub repository](https://github.com/singularity-data/risingwave/), join our [slack](https://join.slack.com/t/risingwave-community/shared_invite/zt-120rft0mr-d8uGk3d~NZiZAQWPnElOfw) community, follow us on [Twitter](https://twitter.com/SingularityData), and stay tuned with RisingWave!


Two months ago, we open-sourced RisingWave, a cloud-native streaming database. RisingWave is developed on the mission to democratize stream processing — to make stream processing simple, affordable, and accessible.

On the Way to Democratized Stream Processing: RisingWave’s Roadmap

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/Building_a_Cloud_Database_from_Scratch_Why_We_Moved_from_C_to_Rust_4c1dae2429.png) 
_As an early-stage database startup, we entirely deleted our C++ codebase after 7 months of development and rewrote everything from scratch in Rust programming language. Here's how we arrived at a decision and why we think this was one of the best decisions ever._

<br />

[RisingWave](http://www.risingwave.dev/) is a cloud-native streaming database. The idea behind the system is to reduce the complexity and cost of building real-time applications in the cloud.

When we started building RisingWave in early 2021, we wrote it in C++. The founding team consisted of several seasoned C++ engineers with 10+ years of relevant experience. So, using C++ was a no-brain decision. The first few months of development seemed smooth. We were in full gear building the most incredible database of a new era, dreaming of how RisingWave could shake the modern data stack. We were on a quest for greater effectiveness.

But as more and more engineers joined us, some shortcomings of C++ came to bite us: unreadable coding style, memory leak, segmentation fault, and more. We started to question ourselves: is C++ the right language for us to write a new database system? After around seven months of development, a whole month's debate followed; finally, we made the hard decision to move from C++ to Rust.

What did the decision mean? It meant we, a team of 10+ seasoned engineers, had to rewrite the entire system from scratch! And our seven months of efforts went in vain. It was an insane decision for an early-stage startup. Note that time is almost everything in the furiously competitive world of tech startups.

After making the decision, we spent around two months removing our C++ codebase entirely and rewriting it in Rust. We deleted 276,406 lines of code in total. 10 months have passed since then. Thanks to the decision, RisingWave is sailing through, with its [source code](https://github.com/singularity-data/risingwave) accessible for everyone under Apache License 2.0. More than 60 contributors have joined us in developing the cloud-native streaming database. We are proud that RisingWave survived the rewriting; we are pleased that RisingWave gained over 1,600 stars on Github within just a month!


 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/remove_cpp_fa0837d072.png) 

The Rust community keeps growing rapidly, and many engineers may consider whether to (re)write their project in Rust, just like what we did 10 months ago. We would love to share our thoughts on how we made the decision: what made us move to Rust, and what kind of pitfalls we confronted.

Now, let's first look back at what went wrong with C++.

## Implement RisingWave with C++: the good, the bad, and the ugly

C/C++ is undoubtedly one of the most popular programming languages for building database systems. Most well-known database systems, including [MySQL](https://db-engines.com/en/system/MySQL), [PostgreSQL](https://db-engines.com/en/system/PostgreSQL), [Oracle](https://db-engines.com/en/system/Oracle), and [IBM Db2](https://db-engines.com/en/system/IBM+Db2), are created in C/C++. It is still a viable, vital, and relevant language. Choosing C++ won't be a wrong decision to build a brand-new database system, but it doesn't mean that C++ is the best choice, especially for an early-stage startup that aims at innovating a large-scale database system from scratch. To understand the reason, let's review the good, the bad, and the ugly parts of this battle-tested programming language.

### Good

- C++ offers developers the opportunity to develop high-performance programs. It provides fine-grained control over both memory and computation without the overhead of automatic garbage collection. Moreover, C++ code can be compiled into assembly language for direct execution on the OS instead of relying on interpreters or language runtime.
- C++ has proved to be a feasible language for system programming. Plenty of databases are built in C/C++. Therefore decision makers can believe that choosing C++ is never a bad idea.

### Bad

- C++ offers a lot of flexibility to programmers, but it comes at a price. It is extremely easy to program a bug, and some are highly non-trivial. But, it is super hard to debug C++ programs, especially for concurrent programming.
- Dependency management can be a hassle. Although there are some tools, for example, CMake, to automatically configure the compilation of C++ projects, developers still need to manually configure and install the dependent libraries.

### Ugly

- The STL library lacks support for some modern programming tools, for example, native co-routine support. As a result, developers must rely on many community projects, and most lack long-term support.
- Quality assurance is challenging. C++ supports so many features that different developers can code C++ in drastically different styles. As more developers with diverse backgrounds joined our team, we could not maintain readability. Furthermore, bugs in C++ code are non-trivial to identify; hence reviewing C++ code can become daunting.

## Why choose Rust over C++

Since C++ is not a bad choice for building a database system, why did we rewrite the whole codebase again? Was it due to the same reason as the majority: it is cool? The answer is no; we decide to move to Rust after careful consideration.

![poll](/posts/poll.png)

A streaming database is typically used for mission-critical tasks that are incredibly latency-sensitive. Hence, we can only build RisingWave in a language that:

- guarantees zero-cost abstraction so that we won't have performance capped
- doesn't require runtime garbage collection so that we can have the latency spike potentially caused by memory management under our control

We cannot compromise on these two essential requirements for cutting-edge performance.

With these goals in mind, we thus chose Rust over C++. Both languages offer developers zero-cost abstraction and complete control of memory management. Rust, in our view, is a much better choice for relieving the developer's mental load and paving the way for efficient and large-scale collaboration. Here are the top four reasons:

- **Rust is safe.** Rust guarantees memory safety and thread safety at compile time by introducing ownership rules. It goes beyond RAII, a common memory management mechanism used in C++. There are two advantages. The first is self-evident: once the Rust compiler validates our program, we won't have segmentation faults or data races at runtime, which would otherwise necessitate dozens of hours of debugging, especially in a codebase that is highly concurrent and primarily asynchronous. The second is more subtle: the Rust compiler simply restricts the types of faults, which reduces the intricately intertwined code fragments that can cause such buggy behavior. The replication of bugs is substantially improved with the help of deterministic execution (we'll have more on this in future blogs).
- **Rust is easy to use.** C++ is based on the philosophy of giving developers the most degree of freedom. But, sometimes, it backfires. For example, in C++, the template expands at compile time to check whether any operation is uninvokable with a particular type. In Rust, traits constrain the methods that a concrete variety can invoke, so the compiler can check the validity of the type at the call site instead of running expansion. This difference makes C++ template error messages more obscure and often requires seasoned C++ veterans' decipherment. Another example is the widespread abuse of implicit conversion in C++. Implicit conversion may help you code less, but things are more likely to go wrong, and when it really goes wrong, the error would be "implicit" and harder to debug. Check out the [Google C++ style guide](https://google.github.io/styleguide/cppguide.html#Implicit_Conversions); implicit conversion only causes more benefits than confusion when properly restricting its usage, especially in a large codebase.
- **Rust is easy to learn.** For seasoned C++ programmers, Rust is easy to learn. When they first start out, Rust learners usually spend most of their time making sense of ownership and lifetime. Even if they don't explicitly express these concepts in code, experienced C++ engineers always keep these two concepts in mind when programming in C++. Rust can be challenging for beginners. But our interns proved otherwise. They picked up Rust within one or two weeks — even with no prior Rust/C++ expertise — because there are fewer implicit conversion and overload resolution rules to remember. And examining the basic Rust code is a breeze for our colleagues. Now, we spend far less time reviewing beginners' Rust code than C++ code.
- **Unsafe Rust is manageable.** Due to the conservative nature of Rust's static analyzer, we may encounter situations where only unsafe Rust can make the impossible possible. A classic example is creating a self-referential type. Or we must gain extra performance by unsafe, i.e., directly manipulating bits in a compacted memory representation. In either case, it is reasonable for skeptics to ask: will this make the codebase vulnerable? For RisingWave, empirically, no. Two major use cases are LRU cache and Bitmap, which take less than 200 lines out of 170000 lines of code in total. Adopting this approach of first coding in Safe Rust and only resorting to Unsafe when there is concrete evidence and solid arguments is now the secret to our good night's sleep.

## Here comes the dark side

While Rust meets most of our requirements, we are also fully aware of the dark sides:

- **Fragmented async ecosystem:** Without making initial decisions on async runtime, we spent months getting rid of futures-rs and async-std and switched to tokio-rs finally.
- **Cumbersome error handling:** We need to store and implement backtrace capture on errors manually to get a backtrace.
- **Insufficient support of AsyncIterator:** Without native support for stable generators and async fn in traits, we used third-party libraries to achieve the same goal. However, these libraries allocate extra Boxes compared to the pending standard implementation, ultimately lowering performance. Also, using macro from these libraries hinders IDE from working properly, making development less programmer-friendly.
- **Practical limitations of Generic Associated Type (GAT):** GAT is the foundation for many existing/pending features, e.g., static/dynamic async fn in traits. However, complete support for GAT has complex technical issues that may require longer than expected time to be solved. Before that, we have to use various tricks to bypass limitations or live with suboptimal solutions.

Nevertheless, with so many talented engineers in our team, we find that, overall, Rust improves productivity and code quality significantly while keeping the negative impact under control.

## Learning from our experience

This blog is not about convincing every database development team to abandon their entire C++ codebase and rewrite their system in Rust from scratch. Instead, its primary purpose is to tell people why we made such a decision. Rewriting the entire code base is no fun; instead, it's excruciating for a startup where wasting time is suicidal. In fact, despite the apparent benefits brought by Rust, we probably wouldn't have made this tough decision without the following key factors:

- We were at that time refactoring our code base to adapt to our new system architecture, and rewriting (at least a portion of) the codebase became inevitable.
- We have a few Rust enthusiasts (Rustaceans!) in our team, and they kept evangelizing Rust to other engineers and convinced the entire team that rewriting in Rust was a practical option.
- We expanded our engineering team rapidly, and more engineers joined us in the summer of 2021, which significantly accelerated the codebase rewriting.

## Summary

Rust is a cool programming language, and everyone should try it. But don't rewrite your project simply because it is cool to do so. If you are considering whether to rewrite your production-level project in Rust, then please ask yourself the following questions:

- Will low-level programming, performance, memory safety, and package management become a concern for your project?
- Do you have any Rust experts who can help avoid potential pitfalls?
- How long will it take to rewrite this project?
- Will you miss any critical deadlines because of the rewriting?
- Do you have in-house training programs on Rust?

You can decide after careful deliberation of answers to these questions. Again, Rust (or any other language) will never determine the destiny of your project. But making a wise choice may save you hundreds or even thousands of man-months.


As an early-stage database startup, we entirely deleted our C++ codebase after 7 months of development and rewrote everything from scratch in Rust programming language. Here's how we arrived at a decision and why we think this was one of the best decisions ever.

Building a Cloud Database from Scratch: Why We Moved from C++ to Rust


 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/Is_Rising_Wave_the_Next_Apache_Flink_a1ca3c73d9.png) 
## TL;DR: No

Two weeks ago, we open-sourced [RisingWave](https://github.com/singularity-data/risingwave), a cloud-native streaming database. And in no time, the project gained lots of attention from the open-source community and became the #1 trending project written in Rust!

We're thrilled that RisingWave is gaining interest from the open-source community. Because we're on a mission to democratize stream processing — to make stream processing simple, affordable, and accessible! One more exciting detail: we're building this cloud-native streaming database from scratch.

You're right. There are many streaming systems in the market, and during the last decade, we've seen an abundance of them. Then why did we build another streaming system? One common question from the community is: How is RisingWave different from [Apache Flink](https://flink.apache.org/)? And this blog is all about that — a simplified, digestible guide from a developer’s perspective.

![Github Trending: https://github.com/trending/rust?since=weekly (screenshot captured on April 17th, 2022)](/posts/trending.png)

<div style="display: flex; flex-direction: row; justify-content: center; margin-bottom: 30px;"><span style="font-size: 12;">Github Trending: https://github.com/trending/rust?since=weekly (screenshot captured on April 17th, 2022)</span></div>

## What is Apache Flink?

Apache Flink, known initially as Stratosphere, is a distributed stream processing engine initiated by a group of researchers at TU Berlin. Since its initial release in **May 2011**, Flink has gained immense popularity in both academia and industry. And it is currently the most well-known streaming system globally (challenge me if you think I got it wrong!).

As one of the early contributors who participated in the initial design of Flink’s stream processing framework (also known as Stratosphere Streaming), I witnessed the growth and evolution of this project over the last decade.

![email](/posts/email.jpg)

<div style="display: flex; flex-direction: row; justify-content: center; margin-bottom: 30px;"><span style="font-size: 12;">My proposal on Stratosphere Stream (submitted in 2014)</span></div>

Born in the big data era, some key ideas behind the design are:

- Flink positions itself as a **stream computation engine**: it doesn’t provide data persistence capability. Instead, it focuses on providing real-time data processing ability across thousands of machines. To achieve this, Flink follows the design of MapReduce and provides the users with a set of low-level programming APIs (in Java) that can be used to implement parallel and distributed algorithms on a large cluster.

- Flink is natively designed for the **big data stack**: it heavily relies on various big data services (such as Zookeeper, HDFS, and Yarn), and can be perfectly integrated with other big data systems (such as Cassandra and HBase)[\*](#note-1). Just like MapReduce, Flink adopts the **shared-nothing architecture**, where each machine stores and processes its own data and is entirely independent of other machines.

- By **coupling the compute and storage**, Flink can obtain extreme scalability, but its elasticity becomes much harder to achieve and manage. Using Flink to empower streaming applications can be super expensive. The system users may have to continuously provide their resources to peak capacity to avoid latency spikes during workload fluctuation.

Flink stands the test of time as reflected by its popularity in the big data world over the past decade. However, when entering the cloud era, everyone is looking for simple and affordable solutions in the cloud. Flink may no longer be the best answer for stream processing. That's what motivated us to build a new cloud-native streaming system. And RisingWave was born!

## What is RisingWave?

RisingWave is a cloud-native streaming database **built from scratch**. I initiated the project in **early 2021** after signing off from Amazon Redshift. RisingWave is now open-sourced under Apache License 2.0 after over 15 months of development, and we're building the project with the open-source community.

![architecture](/posts/architecture.png)

<div style="display: flex; flex-direction: row; justify-content: center; margin-bottom: 30px;"><span style="font-size: 12;">The internal architecture of RisingWave</span></div>

To simplify stream processing, we revisited an old line of research: **streaming databases**. The idea of streaming databases dates back to an [early proposal](http://cs.brown.edu/research/aurora/vldb02.pdf) of [data stream management systems (DSMSs)](https://en.wikipedia.org/wiki/Data_stream_management_system) to manage data streams the same way as data management in a database. RisingWave, like many other databases, can ingest data, store data, and answer concurrent access requests from end-users. All these requests can be expressed by PostgreSQL-style SQL.

More than just a SQL database system, RisingWave offers the capability of stream processing: it consumes streaming data, performs continuous queries, and maintains results dynamically in the form of a **materialized view**. Processing data streams inside a database is quite different from that inside a stream computation engine: streaming data are instantly ingested into data tables; queries over streaming and historical data are simply modeled as table joins; query results are directly maintained and updated inside the database, without pushing into a downstream system.

RisingWave is built on top of the modern **cloud infrastructure**. A key advantage brought by the cloud infrastructure is that RisingWave can scale compute and storage resources separately and infinitely based on the users’ demands. This offers the chance to constantly attain an optimal cost-performance ratio based on the streaming workload. To fully unleash the power of the cloud infrastructure, RisingWave also adopts a **tiered architecture**: the **compute and storage layers are fully decoupled**, allowing on-demand scaling, and a special caching layer is added in the middle of these two layers, thus reducing the frequency of remote storage access.

## How is RisingWave different from Flink?

Now let’s get back to our question: how is RisingWave different from Flink? Although both RisingWave and Apache Flink provide stream processing capability for real-time applications, they are different in many ways:

### Position

Flink is a **stream computation engine** that enables users to process streaming data. It does not persist user data and is not optimized for serving concurrent queries issued from users.

RisingWave at its heart is a **database system** that can consume data and serve queries from end-users. More than a conventional SQL database, RisingWave excels in offering the capability of processing streaming data at low latency.

### Objective

Flink was designed to maximize **performance** given a fixed amount of resources. Its **compute-storage-coupled** architecture enables it to achieve infinite scaling at a high cost.

RisingWave cares about performance. On top of that, RisingWave is tailored for optimizing the **cost efficiency**. It adopts a **tiered architecture** that fully utilizes the cloud resources to give the users fine-grained control over cost and performance.

### Interface

Flink natively provides a **low-level Java API**. While the low-level API offers experts the ability to fully control the underlying dataflow, it is tough to develop and debug, especially for system amateurs. Flink also provides **SQL interfaces**, but its **big-data dialect** requires users to pay special attention when writing any SQL queries.

RisingWave offers a standard **SQL** interface that is **compatible with PostgreSQL**. Users can process data streams in the same way as they use PostgreSQL.

### Ecosystem

Flink is a big data system deeply integrated with the **big data ecosystem**. Users need to deploy Flink with other open-source big data systems, such as Zookeeper, HDFS, and Yarn.

RisingWave, however, is a cloud-native system that is fully integrated with the **cloud ecosystem**. RisingWave serves as an integral part of the modern data stack: users can connect RisingWave with any of their favorite systems in the cloud, such as Apache Kafka, Apache Pulsar, Redpanda, Amazon Kinesis, and many others. RisingWave’s wire compatibility with PostgreSQL further allows users to access the prosperous **PostgreSQL ecosystem**.

### Operation

Flink is fairly **difficult and expensive** to operate since it is deeply rooted in the big data stack. Operating Flink essentially requires users to use several systems, including Zookeeper, HDFS, and many others. Users may have to manually reconfigure the system to achieve higher efficiency in confronting workload fluctuation.

The cloud-native design of RisingWave **dramatically lowers the bar** for system operations. RisingWave does not depend on any big data services, and its decoupled compute and storage architecture vastly simplifies the management of elasticity.

### Target Users

Flink’s target users are mostly **data engineers and experts**. The operation cost and the learning curve make it difficult for amateur users to fully harness the power of Flink.

RisingWave targets not only engineers but also data scientists, analysts, decision-makers, and **anyone who needs fresh results**. It can be easily deployed and maintained in the cloud, and elasticity can be achieved with little human interference. Its PostgreSQL-style SQL also makes it simple for users to program their streaming applications.

## Conclusion

Flink is a big data engine, whereas RisingWave is a cloud database.

RisingWave is not the next Flink. It is the answer for stream processing in the cloud era.

We believe that a great product comes from the combined wisdom of a vibrant and open community. Check out the source code at [GitHub](https://lnkd.in/eepz5Wgi) to get involved, and join our [Slack community](https://risingwave-community.slack.com/ssb/redirect) for lively discussions!

---

_Note_

<a name="note-1"></a>

\*_This was true historically, but more and more users are moving towards Kubernetes with some distributed file systems or object stores for persistent storage. The Hadoop ecosystem has already been made completely optional in the classpath. The Apache Flink community is intentionally facilitating this, with efforts like https://github.com/apache/flink-kubernetes-operator. Credit to: Márton Balassi (Apache Flink PMC)_


Two weeks ago, we open-sourced RisingWave, a cloud-native streaming database. And in no time, the project gained lots of attention from the open-source community and became the #1 trending project written in Rust!

Is RisingWave the Next Apache Flink?


 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/Rising_Wave_A_Cloud_Native_Streaming_Database_7368330bd8.png) 
Stream processing is a well-studied topic. Over the last few decades, researchers and practitioners have devoted substantial efforts to developing **fast**, **scalable**, and **reliable** streaming systems tailored for real-time analytics over streaming data. Thanks to these efforts, today, open-source and commercial streaming systems are running smoothly in big tech’s data centers, powering thousands of applications covering ads recommendation, fraud detection, IoT analytics, and many others.

Witnessing the significant progress made in the stream processing field, more and more companies have started investigating modern streaming systems, hoping to see how modern technologies can transform their businesses. Unfortunately, many of these companies get stuck in their journey, complaining about the high cost of adopting streaming systems, for at least two sets of reasons:

- **Difficult to learn**. It was never easy to learn how to use streaming systems. Unlike conventional databases (e.g., MySQL and PostgreSQL) that provide SQL as the interactive interface, most, if not all, of the streaming systems require users to learn a set of platform-specific programming interfaces (most likely in Java) to manipulate the streaming data. Mastering streaming systems becomes a mission impossible for non-tech persons. To make things worse, streaming systems represent data in a different way from databases. Users have to write complicated data extraction logic to transit data between streaming systems and databases.

- **Expensive to operate**. Many popular streaming systems are open-source. Comprehensive deployment scripts and docker images are easy to get. But open-source never implies being free or affordable. Streaming workloads can fluctuate abruptly based on the usage demand. Companies have to purchase a large cluster of machines to sustain worst-case scenarios. The cost of deploying and maintaining a streaming system can be much more than just purchasing machines. In fact, assembling a team of engineers that are willing to burn their midnight oil to operate the system can be a real headache.

## Democratizing stream processing

Stream processing should not be the privilege of big techs and deep pockets. Stream processing should not be treated as a monster whose power can only be harnessed by talented engineers. Stream processing should benefit everyone, from data scientists to decision makers, from large enterprises to small businesses. At Singularity Data, we invest all our efforts in **democratizing stream processing**. We are building **RisingWave**, a cloud-native streaming database that makes stream processing **simple**, **affordable**, and **accessible** to everyone.

### Stream processing made simple

RisingWave is a distributed streaming database that provides standard SQL as the interactive interface. It speaks in **PostgreSQL** dialect, and can be seamlessly integrated with the PostgreSQL ecosystem with no code change. RisingWave treats streams as tables and allows users to compose complex queries over streaming data and historical data declaratively and elegantly. With RisingWave, users can now purely focus on their analytical query logics, without worrying about learning Java or system-specific low-level APIs.

### Stream processing made affordable

RisingWave is designed for the cloud. The **cloud-native architecture** enables RisingWave to fully leverage elastic resources provided by the cloud platforms. As a fully managed service, RisingWave deploys, maintains, and scales in the cloud on its own, without human interference from the user side. Once users set their service-level agreement (SLA), RisingWave will automatically assemble different tiers of compute and storage resources in the cloud to achieve the performance goal at a minimal cost. RisingWave is **serverless**: users pay for the service on an as-used basis, and users do not need to pay unless they use the service. We keep optimizing the service to ensure that RisingWave is affordable even for small businesses.

### Stream processing made accessible

We believe that a great product comes from the collective wisdom of a thriving and open community. Instead of developing RisingWave by relying on the experience of a small group of experts, we design and implement it with the open-source community. We decided to open-source RisingWave kernel under **Apache License 2.0**, a permissive free software license. The RisingWave community is open: everyone can participate in the design of the RisingWave project roadmap; everyone can deploy the distributed streaming database in their own cloud service; everyone can contribute and send feedback to the community. The RisingWave community is collaborative: we are eager to build the modern real-time data infrastructure stack together with other communities. For example, we are actively working with the community of **[Redpanda](https://redpanda.com/)**, a real-time streaming platform, to unleash productivity in building mission-critical applications.

The next wave of stream processing is coming. Please join us to help define the future of stream processing, for everyone!


Stream processing is a well-studied topic. Over the last few decades, researchers and practitioners have devoted substantial efforts to developing fast, scalable, and reliable streaming systems tailored for real-time analytics over streaming data.

RisingWave: A Cloud-Native Streaming Database

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/cover_mds_d2a345f835.png) 

Everyone is talking about the modern data stack (MDS) nowadays. I am a data system person. I started building core database systems in the big data era, and have witnessed the birth and prosperity of cloud computing over the last decade. But the first time I came across the term “modern data stack,” I felt confused - is it just yet another buzzword that the cloud service vendors created to attract people’s eyeballs? There are so many articles online, but most of them are quite markety and salesy. After running a startup building core systems in the modern data stack domain for a while, I would like to share my thoughts. In this article, I will explain “modern data stack” to you in simple terms, and discuss why modern data stack can really matter in companies.

## What is a modern data stack?

The most general (and perhaps most unclear) explanation of the modern data stack is that it is a set of tools built around a data warehouse to simplify data integration. The fundamental starting point of a modern data stack is to save data engineers and data analysts time. Here, data integration essentially is to put data from different data sources together, and then analyze and operate. For example, in an e-commerce website, there must be data sources such as user access data, user order data, and commodity information data. By putting these data together in certain ways, we can get some knowledge, such as which kinds of users are interested in which products, which types of products are best to sell, and so on. The modern data stack, in general, is all about helping people turn data into knowledge.

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/fivetran_mds_f4bb971fab.png) 

<center>An illustration of the modern data stack from Fivetran</center>

The image above is from the [sharing](https://youtu.be/Zag5yeTPtPU?t=446) of [Fivetran](https://www.fivetran.com/) CEO [George Fraser](https://twitter.com/frasergeorgew) at Tableau Conference 2019. In the sharing George explains what a modern data stack looks like. The data warehouse takes input from various data sources, transforms it internally inside the warehouse, and provides well-modeled data to support business intelligence, 

After reading the above definition of the modern data stack, you may ask: Why is this architecture "modern"? Does it sound like a buzzword for marketing gimmicks? It is true that the term "modern data stack" sounds so marketing-oriented that people might even dislike it. Before we dive deeper into the modern data stack, let's explore who invented the term first.

## Who invented the term "modern data stack"?

Frankly speaking, it is very difficult to find a very clear answer regarding its roots due to existing ambiguities. However, it is certain that the term was popularized by [Fivetran](https://www.fivetran.com/), [dbt](https://www.getdbt.com/), and the venture capitals behind them.

By searching on the internet, we can find that this word first became popular after 2020. The most famous speech/article is from the second half of 2020 by dbt CEO [Tristan Handy](https://twitter.com/jthandy). In his article, “[The Modern Data Stack: Past, Present, and Future](https://www.getdbt.com/blog/future-of-the-modern-data-stack/)", the development of the modern data stack is divided into three eras: Cambrian Explosion I (2012-2016), Deployment (2016-2020), and Cambrian Explosion II (2020-2025). I suggest readers read the original article for the exact definitions of the three eras. In the same year, another report from [a16z](https://a16z.com/), a well-known venture capital, was published, titled "[Emerging Architectures for Modern Data Infrastructure: 2020](https://future.com/emerging-architectures-for-modern-data-infrastructure-2020/)". This article explains the data infrastructure used by various tech companies in the United States in 2020 from VC’s perspective.

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/google_trend_MDS_0b1b6b4089.png)

<center>The search popularity of “modern data stack” on Google Trends</center>

In fact, after a deeper search, you will find that [George Fraser](https://twitter.com/frasergeorgew), CEO of Fivetran, mentioned this concept in [a speech in 2019](https://www.fivetran.com/blog/competitive-analytics-requires-modern-architecture). On Twitter, Tristan also said that he proposed a similar idea of the “modern BI stack” back in 2016.

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/twitter_yingjun_a3e172def3.png) 

<center>Reply from dbt CEO Tristan Handy on Twitter.</center>

## Why do we need a modern data stack?

Knowing the history of the modern data stack, we can go back to the previous question: why is this architecture "modern"? Why do we need a "modern" data stack? Was the previous data stack not perfect? To answer these questions, I think we should focus on "data", not "modern". It is data that lies at the heart of the modern data stack, not technology. The essence of this terminology is that it changes the way companies use technology: companies no longer build a stack for a certain technology product, but build a stack for their own data.

Let’s think back on how companies managed their data 20 years ago. Twenty years ago, corporate data was basically stored in database systems sold by Oracle, IBM, and Microsoft. Some typical characterizations in these use cases are the following. 

* First, the database systems sold by tech giants such as Oracle, IBM, and Microsoft are expensive, and not all companies can afford those database systems. 
* Second, database vendors provide consulting and support services, not operation and maintenance services. Usually, enterprises need to hire a dedicated DBA team to operate and maintain these databases and build applications on top of these databases. 
* Third, neither the size of data nor the number of applications was large. 

Because of these reasons, when an enterprise tried to manage its own data, they were not really building a framework for the data. Instead, they were building a framework for the database.

Now time is different. In the past 20 years, the scale of data and the number of applications of enterprises have grown exponentially. A number of database products have emerged from the needs of modern enterprises. The development and popularization of cloud computing have further stimulated more enterprises to use databases. On the other hand, the explosion of the market has also led to sustainable developments in the database field. Compared with 20 years ago:

* The price and the usage bar of databases have been largely reduced.

* Database vendors do not only sell software, but services. 

* Databases no longer compete for performance, but for ease of use. 

When facing massive data and applications, if the price of data software is low enough and the performance is good enough, companies can save their time from the dirty work of dealing with the databases, and focus on how to make data management easier. That is exactly the original intention behind the modern data stack: making data management easier.

## How to make data management easier?

As mentioned above, the advocates of the modern data stack include dbt, Fivetran, a16z, other data software companies, and many venture capitals. What they all want to change is the way how companies prepare data for analytics: from the traditional ETL (Extract, Transform, Load) to the ELT (Extract, Load, Transform). Below is an illustration also from George Fraser's [sharing](https://youtu.be/Zag5yeTPtPU?t=488) at Tableau Conference 2019 explaining the difference between ETL and ELT.  Here ETL means that to ingest the data source into the data warehouse, it must go through three steps: data extraction, transformation, and loading. On the contrary, what ELT means is that only data extraction and data loading is needed before loading data into the cloud data warehouses. Data transformation is carried out directly inside the data warehouse. 

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/fivetran_ELT_ELT_7c8954776f.png) 

<center>The difference between ETL and ELT</center>

Turning the traditional ETL into the new type of ELT allows the complex computation of data to be moved from outside the data warehouse to the inside. This movement can simplify the management of the entire data. This still sounds a bit confusing: how does ELT simplify our data management? Can you turn your company's "old tech stack" into a "modern data stack" in one simple step?

Imagine what our data management would be like without ELT. Some employees of a company want to analyze raw data, they use ETL tools to build a pipeline to clean and process the data into table X in the data warehouse, and then analyze table X in the data warehouse. In the process of analysis, they find that they have not extracted a column of data, or the data processing method was wrong, or the unit of the data was wrong. At this time, what should she/he do? The only way is to re-fetch the data from the data source, rebuild the ETL pipeline, and re-analyze the data. However, the data in the data source is often temporary. Usually, the enterprise only saves the data source for seven days or 30 days. That is saying, when she/he wants to re-extract the data, build the pipeline, and perform analysis, the original data may be already lost.

ELT solves this problem in a brute-force way: store all the original data in the data warehouse. When the original data is stored in the data warehouse, there is no so-called data loss problem, and all historical records can be found. When we want to process data, we do not build any pipelines for processing but directly write SQL and use the computing power of the data warehouse itself for processing.

It sounds really simple, and doesn't seem like something that only appeared in the 2020s. Why was ELT not used 20 years ago? Why has ELT been promoted only in recent years? I think there are several reasons for this. 

* Firstly, the amount of data 20 years ago was relatively small, the data format was relatively simple compared to today, and the processing requirements were not high. Even if it was troublesome for everyone to develop applications, they could all be solved by stack engineers. 
* Secondly, database systems from 20 years ago were simply too expensive to buy or not want to spend a lot of money on storing large amounts of data. 
* Thirdly, the performance of the database system at that time was still relatively weak. Today, the cost of engineers is high, the applications are complex and diverse, and it is no longer possible to simply rely on engineers to solve problems; the emergence of cloud computing has greatly reduced the cost of data warehouses, making it possible to store massive amounts of historical data cheaply; The performance has been greatly improved, and it is no longer difficult to process large amounts of data in the data warehouse.

If we think about it carefully, we will find that the essence of ELT is actually to turn the data warehouse into the center of data management, and to use the capabilities of the data warehouse to deal with data problems as much as possible. Now back to the topic of this article, this way of building the stack centered on the data warehouse is the way advocated by the modern data stack: the extensive usage of data warehouse capabilities. ELT is only one part of the modern data stack. In fact, the modern data stack also includes other aspects, including data visualization, metadata information management, data discovery, data sharing, etc. We will not cover these topics in this article.

## What does the modern data stack have to do with cloud computing?

Modern data stacks and cloud computing are strongly correlated but not necessarily causal. In fact, cloud computing has indeed catalyzed the development of the modern data stack. The essence of the modern data stack is to simplify data management, while cloud computing is to reduce the cost of data management; hence the starting point is different. Nevertheless, without a low-cost solution, there would be no way to make data management easier.

## Will there be a one-size-fits-all system on the modern data stack?

The one-size-fits-all system here refers to a system that can do everything: it can support operational processing, analytical processing, stream processing, data visualization, data sharing, data governance, etc. All tasks are done with one system. I think it's an ideal vision, but practically speaking, we haven't reached the stage where one size can fit all. The key question here is not whether it is technically feasible, but whether the product can be widely accepted.

I think this question can be approached from both a product competition and user needs perspective. 

* From the product competition, modern data software competition is more than “hard” level competitions, e.g., performance and price. Vendors have expanded the competition to more points in the “soft” level, for example, security and usability. It is often difficult for a one-size-fits-all system to compete with a product specializing in a certain task. A good example is Zoom. Although many office software, including Slack, Microsoft Teams, etc., can do video conferencing, Zoom has become a worthy winner in this field through its professionalism and ease of use. 
* In terms of user needs, users rarely need to use all the features in a one-size-fits-all system. Small and medium-sized enterprises tend to use only two or three functions, and naturally, they may prefer the system that does the best job of these two or three functions. In a large company, users do use a variety of functions, but when trying to convince such users to replace their existing systems, they are often deterred by the high cost of migration.

Of course, various viewpoints justify the existence of a one-size-fits-all system. The most common argument is that enterprise-level users often only want to solve problems but do not want to spend time choosing solutions. The existence of a one-size-fits-all system can significantly save their time in selecting products. Furthermore, such a system can provide a unified user experience that is more suitable for enterprise users.

I don't think there is a standard answer to this question, and I believe that the future trend (at least in the next 3-5 years) is coexistence. Nevertheless, the one-size-fits-all system will still be a minority. Whether it is a Swiss Army knife, a fruit knife, or a utility knife, they all have their own living space, so that the Swiss Army Knife will not occupy the market of other kinds of knives.

## Summary

The modern data stack serves users' data, and it is completely changing the situation in which users are bound with a few tech giants. The goal of modern data stacks is to greatly simplify the difficulty of managing data for users, allowing users to care more about the data itself, rather than the software. Of course, the modern data stack is still quickly evolving. In the next article, I will explain the current landscape of modern data stack, and what's next in the near future.




Everyone is talking about the modern data stack (MDS) nowadays. I am a data system person. I started building core database systems in the big data era, and have witnessed the birth and prosperity of cloud computing over the last decade. But the first time I came across the term “modern data stack,” I felt confused - is it just yet another buzzword that the cloud service vendors created to attract people’s eyeballs? There are so many articles online, but most of them are quite markety and salesy. After running a startup building core systems in the modern data stack domain for a while, I would like to share my thoughts. In this article, I will explain “modern data stack” to you in simple terms, and discuss why modern data stack can really matter in companies.

Everything You Wanted to Know but were Afraid to Ask about Modern Data Stack (part I)

What’s next for real-time data systems? Let’s take a look at what startups are working on!

Future of Real-time Data Systems: A Recap of Current’22

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/Blog_2_6_4e7cff03e9.jpg) 

RisingWave is a cloud-native SQL streaming database that started in early 2021. Since its official open-source release on [GitHub](https://github.com/risingwavelabs/risingwave/blob/main/docs/checkpoint.md) under the Apache 2.0 license on April 8, 2022, RisingWave has grown into a popular project with over 100 contributors and 3,700 stars.  

We are greatly indebted to our community members. A product can become great with the collective wisdom of a thriving and open community more quickly than with the experience of a small group of experts. The community can shape the future of RisingWave by designing the project roadmap, deploying the distributed streaming database in their cloud service, and much more. We look forward to increased community engagement in 2023.  

2022 has been a great year for our company, and we shared the updates in our previous [blog](https://www.risingwave-labs.com/blog/a-look-back-at-2022/). Now, let's take a step back and reflect on our open-source community. What did we achieve? And what more is in the store?

## RisingWave Labs Objectives
  

We launched RisingWave — an SQL streaming database — with the sole mission of democratizing stream processing by making it simple, affordable, and accessible. Since the publication of the research paper "[Monitoring Streams – A New Class of Data Management Applications](https://cs.brown.edu/research/aurora/vldb02.pdf)" at the VLDB 2002, stream processing has been explored by both academia and industry for two decades. Industrial-grade projects such as Apache Storm, Apache Flink, Apache Samza, and Apache Spark Streaming have quickly emerged, further validating the significant value that stream processing can bring to modern companies. However, existing stream processing systems still have several main drawbacks, including high learning curves, development and operation cost, and performance cost. RisingWave was designed to address these issues with the following key features:  

- full wire-compatibility with PostgreSQL enabling users to perform complex analysis over data streams using simple PostgreSQL-style SQL  

- independence from outdated JVM ecosystem components (such as Zookeeper), and native Kubernetes support to simplify system maintenance  
 
- compute-storage separation architecture to greatly reduce performance costs and achieve infinite elastic scaling in the cloud  

As a database system, RisingWave supports data persistence and allows users to perform high-concurrency queries on stored data, enabling users to use one single RisingWave instance to replace the traditional "stream processing framework + database" combination (such as Flink + Cassandra) for efficient support of real-time applications.  

RisingWave's goal is not to replace existing computation frameworks like Apache Flink but to make stream processing more accessible and efficient for a broader audience. It is similar to how Snowflake (a SQL database) and Apache Spark (a big data computation framework) have mutually benefited each other in the batch processing domain. 

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/Blank_diagram_Page_1_2b00776060.png) 

## RisingWave Labs Achievements in 2022
  

### 1. Enhancing stability and performance testing
  

Stability and performance are the foundation of any hard-core system. No matter how powerful the system is, instability and low performance make it difficult for the system to gain wide adoption from user groups. RisingWave Labs has considered stability and performance testing top priorities since day one. In the past year, RisingWave Labs has continuously collected test cases from various products and added them to its integration tests. At the same time, RisingWave Labs also added a deterministic testing framework to make it more convenient to locate and eliminate bugs. RisingWave Labs completed the first version of internal performance testing. Using performance test benchmarks such as Nexmark and TPC-H, we concluded that RisingWave can achieve 50% or more performance improvement over similar products in both stateless and stateful computing.  

### 2. Implementing commonly used functionalities of PostgreSQL
  

RisingWave speaks PostgreSQL to significantly lower the bar for users to adopt stream processing technology. This means that as long as users can use PostgreSQL, they can use RisingWave for stream processing. PostgreSQL is a mature database system that has been battle-tested for more than 30 years. Its supported functions are extremely complex. It would take several years to fully implement the functions of PostgreSQL. However, RisingWave has already implemented the most commonly used PostgreSQL operators and expressions. It supports complex queries well. RisingWave Labs is constantly adjusting the priority of function implementation through community dynamics, hoping to build a more complete system based on real-world use cases.  

### 3. Integrating mainstream systems 
 

RisingWave is fully wire-compatible with PostgreSQL. In other words, RisingWave can be integrated with any system that supports PostgreSQL. But due to the complexity of system implementation, RisingWave still has to conduct a lot of testing on system integration to ensure security and reliability. As a stream processing system, RisingWave also supports data ingestion and delivery. Users can easily import and export data to and from RisingWave through the `create source` and `create sink` syntax.  

Currently, the upstream and downstream systems supported by RisingWave include Kafka, Redpanda, Pulsar, Kinesis, SQL CDC, PostgreSQL CDC, AWS S3, Iceberg, Hudi, and more. The supported BI tools include Grafana, Metabase, Superset, and more. Users can visit [risingwave.dev](http://risingwave.dev/) to learn more about system integration and provide valuable feedback to us.  

### 4. Supporting high concurrency access to persistent data
  

RisingWave is a database that supports stream processing. This means that RisingWave users can not only use SQL to perform stream processing in the same way as operating a database but also store data directly in RisingWave. RisingWave supports persistent data storage and allows users to perform high-concurrency access to the stored data. This functionality enables RisingWave to directly replace the combination of Flink+Cassandra/Redis/DynamoDB and other systems. Therefore, users can use a single RisingWave instance to support complex stream processing applications, significantly reducing costs and increasing efficiency.  

### 5. Open sourcing the RisingWave Operator
  

As a cloud-native database, RisingWave made support for Kubernetes deployment a high priority. Shortly after open sourcing the RisingWave project in April 2022, we open sourced the RisingWave Operator project on [GitHub](https://github.com/risingwavelabs/risingwave-operator). This allows users to deploy RisingWave through Kubernetes quickly and efficiently.  

## RisingWave Labs Directions in 2023
  

### 1. Further enhancing system stability
  

Improved stability has always been one of the core development directions of RisingWave. In 2023, RisingWave Labs will aggressively enhance stability assurance and actively work with community users to identify potential stability issues.  

### 2. Releasing transparent performance reports
  

Performance is one of the main goals of many projects. However, RisingWave Labs does not want to pursue performance just for the sake of performance, nor does it want to turn "benchmarking" into "benchmarketing." The RisingWave Labs developers fully understand that the factors that affect performance are diverse and challenging to enumerate. In order to better inform community users about RisingWave's performance, RisingWave Labs will release an open performance test report in early 2023, allowing anyone to quickly understand the performance strengths and weaknesses of RisingWave compared to other stream systems and easily reproduce the results.  

### 3. Improving the performance of join, window, and other operators
  

In stream processing, join, window, and other stateful operators are complex to implement. Many users have made higher performance demands on RisingWave's join, window, and other functions. In 2023, RisingWave Labs will improve these functions, enrich the types of join and window operators, and conduct stress tests on these operators to ensure the stability and performance of RisingWave in complex scenarios.  

### 4. Enhancing streaming-specific semantics
  

Unlike traditional batch processing systems, stream processing systems have some stream-specific semantics, such as watermark, exactly-once, and others. As a stream processing system, RisingWave has already supported exactly-once processing. At the same time, RisingWave has partially supported watermark semantics. In 2023, RisingWave Labs will fully support watermark and other commonly used stream processing functionalities and provide documentation and tutorials to help users better understand and adopt stream processing technology.  

### 5. Providing a more user-friendly interface
  

To operate complex data systems, a user-friendly UI is essential. In 2023, RisingWave Labs will launch a new version of the RisingWave UI, allowing users to use RisingWave more easily.  

## Conclusion
  

The rapid development of the RisingWave project is only possible with our community's consistent support and encouragement. By the end of 2022, RisingWave had been deployed in financial, internet, and entertainment scenarios. We expect to release the use reports of RisingWave in the second quarter of 2023. We will also host in-person events in multiple regions, including the US, Singapore, Europe, and others. In addition, we welcome individual and corporate users to join the RisingWave community and provide valuable feedback to RisingWave. Thank you all, and happy new year!  

 

Since its official open-source release on GitHub under the Apache 2.0 license on April 8, 2022, RisingWave has grown into a popular project with over 100 contributors and 3,700 stars. Let's take a step back and reflect on our open-source community.

RisingWave Community Retrospective and a Look Forward 

In this blog, our CEO and Founder, Yingjun Wu, will combine his software development and customer engagement experiences to discuss the past, present, and future of stream processing and streaming databases from a practitioner’s perspective.

Rethinking Stream Processing and Streaming Databases

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/Is_Rising_Wave_streaming_database_a_good_choice_for_my_stream_processing_application_8db4012f31.jpg) 

If you are looking for a stream processing system for your new streaming analytics or streaming ETL applications, consider RisingWave. As a distributed SQL streaming database, RisingWave is well-optimized for stream processing.

## Is it meant for your stream processing application?

The underlying premise of this question is whether RisingWave is well-suited for your workload and business requirements. In this article, we will guide you through the different aspects of RisingWave. We cover user interface, performance, correctness, scalability and elasticity, fault tolerance and availability, durability, serving capability, integrations, and deployment.
 
### RisingWave is a streaming database

RisingWave is an open-source distributed SQL database designed for stream processing. Also known as a streaming database, RisingWave processes streaming data in a database way. It uses SQL as the interface to ingest, manage, query, and store continuously generated data streams.

RisingWave is designed to reduce the complexity and cost of building real-time applications. It consumes streaming data, performs incremental computations when new data come in, and updates results dynamically.

As a streaming database system, RisingWave maintains results in its own storage so that users can access data efficiently. You can also sink data from RisingWave to an external stream for storage or additional processing.

The open-source version of RisingWave is released on [GitHub ](https://github.com/risingwavelabs/risingwave) under Apache License 2.0, a *real* open-source license. It is ready for production use and has been deployed in dozens of companies. The managed version, named RisingWave Cloud, is currently in beta release. Early access is [available](https://www.risingwave.cloud/get-early-access/).

### The decoupled compute-storage architecture provides better leverage


RisingWave is a distributed system. It leverages the decoupled compute-storage architecture to achieve infinite scalability at a low cost.


There are currently four types of nodes in the cluster:

- Frontend: Frontend is a stateless proxy that accepts user queries through Postgres protocol. It is responsible for parsing, validating, optimizing, and providing each query's results.
- ComputeNode: ComputeNode is responsible for executing the optimized query plan.
- Compactor: Compactor is a stateless worker node responsible for executing the compaction tasks for the storage engine.
- MetaServer: MetaServer is the central metadata management service. It also acts as a failure detector that periodically sends heartbeats to frontends and ComputeNodes in the cluster.

 ![](https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/Untitled_21_74455e704e.png) 

<center>The overall architecture of RisingWave</center>


The topmost component is the Postgres client. It issues queries through [TCP-based Postgres wire protocol](https://www.postgresql.org/docs/current/protocol.html). The leftmost component is the streaming data source.

[Kafka](https://kafka.apache.org/intro) is the most representative system for streaming sources.
Alternatively, [Redpanda](https://docs.redpanda.com/docs/get-started/quick-start/), [Apache Pulsar](https://pulsar.apache.org/docs/2.11.x/), [AWS Kinesis](https://aws.amazon.com/kinesis/getting-started/?nc=sn&loc=3), [Google Pub/Sub](https://cloud.google.com/pubsub/docs/overview) are also widely used. Streams from Kafka will be consumed and processed through the pipeline in the database. The bottom-most component is AWS S3, or MinIO (an open-sourced s3-compatible system).

### User interface

RisingWave is a SQL streaming database. Users can compose Postgres-style SQL to consume, process, and manage streaming data. Unlike big-data-style stream processing systems (such as Flink and Spark Streaming), RisingWave does not provide native Java, Scala, or Python API. However, since RisingWave is wire- compatible with PostgreSQL, users can directly use third-party PostgreSQL drivers to interact with RisingWave using other programming languages such as Java, Python, and Node.js.

To extend RisingWave's expressiveness in supporting complex queries, we are developing our UDF framework, and Python will be the first supported language. RisingWave will host the UDFs on a dedicated process and call them remotely. We are also exploring deploying UDF with serverless computing services such as AWS Lambda, which offers excellent scalability and low cost.

### Performance

RisingWave is highly optimized for high-throughput stream processing. Our performance evaluation using the Nexmark benchmark shows RisingWave can achieve 10-50% higher throughput than Apache Flink for most queries. A detailed performance report will be released in Q2 2023.

RisingWave is *not* an in-memory system. It uses tiered storage architecture to manage its computation states and persistent data: the newly ingested and frequently accessed data are maintained in memory, and the infrequently accessed data are stored in S3 or S3-equivalent remote storage.

RisingWave is written in Rust, a low-level programming language designed for high- performance programming. Compared to JVM-based systems, RisingWave does not suffer any performance penalty caused by runtime management or garbage collection.

### Correctness

RisingWave is a stream processing system. It guarantees “[consistency and completeness](https://www.confluent.io/blog/rethinking-distributed-stream-processing-in-kafka/).” Specifically:

1.	RisingWave ensures exactly-once semantics, meaning that every single data event will be processed once and only once, even if a system failure occurs.
2.	RisingWave supports out-of-order processing. Users can enforce RisingWave to process data events in a predefined order, even if data events arrive out of order.

RisingWave is also a database system. But it is not an OLTP database (≤ version 0.1.16) and cannot be used to process database transactions. RisingWave guarantees that all reads will see a consistent snapshot of the database.


### Scalability and elasticity

RisingWave was born as a distributed system. All data is stored in an underlying object store, while the compute layer can be swiftly scaled in and out whenever workload fluctuation occurs. The scaling process can be completed almost instantaneously without interrupting data processing.

Internally, RisingWave dynamically partitions data into each compute node using a consistent hashing algorithm. These compute nodes collaborate by computing their unique portion of the data and then exchanging output with each other. Thanks to partitioning, RisingWave can achieve cache locality on each node and get an overall performance comparable to shared-nothing systems.

### Fault tolerance and availability


Like many other stream processing systems, RisingWave implements the Chandy- Lamport algorithm to take a consistent snapshot. The consistent snapshot helps RisingWave efficiently recover from failure.

The recovery process is faster than similar systems due to its specialized shared- storage design. The system can continue from the last checkpoint almost immediately after recovery. The only thing affected is the cache, which is rebuilt gradually afterward.

Currently, the default checkpoint frequency is 10 seconds. Hence, the recovery process will bring the system back to where it was 10 seconds ago and continue from there, aka RPO (Recovery Point Objective). In addition, it may take about 10 seconds to detect the failure through heartbeat timeout. Together they make up the Recovery Time Objective or RTO.


### Durability


RisingWave persists its stored data. Users can insert data into RisingWave by issuing DML statements like `INSERT` or directly using connectors.

We recommend using connectors from message brokers or direct CDC from upstream systems for production use. Incoming data is stored in RisingWave’s storage along with the checkpoint. To use a connector, users only need to add some extra properties when creating the table. Connector offsets are persisted in checkpoints to achieve data durability and exactly-once delivery.

In the case of a system failure, RisingWave will continue to consume data from the checkpointed offset, ensuring that no data is missing or lost. However, the users are responsible for ensuring that data stored in message brokers is persisted.

DML statements, including` INSERT/ UPDATE/ DELETE` statements, are beneficial for manually modifying data. However, they may suffer from data loss if the system inadvertently fails before they are persisted. The `FLUSH` command can explicitly tell them to persist data. In future releases, we plan to provide a write-ahead log (WAL) to ensure zero data loss — better data management.


### Serving capability

RisingWave uses materialized views to store stream processing results. It can serve concurrent queries from users at ultra-low latency. Our experimental evaluation using a standard benchmark is a good example. It showed that RisingWave can answer concurrent queries with single-digit millisecond latency. We will release a public experiment report later this year.

### Integrations


Check our comprehensive [list](https://www.risingwave.dev/docs/current/rw-integration-summary/) of integrations. Here we highlight some of them.

#### Source


- Message brokers: Apache Kafka, Redpanda, Apache Pulsar, AWS Kinesis, Google Pub/Sub
- Database (direct CDC connection): MySQL, PostgreSQL
- File: S3

#### Sink

- Message brokers: Apache Kafka, Redpanda
- Databases: MySQL, PostgreSQL, Redis, Cassandra, DynamoDB
- Data lake: Apache Hudi, Apache Iceberg

#### 3rd-Party Tools


- Realtime dashboards: Grafana
- BI tools: Superset, Metabase, DBeaver


### Deployment


The easiest way to deploy RisingWave is through our official cloud hosting platform, [RisingWave Cloud](https://www.risingwave.cloud/get-early-access/). We now support AWS in most regions, with support for GCP coming soon.
Alternatively, we provide an open-source Kubernetes operator to run RisingWave on your local Kubernetes cluster. For storage, any S3-compatible storage service is accepted, including MinIO or Ceph.

We suggest not deploying RisingWave using only binaries, as it may incur additional efforts to achieve high availability.

### Others: OLTP and OLAP


RisingWave does not support transaction semantics. So if you are looking for a system to support transactional workloads, there may be better choices than RisingWave. Consider cloud databases like [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html) or [Google Spanner](https://cloud.google.com/spanner).
RisingWave does not implement a columnar store and is not optimized for interactive analytical workloads. But RisingWave can easily be integrated with popular OLTP and OLAP databases.

### Conclusion

RisingWave is a modern distributed SQL database for stream processing. It greatly saves the efforts of building streaming analytics or streaming ETL applications.
 
RisingWave can achieve higher cost efficiency than other stream processing systems due to its decoupled compute-storage architecture. RisingWave has already been deployed in many pioneering companies. For more information about RisingWave, check [here](https://www.risingwave-labs.com/product/).



As a distributed SQL streaming database, RisingWave is well-optimized for stream processing. Let's take a look at whether it is meant for your stream processing application.

Is RisingWave the Right Streaming Database for Your Stream Processing Application?

Yingjun Wu

CEO @ RisingWave Labs

Rayees Pasha

Head of Product @ RisingWave Labs

Jovana

Community Manager @ RisingWave Labs

We are excited to announce the launch of Streaming Stories. This platform brings together stories from users around the world; all centered around their experiences with stream processing.

RisingWave Labs Launching a Webinar Series: Streaming Stories

Jon Chuang

Software Engineer

Zhidong Guo

Storage Intern

Siyuan Wang

In this blog, we'll look at three techniques in RisingWave that leverage the power of S3-compatible object stores to achieve better performance.

Object Store Optimizations in RisingWave 

Sooter Saalu

Community Contributor

Materialized views are database objects that contain the results of your operations, calculations, or queries. They exist as a locally saved representation of your data that you can use and reuse without recalculating or computing data values. Using materialized views saves processing time and cost and allows you to front-load your frequently used or most time-consuming queries.  This article explores materialized views in modern-day databases, their use cases, why they are important, how they are connected with data analytics, and their limitations.  

How to Update Materialized Views in Real Time

Somrita Ganchoudhuri

Marketing Communications Manager @ RisingWave Labs

Let's take a look back at 2022! In this blog, we focus on five key updates from 2022.

A Look Back at 2022: 5 Key Highlights

Chi Zhang

Community Contributor @ RisingWave Labs

In database systems, we can speed up join computations by creating indexes. Here we examine RisingWave's data persistence process on its cloud-native storage, the implementation of shared state, and delta join.

Shared Indexes and Joins in Streaming Databases 

Tushar Malik

RisingWave Community Contributor

In a nutshell, RisingWave is a distributed, cloud-native streaming database. It is built from the ground up to ingest and process data from streaming messaging queues in a horizontally scalable way. I discovered the RisingWave Code Camp, a mentorship program for junior developers to advance their skills in software and database systems, through a friend who'd met the team at Kubecon EU. Unfamiliar with Rust and admittedly daunted at the prospect of working on a database, I was hesitant to apply but hey, I needed an excuse to learn Rust, and I was intrigued by the idea of working with the amazing people building RisingWave! 

Building a Source Connector: A RisingWave Code Camp Experience 

Tianyi Zhuang

Software Engineer @ RisingWave Labs

Seven months ago, we open-sourced RisingWave, a cloud-native streaming database written in Rust. In this blog, I will introduce how RisingWave manages the streaming state in our Hummock cloud storage engine.

State Management for Cloud Native Streaming: Getting to the Core

The Cloud Native Computing Foundation brings Kubecon, its flagship conference, to promote education and advancement of cloud native computing. Kubernetes, Envoy, and Prometheus are just a few of the rapidly expanding cloud-native projects that the Cloud Native Computing Foundation supports, oversees, and directs as a member of the Linux Foundation. This time it's happening from October 24 – 28, 2022, in Detroit, Michigan.

Kubecon + CloudNativeCon: Top 3 reasons for attending it

We’re excited to announce that we have secured $36M in Series A funding to meet surging demand for streaming databases that enable users to make real-time decisions. This brings the total amount raised to more than $40 million. We plan to use this funding to accelerate our mission of democratizing stream processing technology that until now has been available to only a select few organizations.

RisingWave Labs Raises $36M in Series A Funding for its Stream Processing Platform 

Recently, RisingWave Labs' team attended Current 2022, where over 2,000 people from around the globe gathered to see what is new and exciting in the world of data streaming technologies. There was a wide range of sessions and events available — attendees could get a taste of what is happening in the world of streaming.

Current 2022: 4 themes you should not miss

Hey, we are at Booth #ST9. Welcome to stop by and say hi! 

Current 2022: The Next Generation of Kafka Summit

We believe focusing on messaging is just as important as the focus on product differentiation. Keeping this in mind, we are changing our company name to RisingWave Labs. After all, RisingWave is at the heart of our innovation. And this change will allow us to stay close to our product and the RisingWave community.

We are RisingWave Labs now!

Shunjie Ding

RisingWave is a cloud-native streaming database. It is designed to fully leverage the elasticity provided by the cloud. However, deploying and maintaining RisingWave on the cloud is still a challenging technical problem. To tackle the problem, we developed the RisingWave operator, an extension that features the operations of managing RisingWave on Kubernetes. It supports out-of-the-box restarting, scaling, and upgrading operations on a RisingWave cluster. The RisingWave operator is now open-sourced and it is one step further toward the cloud era for RisingWave.

One Step Towards Cloud: The RisingWave Operator

Emily Le

Technical Writer @ RisingWave Labs

As companies collect increasing amounts of data daily, it is crucial to query and analyze the data to make timely decisions quickly. To support such business scenarios, OLAP and stream processing are two popular techniques that promise to deliver analytical results with low latency. However, since both techniques claim to target real-time analysis, it can be quite confusing when trying to understand the differences between them and when to use them. What is OLAP? What is stream processing? How are they different from each other? This blog answers these questions by comprehensively comparing OLAP and stream processing.

Real-time analytics: how is OLAP different from stream processing?

Heng Ma

Tao Wu

Product Manager @ RisingWave Labs

Social media platforms like Twitter process a high volume of messages every second. To gain insights from social media platform data in real-time, companies need to adopt stream processing in their tech stack. However, this can be a struggle as many stream processing frameworks are complicated to design and implement. This blog presents how RisingWave can seamlessly work with Apache Pulsar to provide a real-time data streaming solution, making real-time applications much easier to build and maintain.

Tutorial: Pulsar + RisingWave for Fast Twitter Event Processing

Open-source mascots have been widely considered synonyms for their projects/products. So, it should come as no surprise that we thought RisingWave too, deserves a mascot that will add an element of fun to the often-intimidating world of stream processing. Meet Wavy, a squid that won our hearts and will help us keep RisingWave a welcoming and friendly community.

Hello World, Meet Wavy

Building the development environment manually for RisingWave is painful. We have developed RiseDev, the developer tool, to drastically simplify the development process. It automates the process of starting and deploying a full RisingWave cluster and makes the development process much smoother and more efficient.

Developer Experience Counts: Behind the Scenes of RiseDev

The amount of streaming data has grown explosively over the past few years. A lot of businesses realize that they need to move to stream processing, but they have a hard time figuring out the route to take. 