OLAP and OLTP

In our data system, we have two approaches to seeing data and data flow. Sometimes we get data like a box: for example, each day we get an update of 10K new users, or a single user creates an account and it is stored in our database. On the other hand, we have something like water in a pipe! We have a flow of data, like a chain where each piece comes after another, and we call it stream data. So now we know how we can get and receive data. This is the first part: how we get data.

We also have some concepts and words that describe certain ideas, like transaction. A transaction is something that has been done. For example, you buy a car and you have a bill in your hand, or you receive an email saying your purchase has been completed. That is a transaction, and it shows an action that has been done. Our dear engineers came and created special databases for these requirements, and we have relational databases like Postgres, MySQL, and others. It is important that inserts and updates are fast, and that all CRUD operations are fast and reliable. This is OLTP.

But sometimes we have another concept, which is about business: we want to read data, not perform actions on it. Here, to read fast and move data between sections, our dear engineers created another technology, OLAP databases like ClickHouse, which are bad at delete and update but very fast at reading data. Let's assume we have a data warehouse with a lot of data, and now it is time to use it and build some reports. So we go with OLAP and collect some data, and it helps us produce report-focused data (because there are no transactions on the data, only reads).

The important thing is that most of the time, the data (the "food") of an OLAP system comes from an OLTP system. OLTP stores the data, and OLAP starts moving it to another section. We have a system where all units are trying to talk to each other.

We have many topics that we can learn and read about, but it is not the right time to go through them now :)