2024-07-12
한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina
The company wants to use a new big data architecture and plans to use domestic big data platforms instead of foreign ones. So here I am struggling whether to use Doris or StarRocks. If I use Doris, because it is open source, I can directly use cloud vendors in the future. If I use StarRocks, I have to build it myself, but it will definitely be commercialized in the future and will also charge money. I have used Doris before, but not StarRocks. I want to try whether it is really as high-performance as the reference link below, so I chose StarRocks. Use it as a data warehouse to replace Hive-Presto or Kudu-Impala in our previous CDH.
Do you think starRocks can replace hive? I think it can. Hadoop was used by Google 20 years ago and it is not popular anymore. Hadoop has too many components. The advantage is that it is very stable. Almost all the major bugs have been fixed in 20 years. You can use starRocks instead of Hadoop.
cat /proc/cpuinfo | grep avx2
If nothing is printed, then you can replace the cpu.
Be is responsible for calculation. If you don't have this instruction set, you can't deploy it.
Startrocks is a data warehouse for data analysis, which can replace the traditional hive.Vectorization，MMP ArchitectureofColumn storage engine,supportReal-time analytics, concurrent computing. Compatible with MySQL protocol, can be usedmysql client connection.supportHorizontal ScalingThe entire system has no external dependencies, which means that it does not need to be managed by Zookeeper, or the metadata is stored in MySQL, and only needs to maintain itself.
It is not suitable for transaction operations, such as update operations. The data used for analysis are unchanged once written, such as log data or physical examination reports.
Vectorization: After the data is vectorized, the CPU can process multiple data at the same time instead of processing only one data item.
MPP architecture: Large-scale parallel processing architecture, splitting data to multiple machines for execution and processing large amounts of data
Column storage engine: manages columns and supports storage and analysis of large wide tables. MySQL cannot do this because it crashes when there are too many fields. It is very fast to query columns individually and columns can be updated in real time.
Real-time analysis: query and analysis speed is relatively fast, in milliseconds
MySQL client connection: For example, navicat or jdbc can directly connect to it? To be verified
Horizontal expansion: One machine is too weak, I can continue to add machines to make its analytical capabilities stronger
supportthe followingBI Integration: Includes Tableau, Power BI, FineBI and Smartbi.
As a real-time data warehouse, he can only[Seconds] Synchronize data, can be real-timeMillisecond level】Inquire。
The core of the system is only FE (Frontend), BE (Backend) or CN (Compute Node) process.
Front-end (display interface), back-end (logic control), node
3.0 supports storage and computing separation, and persistent data must be stored on HDFS. Of course, you can also choose storage and computing integration.
3.0 also supports these two architectures.
What is the difference between these two? If the storage and computing are integrated, you must copy the data to startRocks. If the storage and computing are separated, you can just use the data in hdfs. There is one less step of copying. Storage and computing are separated, which saves money and disks, and allows for better dynamic expansion. When expanding capacity, you don't need to worry about storage, you can directly expand the computing nodes. The disadvantage is that you have to maintain an extra set of external data.
Hybrid deployment is not supported. If you choose storage-computing integration, you cannot separate storage and computing.
Fe is responsible for coordination and catalog management
Be Responsiblestorageandcalculate
Fe (detailed version)：
Responsible managementMetadata, manage client connections,Query Planning, query scheduling.
FE metadata is stored in memory and also has a copy on disk.
FE has three roles: leader, follower, observer
The leader is elected and is responsible for reading and writing. After writing, the metadata is updated and synchronized to followers and observers. Only when half of the followers succeed is it considered successful.
Follwer has no write permission, only read permission
Observer is optional, just like follower. It can improve query speed and does not participate in elections, which is like adding wings to a tiger.
Be (detailed version)：
Each BE is the same (there is no leader or follower), but not every BE has complete data. BE is responsible for storage and calculation. FE distributes data to BE, which stores it and generates an index.
Be calculation will divide the SQL into logical units (code level) according to the grammatical meaning, and thenAccording to data distributionIt becomes a physical unit (hardware level) and then executes locally.
Metadata: I don’t think anyone knows this. To be honest, I don’t even want to write it, just to take care of the newbies. For example, what type of data is this, a string or a number? This is metadata, which is used to modify the data.
Query planning: how much performance the plan will consume, what SQL to use, how to optimize, and how to convert it into a physical plan
Query scheduling: select which be to execute the physical plan
The smallest storage unit of starRocks is called tablet. We can partition it by ourselves and then specify the bucket.
The figure shows partitioning by time column, and then bucketing the specified 4 fields (4 columns, but 1 column is also acceptable), and then specifying 3 replicas. The data of each column and each unit of data is distributed on different nodes. A-1, A-2 and A-3 all have the same data and are backups of A.
When it expands capacity, there is no need to stop the service. When nodes are added, they will be automatically migrated. When nodes are reduced, the data will be automatically distributed evenly.
introduced【cache】The concept of Be is only responsible for calculation, and thenChanged its name to Cn(Compute Node)
Cache: automatically caches data according to query frequency.Dynamic changes
Dynamic changes: divided into 3 levels, memory, local, and external. The hottest data is in memory, the rest is in the local disk, and the cold data (not frequently used) is in the external source. Dynamic data adjustment based on your access frequency
When creating a table with storage and computing separation, you need to tell it whether to enable caching.
The following backend storages are supported:
I have explained every sentence of the system architecture of the official website in my own words. Let's start building it.
He uses the Docker container to package the environment for you, so you can start it directly.
First install docker, at least 4G memory and 10GB space.
The CPU of our server does not support avx2. I am downloading a virtual machine here and plan to install Ubuntu.22 on Windows, because my personal computer supports avx2. ---Wait until I finish downloading, then start writing the rest.
refer to:
📚 【源码解析】StarRocks 查询优化系列文章 - 原理解读 - StarRocks中文社区论坛
I have devoted myself to the research of technology for more than 30 years. I am proficient in various languages such as Java, Linux, JavaScript, PHP, CSS, etc. I have made many contributions in the field of open source. I have established a developer documentation site to share some problems in technology development for everyone to read.
