Technology Sharing

Data Lake and Warehouse Integration (I) Compile hudi

2024-07-12

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina


1. Big Data Component Version Information

  • hudi-0.14.1
  • zookeeper-3.5.7
  • seatunnel-2.3.4
  • kafka_2.12-3.5.2
  • hadoop-3.3.5
  • mysql-5.7.28
  • apache-hive-3.1.3
  • spark-3.3.1
  • flink-1.17.2
  • apache-dolphinscheduler-3.1.9

2. Data Lake Warehouse Architecture

3. Data Lake Warehouse Component Deployment Planning

node106 node107 node108 node109 node110

ZK

ZK

ZK

seatunnel

seatunnel

seatunnel

kafka

kafka

kafka

RM

RM

NM

NM

NM

NN

NN

DN

DN

DN

JN

JN