Modern data stack: develop data efficiently and assist enterprise decision-making

Produce | CSDN Cloud native

As storage and data gradually migrate to the cloud , With Snowflake The model of Lake Warehouse Integration represented by the rapid rise , The new architecture of Lake warehouse integration has brought a series of far-reaching changes to the big data processing architecture , From the concept of data engineering 、 The method of practice , To the whole stack of data processing, great changes have taken place , There are also many start-ups and tools emerging rapidly , It also represents the development direction of future data technology .

These new data engineering methods and new data stacks are called modern data stacks , There are also a lot of new technologies around modern data stacks , Including Lake Warehouse Integration 、ETL、 Data pipeline 、Dataops、 reverse ETL、Metric Store、Data Engineering wait .

CSDN Cloud native series online Summit 13 period “ Modern data stack summit ” Li Yuanjia, co-founder of next second data, is the producer , Hand in hand next second data CEO Caizhinuan 、 Yan Zhitao, co-founder of quick use cloud technology 、 Dai Lidong, co-founder of Moby Dick open source 、Kyligence Partner and vice president Lidong , Share the latest technology and best practices of modern data stack , Help everyone from multiple perspectives 、 Have a comprehensive understanding of modern technology stack .

Li Yuanjia, co-founder of next second data, said in the opening sharing , Limited by storage costs and computing power , Traditional data architecture is mainly segmented when processing data . The advantages of storage cost and high computing performance of cloud provide new possibilities for the development of data architecture , Modern data stack （Modern Data Stack,MDS） The ecosystem is taking shape .

Let's briefly review the contents of this summit . Video of the speech about the summit 、 The full article will be in CSDN Cloud native official account has been released in succession , Stay tuned .

Cloud native data pipeline

In today's data is called the oil of the new economic era , What is a data pipeline ？ In what position ？ What is its technology and evolution trend ？

For these continuous questions , Next second data CEO Cai zhinuan aims at data pipeline 1.0 to 3.0 The iterative process and technical characteristics of .

Cai zhinuan said , With SaaS The rise of , Enterprise data silos are further exacerbated . What data pipeline solves is how to automatically transfer data from different sources to people in the organization 、 System 、 Applications , And turn data into business value , Ultimately, organizations can use data like water and electricity .

Data Modeling in Modern Data Stack

Yan Zhitao, co-founder of quick use cloud technology, made an in-depth sharing on data modeling and its application in modern data stack . Yan Zhitao said , Enterprise cloud is an irreversible trend , It is also an irreversible trend for enterprises to use data to drive .

meanwhile , Yan Zhitao also shared four modern data modeling products ：

• Dbt;

• Dataform;

• Datameer;

• QuickTable.

Modern data stack DataOps

The digital age has brought us a large amount of information and data , With the maturity of cloud computing technology, data-driven business decision-making is no longer a high threshold thing . Dai Lidong, co-founder of Moby Dick open source, started from DataOps Task arrangement 、 Data integration capability and its relationship with open source , The modern data stack is deeply analyzed and practical application cases are shared .

Dai Lidong pointed out , Cloud native distributed big data workflow scheduling system DolphinScheduler High reliability 、 Simple and easy to use 、 High expansibility 、 Cloud native ability and other advantages , That's why it's popular .

meanwhile , For developers how to start from 0 To 1 Participate in open source , Dai Lidong suggested ：

• Subscribe to mailing lists , Participate in development activities , Participate in technology 、 Non technical discussion , Connect with the community ;

• First, locally run get up ;

• From childhood , steer a steady course ;

• Learn the official website contribution specification 、 technological process .

​​​​​​​

From data lake to index middle stage , Improve data analysis ROI

Kyligence Li Dong, partner and vice president, introduced the current development trend of indicators in Taiwan , And point out the current impact data analysis ROI Of the three ⼤ obstacle ：

• Indicators vary —— Large amount of data , But lack of trust ;

• “ Muddy ” The data lake of —— Wide watch explosion , Data storage and ETL Task redundancy ;

• IT cost & timeliness ——IT The cost increases rapidly with the growth of data volume and user volume .

Last , Aiming at how to use the index middle stage to solve the problems caused by the traditional data Lake , Li Dong starts with the actual case , Through explanation and Demo demonstration , Provide you with new ideas for efficient data development .

Here comes the egg

Scan the code and join the communication group , Get this summit PPT