Open source real-time data warehouse Apache Doris graduated, how to go further in the future?

In recent years , In the digital 、 Under the trend of intelligence , Data demand is growing rapidly , The database market continues to rise . lately , Open source real-time database donated by Baidu Doris from Apache Software Foundation incubator graduate , Officially become Apache Top projects , It has become an important event in the database industry recently . meanwhile , This is also a major success of the emerging data warehouse and data analysis open source projects led by Chinese open source developers .

Congfeng nest 、 Baidu statistics to the open source community ,Doris Data analysis for 14 years

Doris It was developed by Baidu and donated to Apache Open source software foundation MPP database , Its predecessor is Baidu PALO（PALO originate 2012 The core scenario positioned at that time ——OLAP Reverse alphabetic order of online analysis and processing ）.Doris Positioned as a high-performance analytical database , On 2008 The year was born , In the Chinese Internet at that time , Baidu's marketing and promotion system Phoenix Nest is already an enterprise level business system with a very large scale and popularity , behind , By Doris brace , And greatly improved the timeliness of data analysis of Baidu Fengchao business .

2009 year ,Doris Start to support other report systems in Baidu , Including website analysis products that support Baidu's other trump card —— Baidu statistics .2012 year Doris Grow into Baidu's first company level OLAP Analysis platform , And in the next few years, we will upgrade various core technologies , Greatly improve various technical indicators . so to speak , From its inception to now ,Doris Followed the development of Baidu business and data analysis for 14 years , Support within baidu 140 Operation of multiple business lines .

Such experience is rich enough , Worth saying , But yes Doris Come on , The above improvements and changes are just Doris The beginning of the road of development .2017 Baidu announced PALO Official open source ;2018 The year will be PALO Our core engine is donated to Apache Software foundation , And named it Apache Doris, This marks Baidu PALO The team began to push forward Doris Community development ;2020 Since then , At Baidu PALO With the joint efforts of the team and community partners ,Apache Doris The community is on the fast track of development , After a year of development ,Doris Various core competencies have been greatly enhanced , Its influence in the open source community and industry has also been further enhanced , except GitHub star The number is growing rapidly ,Doris The community has also become the China Academy of communications 2021 year “OSCAR Peak open source projects and communities ”, get “ The first trusted open source community （TWOS）” Full member certification . here we are 2022 year , With 1.0 The official release of the large version ,Doris Increasingly perfect and mature , Baidu officially launched trademark donation , Successful promotion Apache Doris Finish graduation , Become Apache Software foundation top projects .

Real time of the whole scene 、 High maturity ,Doris Polish the hard power of real-time data

Open source database products emerge in endlessly , and Apache Doris The reason why we stand out , This is closely related to its unique differentiation advantages . The first point , stay Full scene real-time aspect ,Doris Since its birth 14 In the year , It has experienced a lot of complex scene testing and work inside and outside Baidu , Except inside 140 Multiple business lines are using , On the outside there are more than 500 Many enterprises use .Doris It can fully support various data models and query scenarios of data warehouses , And provide Full scene real-time 、 High performance Support for , for example , The historical data 、 Incremental data 、 Single table query 、 Stream write 、 Batch import, etc .

Compared with , In some other real-time data warehouses on the market , Such as MOLAP Class data warehouse is mainly for historical data , Provide multidimensional modeling of relative solidification , Poor flexibility ; in addition ,MOLAP Data explosion will also occur in pre modeling ; Besides ,MOLAP The ability to support incremental data is not ideal . and Doris Can support various scenarios , And can provide good performance . in addition , Some real-time data warehouse engines with high performance in specific scenarios , Limited by its own architecture , Poor support for multi table queries , There are also significant shortcomings in concurrency . by comparison ,Doris It can provide good support in all major scenarios of the data warehouse , And can provide excellent performance .

Second point , stay Ease of use and maturity aspect , Data warehouse itself is a very complex system , System deployment 、 Operation and maintenance 、 The use threshold is quite high , After years of polishing and large-scale application Doris It is already a product with high maturity , Have excellent Ease of use and operability . for example Doris Yes SQL With perfect support , And highly compatible MySQL agreement , The threshold of using data warehouse is reduced to the greatest extent .

In deployment , In the past, data warehouses usually relied on Hadoop Etc. external components , The cost of maintenance and deployment is high ,Doris Has a minimalist core module , And there is no external dependence . meanwhile Doris With perfect MPP framework , Built in powerful distributed clustering capability , And provide easy-to-use self operation and maintenance support . for example ,Doris It can support automatic online expansion and contraction , Data can be automatically balanced in the background , There is no need for users to participate in the whole process , It significantly reduces the difficulty and risk of database expansion and contraction .

With these comprehensive functions 、 Ease of use and performance ,Doris stay 2018 Was donated to Apache after , It has quickly attracted a large number of top Chinese Internet enterprises to use and participate in the contribution . Data volume and high concurrency demand of Internet business in China , It may be unique in the world ,Doris It can meet the business needs of the whole scenario , Whether it's aggregation 、 Detailed 、 Single table 、 Various data analysis scenarios in Internet factories such as multi tables ,Doris Can better support , Easy to use .

in addition , In performance 、 Functional richness and ease of use , The technical team of the Internet company uses Doris It is more convenient for the cluster to realize dynamic expansion and contraction , When a node fails , Data can be migrated automatically , Completely independent of external systems , It will not affect the upper business system , Make cluster operation and maintenance very simple . meanwhile Doris It provides an online query experience with high concurrency and high timeliness , Help the technical team of Internet enterprises with extremely low construction cost , Meet the business side report 、 Large screen or ad hoc query .

Doris As a complex system , The community will continue to polish its core competence in the future , Keep leading in core technical indicators . for example , Baidu will continue Polish and optimize the existing query layer 、 Core components such as execution layer and storage layer , Optimize it to be more perfect 、 A more stable 、 Better performance ; Realization A complete column oriented vectorization execution engine , Completely eliminate bank deposits ; Achieve a new CBO Optimizer and more refined and rich Statistics , take Doris The performance and ease of use of ; strengthening Doris Of Profiling、Trouble Shooting、 Fine grained resource monitoring and control capabilities ; Promote research and development of hot and cold separation 、 Multi table materialized view 、 Complex type 、 Update and other new features with the highest voice in the community .

Firmly open source , Give Way Doris go further

Open source , It has always been a way for developers to show the strength of hard core technology , It is also the purest goal pursued by excellent technicians . Developers in China , Already from open source users , Gradually become the contributor of open source and the leader of core projects . from Apache Doris Community development , We see Baidu and Doris The community's adherence to the spirit of open source , And an open technological mindset . Now ,Apache Doris Successfully completed graduation , Become Apache Top projects , Graduation is a new beginning , It also means new responsibilities and missions . As Apache Doris Of “ Mother's family ”, Baidu has accumulated many years of experience in open source , And successfully launched the propeller 、 Apollo 、ECharts、BFE、BRPC、XuperChain And a number of well-known open source projects . about Doris Future development , Baidu will continue to devote itself to the open source community , And will be in PALO The technical experience gained in the commercialization practice of continues to be shared with the community .

Apache Doris The development of , Cannot do without Apache Professional guidance of the software foundation , And meituan 、 millet 、 JD.COM 、 tencent 、 The full participation of a large number of community partners such as byte . future , With the development of the project , More users and enterprises will participate in the use and contribution of the community . Facing such a new situation , Baidu PALO The team will Apache Way For guidance , Work with your community partners , Establish and maintain fairness 、 to open up 、 Code of conduct for collaboration , Strive to build a vibrant open source community .

Apache Doris Graduation meeting ,Apache Software Foundation Project mentor 、 All core community developers were present

PALO Cloud It's Baidu. PALO The team with Apache Doris An enterprise level cloud native real-time data warehouse built for the core engine , After years of large-scale commercialization ,PALO It has perfect enterprise level features and leading cloud native capabilities , Including the separation of deposit and settlement 、 Multi warehouse shared storage 、 Heterogeneous computing 、 Disaster recovery backup and other proprietary capabilities . meanwhile , Based on the unique offline integration 、 Lake Warehouse Integration Capability ,PALO Cloud It will help users greatly simplify the traditional big data infrastructure , Reduce resources and maintenance costs , Truly build a new generation of data centric unified Lake warehouse with full scene . future ,PALO Cloud It will provide comprehensive cross cloud and hybrid cloud capability support , Including multi cloud deployment 、 Cross cloud disaster recovery 、 Cross cloud cold heat separation 、 Cross cloud elasticity , Help enterprises enter the data analysis “ Cloudy primordial ” Time .

China is the starting point and foundation of Baidu open source , future , Baidu will also continue to go global . From the beginning of donation ,PALO The goal of the team is to Doris Build a world-class open source data warehouse project , Baidu's official website 、 file 、 From the very beginning, discussion groups are built and operated according to the standards of globalization . In recent years , As the project matures ,Doris It also began to attract some overseas enterprises and technical teams to join and use . at present , Baidu is going to Apache Doris and PALO Cloud Go global , With excellent product power , Believe in Doris and PALO In the future, it will be accepted and recognized by more overseas users . blessing Doris, blessing PALO, Keep growing , Using technology , To change the world ！