It's another summer of open source. 12000 project bonuses are waiting for you!

Another year 【 Open source summer 】 season . This year, ,Apache DolphinScheduler Also involved in 【 Open source summer 】 In the activity , I hope more students will pay attention to Apache DolphinScheduler, And actively participate in the joint construction of the project and get bonuses ！

No.1 Activity Introduction

The summer of open source is made up of “ Open source software supply chain lighting plan ” A summer open source activity initiated and long supported , By Institute of software, Chinese Academy of Sciences and openEuler The community co organized , It aims to encourage students to actively participate in the development and maintenance of open source software , Promote the vigorous development of excellent open source software community , Cultivate and explore more excellent developers .

Students can choose the project they are interested in to apply for , And get the personal guidance of community tutors after winning the election . According to the difficulty and completion of the project , Participants will also receive open source summer bonus and closing Certificate .

Open source summer official website ：https://summer.iscas.ac.cn/

No.2 Schedule of activities

No.3 Project introduction

Apache DolphinScheduler It is a cloud native and easy to expand visualization DAG Workflow task scheduling system . Committed to solving complex task dependencies in data processing processes , Make various task types (Spark /Flink/MR/Shell/Python/SQL etc. ) Out of the box in workflow orchestration .

No.4 Bonus settings

• Advanced ： Bonus RMB 12000 element
• Optimize the task of the class , For example, improve performance , Reduce resource occupancy
• Basics ： Bonus RMB 8000 element
• Task of function class , For example, add one or several important features to the community's open source projects

No.5 Eight topics

This activity ,Apache DolphinScheduler The open source community has 8 The project topics are as follows , Welcome to participate ：

01 by DolphinScheduler Python API Add resource file

【 Item number 】222290294

【 Project difficulty 】 Advanced /Advanced

【 programing language 】Python

【 Project description 】 at present ,DolphinScheduler Python API You can only pass the parameters of the task in string format , For example, our tutorial

dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py

We just pass a string to shell Task type . But generally speaking , The user wants the specific execution code to be stored in other systems （ Hope for better maintenance and integration ）, For example, store specific files in the local file system 、GitHub、GitLab、Amazon S3、 Alibaba cloud OSS etc. . Maybe we can add syntax sugar to make it easier for users to operate external files

Mission task_parent Will be taken from URL https://github.com/apache/dolphinscheduler/blob/dev/script/install.sh Load the contents of the file and pass it to the parameter command, It makes our DAG Files are easier and maintainable

【 Project output requirements 】

Realization Resource Plug in of

Implement specific plug-ins Resource Including but not limited to local file system ,GitHub, GitLab, Amazon S3, Alibaba cloud OSS

Python API Pass test coverage , The current threshold is 90%

file , Including the development of （ How to implement plug-ins in binary design ） And use （ Every Resource The plug-in USES ）

【 Project technical requirements 】

be familiar with python, Include decorator、io、network、unittest、document

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】 Zhong Jiajie

Contact information ：[email protected]

02 Resource center optimization

【 Item number 】222290456

【 Project difficulty 】 Advanced /Advanced

【 programing language 】Java、Shell

【 Project description 】 restructure Dolphinscheduler Reading and writing logic of Resource Center

Read write resource center HDFS/S3/ Local directory By in the database table t_ds_resources Log files / Folder directory to achieve file recording , But after user initialization DS Or the user is in DS When the content stored by a third party is modified by the outside of DS Unable to synchronize in real time , Hope to ensure real-time synchronization with the content stored by a third party （ Call a third party API Add, delete, modify and check the documents ）

【 Project output requirements 】

DS Do not store the path of files in the content Center , The reading and writing of the resource center are all done by calling a third party API Realization , Only the content path that the task instance depends on is saved in the database for file deletion verification .

Realize all the functions of the existing resource center , These include file / Folder Additions and deletions , The task instance calls the resource center

Back end API file

【 Project technical requirements 】

understand Linux System , Yes Linux Familiar with document management

understand HDFS/S3 Components , Be able to quickly learn from official documents API Use

be familiar with Java Language

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】 Xiang Zihao

Contact information ：[email protected]

03 master/worker/api Module log optimization

【 Item number 】222290457

【 Project difficulty 】 Advanced /Advanced

【 programing language 】Java

【 Project description 】 because DS Logs are not convenient for real location problems , Print logs need to be standardized and optimized

【 Project output requirements 】

Sort out DS Print log specification guidance document , And submit to the document library

take master/worker/api The module is implemented according to the specification documents and submitted to the development branch

【 Project technical requirements 】

be familiar with Java Language

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】 Jin Yong

Contact information ：jiny.li@foxmail.com

04 Automatic collection 、 Save the submitted Yarn Operational application id

【 Item number 】222290458

【 Project difficulty 】 Advanced /Advanced

【 programing language 】Java

【 Project description 】DolphinScheduler Support a variety of jobs , such as Python、Shell etc. , One of the basic functions is to collect the data created by these jobs Yarn Job Of application id. The current implementation is to parse from the job log with a regular expression , But this is not a perfect way to achieve , The task of this subject is to use a new way to automatically collect 、 Report errors Yarn Job Of application id Information , Yes, of course , Must be transparent to users , That is, there is no intrusion into the job created by the user .

【 Project output requirements 】

Automatic collection 、 Report errors Yarn Job Of application id Information

Must be transparent to users , That is, there is no intrusion into the job created by the user

【 Project technical requirements 】

be familiar with Java, Yarn, Hive, Spark SQL, AOP

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】gabrywu

Contact information ：[email protected]

05 by DolphinScheduler Add workflow for service initialization demo

【 Item number 】222290459

【 Project difficulty 】 Advanced /Advanced

【 programing language 】Java、Shell

【 Project description 】 The user is starting DolphinScheduler After service , have access to demo-tool Program preset workflow demo, Including but not limited to simple shell Mission 、 Logical component tasks （switch、dependent、subprocess、condition）、 Parameter transfer and other functions , Guide users to use DolphinScheduler.

【 Project output requirements 】

Preset workflow demo Of tool Program

Service startup script add init Pattern

workflow demo Including but not limited to shell Mission 、 Logical component tasks （switch、dependent、subprocess、condition）、 Parameter transfer, etc

【 Project technical requirements 】

be familiar with Java、Shell

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】 Cai Shunfeng

Contact information ：[email protected]

06 by DolphinScheduler add to Java Task type

【 Item number 】222290460

【 Project difficulty 】 Advanced /Advanced

【 programing language 】Java、Shell、Vue、TypeScript

【 Project description 】 Provide Java Type of scheduling task , Include Jar And customization Java There are two ways to code , adopt WEBUI Definition Java What you need to run Resources、Libs Resource files and JVM Parameters , Override scheduled task call Java Program scenario and improve the user's convenience of configuration .

【 Project output requirements 】

The executable Java Type task plug-in

Java Type plug-ins support Jar The way , Provide Resources、Libs Resource files and JVM Dynamic parameter input of parameters

Java Type plug-ins support WebUI Customize Java Code mode , Provide Java Class just in time compilation , perform

【 Project technical requirements 】

be familiar with Shell、Java、SPI、Java compiler、VUE3、TypeScript

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】Kerwin

Contact information ：[email protected]

07 Support test tasks

【 Item number 】222290461

【 Project difficulty 】 Advanced /Advanced

【 programing language 】Java、Vue

【 Project description 】

The current new / The modified workflow is difficult to verify whether the workflow configuration is correct before going online , Can only be verified by executing in an online environment , Therefore, it is necessary to support the test environment to perform tasks .

The data source center supports configuring the test environment , The online environment source can be bound to a test environment source . Task execution and workflow execution support “ Whether to test ” Options , When choosing a test ,Worker Node before task execution , Automatically replace the online environment source with the bound test environment source , Combined with Worker Grouping realizes online communication 、 Isolation of test environment , Finally achieve the goal of supporting test tasks .

【 Project output requirements 】

The data source configuration page adds online data 、 Test options , Support binding test source

Workflow execution page 、 The task execution page adds online 、 Test options

Master node 、Worker Node resolution execution environment identification , Replace the data source with the test data source

Workflow instance 、 Add the execution environment ID on the task instance page

【 Project technical requirements 】

be familiar with Java、Spring、VUE3

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】 Wen He Min

Contact information ：[email protected]

08 by DolphinScheduler Increase the authorization management of read and write permissions in the project and resource center

【 Item number 】222290462

【 Project difficulty 】 Basics /Basic

【 programing language 】Java、Vue、TypeScript

【 Project description 】 At present DolphinScheduler The control of permissions is relatively simple , It is hoped to add operation permission control during project and Resource Center authorization to separate read and write permissions .

【 Project output requirements 】

Distinguish read and write permissions for all contents of a single item , Grant read-only or read-write permissions to a single user for a single item

Distinguish read and write permissions for all contents of the resource center , Distinguish read-only or read-write permissions based on the authorized content of the current resource center

During authorization, read-only and read-write authorization operations are added to the front page

【 Project technical requirements 】

be familiar with Java、Spring、Vue3、TypeScript

【 Project achievement warehouse 】

https://github.com/apache/dolphinscheduler

https://github.com/apache/dolphinscheduler-website

【 Project Master Tutor 】 Sun Chaohe

Contact information ：[email protected]

No.6 Apply to participate in the project

01 Apply for qualification

This activity is for those who have reached the age of 18 One year old student at school .

Summer graduates , As long as the student ID is valid at the time of application , You can submit an application .

Overseas students can provide admission notice / Student cards / Proof of student identity .

02 Participate in the process

03 Apply for advice

Welcome to take the initiative to send email to the tutor for communication , Attach your own plan and resume , Know you better ~

Carefully understand the community project documents , Fully communicate with your mentor .

Welcome to join the exchange group

Participation and contribution

With the rapid rise of domestic open source ,Apache DolphinScheduler The community is booming , In order to make better use of 、 Easy to use scheduling , Sincerely welcome partners who love open source to join the open source community , Contribute to the rise of China's open source , Let local open source go global .

Participate in DolphinScheduler The community has a lot of ways to participate and contribute , Include ：

Contribute the first PR( file 、 Code ) We also hope it's simple , first PR It's used to familiarize with the submission process and community collaboration, and feel the friendliness of the community .

The community summarizes the following list of questions for novices ：https://github.com/apache/dolphinscheduler/issues/5689

List of non novice questions ：https://github.com/apache/dolphinscheduler/issues?q=is%3Aopen+is%3Aissue+label%3A"volunteer+wanted"

How to participate in contribution link ：https://dolphinscheduler.apache.org/zh-cn/docs/development/contribute.html

Come on ,DolphinScheduler The open source community needs your participation , Contribute to the rise of China's open source , Even if it's just a small tile , The power that comes together is enormous .

If you participate in open source, you can compete with experts from all walks of life , Quickly improve your skills , If you want to contribute , We have a donor seed incubation group , You can add a community assistant wechat (Leonard-ds) , Hand in hand teaches you ( Contributors of all levels , Have a craigslist , The key is to have a willing heart to contribute ).

When adding a small assistant wechat, please explain that you want to participate in the contribution .

Come on , The open source community is looking forward to your participation .

Activity recommendation

When data resources become an essential element in the development of production and even in the process of survival , How can enterprises help enterprises implement data services in the whole life cycle through data integration ？5 month 14 Japan , Data integration framework Apache SeaTunnel(Incubating) One stop data integration platform will be invited Apache InLong(Incubating) Technical experts and open source contributors , Come to the live studio , Talk to everyone about using Apache SeaTunnel(Incubating) And Apache InLong(Incubating) Practical experience and experience after .

Live link ：https://live.bilibili.com/24523914?broadcast_type=0&is_room_feed=1&spm_id_from=333.999.0.0