当前位置：网站首页>Azkaban installation documentation
Azkaban installation documentation
2022-07-19 12:57:00 【InfoQ】
Hello everyone , I'm Huai Jinyu , A big data sprouts new , There are two gold eaters at home , Jia and Jia , Up can code Lower energy teach My all-around Daddy
If you like my article , Sure [ Focus on ]+[ give the thumbs-up ]+[ Comment on ], Your third company is my driving force , Looking forward to growing with you ~
1 Azkaban Introduce
Azkaban It's something like Oozie Workflow control engine , It can be used to solve multiple Hadoop（ or Spark etc. ） The dependency between offline computing tasks . It can also be used to replace crontab To schedule periodic tasks , And more intuitive , reliable , At the same time, it provides a beautiful visual management interface .
Azkaban It consists of three parts ：
1、Relational Database(Mysql)
azkaban Store most of the status information in MySQL in ,Azkaban Web Server and Azkaban Executor Server You also need to visit DB.
2、Azkaban Web Server
Provides Web UI, yes azkaban The main manager of , Include project Management of , authentication , Dispatch , Monitor the workflow execution process, etc .
3、Azkaban Executor Server
Scheduling workflows and tasks , Record the log of workflow live tasks , The reason will be AzkabanWebServer and AzkabanExecutorServer Separate , Mainly because after a task flow fails , It is more convenient to re execute . And it's more conducive to Azkaban System upgrade
2 Preparation before construction
2.1 Description of the overall installation process
1、 stay mysql Created in azkaban Relevant account , library , surface
2、 take azkaban Import the table creation script of into the database
3、 install azkaban-executor- server
4、 install azkaban-web-server
2.2 Installation version description
JDK：jdk1.8.0_77 64 position
MySQL-5.6 Above version
db Script ：azkaban-db-0.1.0-SNAPSHOT.tar.gz
perform Server：azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
Web Server：azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
2.3 Resource download address
2.4 Description of this installation node
This deployment
10.60.247.50 Deploy azkaban-web-server
10.60.247.193 Deploy azkaban-exec-server
10.60.247.187 Deploy azkaban-exec-server
10.60.247.213 by mysql node
3 Set up the environment
3.1 Preparations before installation
3.1.1 Check java Environmental Science （ All nodes ）
[[email protected] flume]# java -version
3.1.2 Check if there is spark（exec node ）
[[email protected] ~]# spark-submit --version
3.2 Import azkaban-db Related scripts
- take azkaban-db-0.1.0-SNAPSHOT.tar.gz Upload to /app/src/azkaban
- decompression azkaban-db-0.1.0-SNAPSHOT.tar.gz
[[email protected] ~]# tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz
among create-all-sql-0.1.0-SNAPSHOT.sql For the total of all statement creation
- Execute table creation statement
[[email protected] bin]$ mysql -uazkaban -pAzkaban*123qwe! -h10.60.247.213
mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'azkaban'@'%';
mysql> FLUSH PRIVILEGES;
mysql> CREATE DATABASE azkaban;
mysql> use azkaban;
mysql>source /app/src/azkaban/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql
mysql> show tables;
Through the above process, the table is created successfully
3.3 install azkaban-exec-server
- take /app/azkaban The directory is assigned to hdfs user , And switch to hdfs
[[email protected] app]# chown -R hdfs:hdfs azkaban/
- take azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz Upload to the node planned in advance , The catalogue is /app/azkaban And unzip azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
[[email protected] azkaban]# tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
- Modify configuration item conf The configuration file under the directory azkaban.properties
# Azkaban Personalization Settings
azkaban.name=Sea
azkaban.label=Beluga Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=19099
# Where the Azkaban web server is located
azkaban.webserver.url=http://10.60.247.50:19099
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=10.60.247.213
mysql.database=azkaban
mysql.user=azkaban
mysql.password=Azkaban*123qwe!
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30
executor.props.resolve.overrideExisting.enabled=false
\
- Startup and shutdown
Calling the startup and shutdown scripts requires
Start node
[[email protected] azkaban-exec-server-0.1.0-SNAPSHOT]$ ./bin/start-exec.sh
Now the node has started , But the node is still inactive , You need to execute the following statement
[[email protected] azkaban-exec-server-0.1.0-SNAPSHOT]$ curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo
Query log
Close the node
[[email protected] azkaban-exec-server-0.1.0-SNAPSHOT]$ ./bin/shutdown-exec.sh
3.4 install azkaban-web-server
- /app/azkaban The directory is assigned to hdfs user , And switch to hdfs user
[[email protected] app]# chown -R hdfs:hdfs azkaban/
- take azkaban-web-server-0.1.0-SNAPSHOT.tar.gz Upload to the node planned in advance , The catalogue is /app/azkaban And unzip azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
[[email protected] azkaban]# tar -zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
- Generate the key
Generate security authentication secret key ：Azkaban Very good for safety management , adopt SLA、jetty And so on
[[email protected] azkaban-web-server-0.1.0-SNAPSHOT]# keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Keystore file location
- Modify configuration item conf The configuration file under the directory azkaban.properties
# Azkaban Personalization Settings
azkaban.name=Sea
azkaban.label=BelugaAzkaban
# Azkaban Personalization Settings
azkaban.name=Sea
azkaban.label=BelugaAzkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=19099
# Appoint keystore The location of the file
jetty.keystore=/app/soft/azkaban-web-server-0.1.0-SNAPSHOT/keystore
#keystore The password
jetty.password=123456
jetty.keypassword=123456
# Appoint keystore The location of the file
jetty.truststore=/app/soft/azkaban-web-server-0.1.0-SNAPSHOT/keystore
#keystore The password
jetty.trustpassword=123456
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=10.60.247.213
mysql.database=azkaban
mysql.user=azkaban
mysql.password=Azkaban*123qwe!
mysql.numconnections=100
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1
\
- Startup and shutdown
Execute startup script
[[email protected] azkaban-web-server-0.1.0-SNAPSHOT]$ ./bin/start-web.sh
Execute the close script
[[email protected] azkaban-web-server-0.1.0-SNAPSHOT]$ ./bin/shutdown-web.sh
Conclusion
If you like my article , Sure [ Focus on ]+[ give the thumbs-up ]+[ Comment on ], Your third company is my driving force , Looking forward to growing with you ~
But watch the Penguin 【 Huai Jin holds Yu's Jia and Jia 】, Get the resource download method
边栏推荐
- When will the moon appear
- About the "bottom reading" mentality, it makes you exhausted 2020-03-15
- Dynamic memory planning
- 面试难题：分布式 Session 实现难点，这篇就够！
- 光大期货网上开户安全吗？有没有开户指引？
- S32K148_ Can drive (bare metal development)
- Ultrasonic sensor (chx01) learning notes Ⅲ - I2C reading and writing operation
- Label ball problem
- Binary tree 2-symmetry recursion problem
- Installation and use of MySQL under Linux
猜你喜欢
Mycat2 builds MySQL master-slave separation
2022 global developer salary exposure: China ranks 19th, with an average annual salary of $23790
Interview difficulties: difficulties in implementing distributed session, this is enough!
Att & CK actual combat series - red team actual combat (-)
虞美人·寄公度
When will the deflationary market reverse? How should we operate? 2020-03-13
In depth sorting: summary of machine learning modeling and parameter adjustment methods
MOF customized materials | ultra thin MOF nanobelts | magnetic Fe3O4 @cd-mof nanocomposites | zif-8/ graphene composite nanoparticles
Yu Meimei, Ji Gongdu
10 minutes to customize the pedestrian analysis system, detection and tracking, behavior recognition, human attributes all in one!
随机推荐
最小交換次數
CMOS开关学习（一）
The active and standby cache of redis cluster is full, causing frequent active and standby switchover
Is the career direction of test / development programmers over 35 a turning point in the workplace?
Differences between get requests and post requests and usage examples
结构体内存对齐、位段、联合
The combination of fastadmin with and filed causes field failure
2022全球开发者薪资曝光：中国排第19名，平均年薪23,790美元
Do you still need to release the database connection manually with typeorm
JS operation string string string
【错误记录/selectpicker】dropdown menu显示位置出现偏移
动态内存规划
506. Relative ranking
基于STM32设计的云端健康管理系统(采用阿里云物联网平台)
GO 单元测试
XML建模(简单易学)
数组去重 数组排序 最大值 求和 类数组转化 过滤数组
[dynamic planning]dp27 jumping game (II) - medium
How can MySQL delete data tables and associated data tables
35岁以上的测试/开发程序员职业生涯走向，是职场转折点吗？