当前位置:网站首页>Configure map reduce workflow in oozie
Configure map reduce workflow in oozie
2022-07-19 02:05:00 【sun_ xo】
## prepare wordcount case
$ cd ~/work/oozie-5.2.1
$ tree oozie
oozie
├── apps
│ └── wordcount
│ ├── job.properties
│ ├── job.properties.orig
│ ├── lib
│ │ └── wordcount.jar
│ ├── workflow.xml
│ └── workflow.xml.orig
└── data
└── wordcount
└── input
└── in.txt
$ cat oozie/apps/wordcount/job.properties
nameNode=hdfs://localhost:9000
resourceManager=localhost:8032
queueName=default
oozieRoot=user/${user.name}/oozie
oozie.wf.application.path=${nameNode}/${oozieRoot}/apps/wordcount/workflow.xml
inputDir=data/wordcount/input
outputDir=data/wordcount/output$ cat oozie/apps/wordcount/workflow.xml
<workflow-app xmlns="uri:oozie:workflow:1.0" name="wordcount-wf">
<start to="wordcount-node"/>
<action name="wordcount-node">
<map-reduce>
<resource-manager>${resourceManager}</resource-manager>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/${oozieRoot}/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapred.mapper.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.reducer.new-api</name>
<value>true</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
<property>
<name>mapreduce.job.map.class</name>
<value>WordCount$TokenizerMapper</value>
</property>
<property>
<name>mapreduce.job.reduce.class</name>
<value>WordCount$IntSumReducer</value>
</property>
<property>
<name>mapreduce.job.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.job.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.inputdir</name>
<value>${nameNode}/${oozieRoot}/${inputDir}</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>${nameNode}/${oozieRoot}/${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
Note: all key / value pairs in workflow.xml can be searched from
http://localhost:8088/cluster -> Application's History -> Job / Configuration
$ cat oozie/data/wordcount/input/in.txt
hadoop
mapreduce
hdfs
hbase
hdfs
spark
hive
hcatalog
sqoop
flume
kafka
oozie
hue## upload case
$ hdfs dfs -put oozie /user/sun_xo/oozie
$ hdfs dfs -rm -f -r /user/sun_xo/oozie/data/wordcount/output
## start oozie
$ cd ~/work/oozie-5.2.1
$ bin/oozied.sh start
$ export OOZIE_URL=http://localhost:11000/oozie
$ bin/oozie job -config oozie/apps/wordcount/job.properties -run
job: 0000000-220523112321053-oozie-sun_-W
$ bin/oozie job -info 0000000-220523112321053-oozie-sun_-W
ob ID : 0000000-220602202911687-oozie-sun_-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : wordcount-wf
App Path : hdfs://localhost:9000/user/sun_xo/oozie/apps/wordcount/workflow.xml
Status : SUCCEEDED
Run : 0
User : sun_xo
Group : -
Created : 2022-06-02 12:30 GMT
Started : 2022-06-02 12:30 GMT
Last Modified : 2022-06-02 12:30 GMT
Ended : 2022-06-02 12:30 GMT
CoordAction ID: -
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
[email protected]:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
[email protected] OK application_1653709899071_0004SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
[email protected] OK - OK -
------------------------------------------------------------------------------------------------------------------------------------$ hdfs dfs -cat "/user/sun_xo/oozie/data/wordcount/output/*"
flume 1
hadoop 1
hbase 1
hcatalog 1
hdfs 2
hive 1
hue 1
kafka 1
mapreduce 1
oozie 1
spark 1
sqoop 1reference: build Ozzie Environmental Science
reference: build map-reduce development environment
边栏推荐
- 中心极限定理
- 关于List<T>的属性与方法
- bag of visual words(BoVW)视觉词袋 个人理解
- [MySQL] windows install MySQL 5.6
- 集成学习
- The differences and usage of cookies, localstorage and sessionstorage
- L1,L2范数
- 06 design of smart electronic medicine box based on stm32
- Rivaliser pour la guerre clé des utilisateurs de stock, aider les entreprises à construire un système d'étiquetage parfait 丨 01 examen en direct
- CAN协议通信
猜你喜欢

03 design of urban road dedusting and cooling system based on ZigBee

02基于ZigBee的智能家居系统设计

二阶边缘检测 - Laplacian of Guassian 高斯拉普拉斯算子

Mxnet network model (V) conditional Gan neural network

Recursive and recursive learning notes

Integrated learning

关于1000BASE-T1 1000BASE-TX和100BASE-T1

iFair: Learning Individually Fair Data Representations for Algorithmic Decision Making

CAN协议通信
![[literature reading] vaqf: full automatic software hardware co design framework for low bit vision transformer](/img/41/aca6596b1fa50a1940791e6e00a17a.png)
[literature reading] vaqf: full automatic software hardware co design framework for low bit vision transformer
随机推荐
运算放大器基本原理与参数解读
禁止自作聪明的Safari打开网页时自动播放
【pycharm】Cannot find reference ‘XXX‘ in ‘__ init__. Py 'solution
bais mintigation post-processing for individual and group fairness
动手学深度学习--多层感知机篇(MLP)
Fairness in Deep Learning: A Computational Perspective
电解电容特性及应用要点
S32K148EVB 关于ENET Loopback实验
L1,L2范数
指针常量与常量指针爱恨情仇
Monitor browser return operation - prohibit returning to the previous page
05 design of street lamp control fault detection system based on ZigBee
01基于RFID的智能仓储管理系统设计
Opengauss Developer Day 2022 dongfangtong sincerely invites you to visit the "dongfangtong ecological tools sub forum"
04 design of indoor wireless positioning system based on ZigBee
笔记:光源选型与应用
01 design of intelligent warehouse management system based on RFID
C language operator priority
集成学习
搭建Spark on yarn环境