当前位置:网站首页>Configure map reduce workflow in oozie
Configure map reduce workflow in oozie
2022-07-19 02:05:00 【sun_ xo】
## prepare wordcount case
$ cd ~/work/oozie-5.2.1
$ tree oozie
oozie
├── apps
│ └── wordcount
│ ├── job.properties
│ ├── job.properties.orig
│ ├── lib
│ │ └── wordcount.jar
│ ├── workflow.xml
│ └── workflow.xml.orig
└── data
└── wordcount
└── input
└── in.txt
$ cat oozie/apps/wordcount/job.properties
nameNode=hdfs://localhost:9000
resourceManager=localhost:8032
queueName=default
oozieRoot=user/${user.name}/oozie
oozie.wf.application.path=${nameNode}/${oozieRoot}/apps/wordcount/workflow.xml
inputDir=data/wordcount/input
outputDir=data/wordcount/output$ cat oozie/apps/wordcount/workflow.xml
<workflow-app xmlns="uri:oozie:workflow:1.0" name="wordcount-wf">
<start to="wordcount-node"/>
<action name="wordcount-node">
<map-reduce>
<resource-manager>${resourceManager}</resource-manager>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/${oozieRoot}/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapred.mapper.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.reducer.new-api</name>
<value>true</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
<property>
<name>mapreduce.job.map.class</name>
<value>WordCount$TokenizerMapper</value>
</property>
<property>
<name>mapreduce.job.reduce.class</name>
<value>WordCount$IntSumReducer</value>
</property>
<property>
<name>mapreduce.job.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.job.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.inputdir</name>
<value>${nameNode}/${oozieRoot}/${inputDir}</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>${nameNode}/${oozieRoot}/${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
Note: all key / value pairs in workflow.xml can be searched from
http://localhost:8088/cluster -> Application's History -> Job / Configuration
$ cat oozie/data/wordcount/input/in.txt
hadoop
mapreduce
hdfs
hbase
hdfs
spark
hive
hcatalog
sqoop
flume
kafka
oozie
hue## upload case
$ hdfs dfs -put oozie /user/sun_xo/oozie
$ hdfs dfs -rm -f -r /user/sun_xo/oozie/data/wordcount/output
## start oozie
$ cd ~/work/oozie-5.2.1
$ bin/oozied.sh start
$ export OOZIE_URL=http://localhost:11000/oozie
$ bin/oozie job -config oozie/apps/wordcount/job.properties -run
job: 0000000-220523112321053-oozie-sun_-W
$ bin/oozie job -info 0000000-220523112321053-oozie-sun_-W
ob ID : 0000000-220602202911687-oozie-sun_-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : wordcount-wf
App Path : hdfs://localhost:9000/user/sun_xo/oozie/apps/wordcount/workflow.xml
Status : SUCCEEDED
Run : 0
User : sun_xo
Group : -
Created : 2022-06-02 12:30 GMT
Started : 2022-06-02 12:30 GMT
Last Modified : 2022-06-02 12:30 GMT
Ended : 2022-06-02 12:30 GMT
CoordAction ID: -
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
[email protected]:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
[email protected] OK application_1653709899071_0004SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
[email protected] OK - OK -
------------------------------------------------------------------------------------------------------------------------------------$ hdfs dfs -cat "/user/sun_xo/oozie/data/wordcount/output/*"
flume 1
hadoop 1
hbase 1
hcatalog 1
hdfs 2
hive 1
hue 1
kafka 1
mapreduce 1
oozie 1
spark 1
sqoop 1reference: build Ozzie Environmental Science
reference: build map-reduce development environment
边栏推荐
- 不会的查一查
- Characteristics and application points of electrolytic capacitor
- JS practical tips
- 02基于ZigBee的智能家居系统设计
- Hue Oozie Editor 调度 shell
- [literature reading] small footprint keyword spotting with multi scale temporary revolution
- 搭建Spark on yarn环境
- Neutralizing Self-Selection Bias in Sampling for Sortition
- Prohibit smart Safari from playing automatically when opening a web page
- [literature reading] isl: an integer set library for the polygonal model
猜你喜欢

Basic principle and parameter interpretation of operational amplifier

Apt get update error: hash checksum does not match

Characteristics and application points of electrolytic capacitor
![[literature reading] multi state MRAM cells for hardware neural computing](/img/55/141cb88dff35d5d0e7af1f860b78df.png)
[literature reading] multi state MRAM cells for hardware neural computing

Fairness in Semi-supervised Learning: Unlabeled Data Help to Reduce Discrimination

ViLT Vision-and-Language Transformer Without Convolution or Region Supervision

Neutralizing Self-Selection Bias in Sampling for Sortition

动手学深度学习---从全连接层到卷积层篇

On the properties and methods of list < t >

CAN协议通信
随机推荐
Problems encountered in yolov3 training its own data set
Array definition format
Suivi du mode de méthode de l'usine
Handling Conditional Discrimination(可解释歧视和确切的歧视)
VIM 配置文件
不会的查一查
偏差(bias)和方差(variance)
switch详解
Saber's most powerful digital analog mixed signal simulation software
霍夫变换讲解
DGC最佳实践:机密数据入湖,如何保证数据不被泄露?
SAE J1708/J1587 协议详解
Compilation and link of C language program
Apt get update error: hash checksum does not match
JS tree view array batch circular operation
Labelme正常启动,但无法打开
03 design of urban road dedusting and cooling system based on ZigBee
【pycharm】Cannot find reference ‘XXX‘ in ‘__ init__. Py 'solution
在Oozie中配置 map-reduce workflow
Foo bar 什么鬼?