Sparkcore core design: RDD, 220716,

port Already in user

Chain programming ,

No such function ： It is recommended to install professional version , Ask me to install the package

upload

remember ：Windows In the revised , Be sure to synchronize

Running in a distributed cluster , Printing is also in the cluster

Go to log See, get

The cluster is used to test whether there is any problem in the cluster environment

RDD Function operation ：Executor

Printed on Executor In the running log of

Executor function Worker Node

Printed on worker journal ,

 1-Driver Printed in , We can see

2-Executor Printed in , We can't see （18080,stdout, Can see ）

Can you see the results of all local mode outputs It's not that there is only one local mode driver

RDD. Operator printing

print：Driver Running in

  Operator by Task To call execution ,Task function Executor in ,Executor Running on the Worker Node

sbin： Cluster management commands

bin： Client commands

demand ： Use the client command to submit the program to the cluster for running

pyspark：python Command line client

spark-sql/beeline： Submit SQL Client you

spark-submit： Submit python File client

argv

Each program has only 1 individual Driver

kill,status It can be displayed in the monitoring interface ,

0- command ,1- Options ,2- file ,3- Parameters

1-Master,2-Worker 3-Client

Driver Where does the process run ？ 

Why? Driver The print in will show ？ 

Local mode ,18080 Can see ,8080 Out of sight ,

0- command ,1- Options ,2- file ,3- Parameters

hdfs://node1:8020/export/data/pyspark_core_word_args.py

YARN In the interface Spark The program can jump directly 18080

YARN In the interface MR The program directly jumps to 19888

yarn Submit ,8032 ,

It will start according to the resources of the slave node , You can start as many as you can

driver Where to run  

Only drive branch task,

towards drive Reverse registration ,

appmaster Only on the slave node ,

Development ： Analyze and process the data

python It's a single node ,

 dataframe： Data sheet , data + The structure of the table

Yes RDD The conversion operation of , Essentially, RDD All partitions operate in parallel

Files are logic , Physically stored blocks

3 Zones 3 individual task,

Global grouping should shuffle,

tonight,review,

Spark When running in program cluster mode, two processes will be started ：Driver Drive process + Executor Computing process , Every process needs resources to run

ResourceManager 

NodeManager

AppMaster

Container

MapTask and ReduceTask

preview

Be careful ：PySpark In local mode wholeTextFiles Yes Bug, This will result in insufficient memory for a single process , The cluster environment can be used normally