Namenode and secondarynamenode

24

Catalog

One 、NameNode Metadata storage in ：

Two 、2NN The arrival of the

3、 ... and 、NameNode Working mechanism

Four 、Fsimage and Edits Concept

5、 ... and 、oiv see Fsimage file

6、 ... and 、oev see Edits file

NameNode Metadata storage in ：

In memory ：

benefits ： Fast calculation

Disadvantage ： Poor reliability

On disk ：

benefits ： High reliability

Disadvantage ： Slow calculation

therefore , introduce Edits file （ Only append , It's very efficient ）. Whenever metadata is updated or added Canadian dollar data , Modify the metadata in memory and append to Edits in . such , once NameNode The node power , can In order to pass the FsImage and Edits The merger of , Synthetic metadata .

2NN The arrival of the

however , If it takes a long time to add data to Edits in , This causes the file data to be too large , Low efficiency , And once power failure , It takes too long to recover the metadata . therefore , It needs to be done on a regular basis FsImage and Edits The merger of , If this Operations by NameNode Node to complete , It's too inefficient . therefore , Introduce a new node SecondaryNamenode, Dedicated to FsImage and Edits The merger of .

NameNode Working mechanism

The first stage ：NameNode start-up

1、 First boot NameNode After the formatting , establish Fsimage and Edits file . If it's not the first time , Directly loads the edit log and mirror files into memory

2、 The request of the client to add, delete or change the metadata

3、NameNode Record operation log , Update rolling log

4、NameNode Add, delete and modify metadata in memory

The second stage ：Secondary NameNode Work

1、Secondary NameNode inquiry NameNode If you need CheckPoint. Direct return  NameNode Whether to check the result .

2、Secondary NameNode Request execution ChectPoint.

3、NameNode Scroll what you're writing Edits journal

4、 Copy the edit log and mirror files before scrolling to Secondary NameNode

5、Secondary NameNode Load the edit log and mirror files into memory , And merge .

6、 Generate a new image file fsimage.chkpoint.

7、 Copy fsimage.chkpoint To NameNode.

8、NameNode take fsimage.chkpoint Rename as fsimage.

Fsimage and Edits Concept

NameNode After being formatted , Will be in /opt/module/hadoop-3.1.3/data/dfs/name/current The following files are generated in the directory

 1、Fsimage file ：HDFS A permanent checkpoint for file system metadata , It includes HDFS All the directories and files of the file system indoe Serialization information for .

2、Edits file ： Deposit HDFS The path of all update operations to the file system , All writes performed by the file system client are first recorded to Edits In file .

3、seen_txid The file holds a number , It's the last one edits_ The number of .

4、 Every time NameNode When it starts, it will Fsimage File read into memory , load Edits Update operation inside , Make sure the metadata information in memory is up to date 、 synchronous , Can be seen as NameNode When it starts, it will Fsimage and Edits The documents were merged .

oiv see Fsimage file

Basic grammar

hdfs oiv -p file type -i Image file -o The output path of the converted file

Practice ：

  After cluster startup , requirement DataNode Report block information , And report it again after an interval of time

 oev see Edits file

Basic grammar

hdfs oev -p file type -i Edit log -o The output path of the converted file

Practice