fastdfs What is it?

FastDFS Is an open source lightweight distributed file system . It solves the problems of large data storage and load balancing .

FastDFS The main functions include ：⽂ Piece storage , Synchronization and access , The design is based on ⾼ can ⽤ And load balancing .FastDFS⾮ Always ⽤ Based on ⽂ Service site .

FastDFS By the tracking server （tracker server）、 Storage server （storage server） And the client （client） Three parts , Especially suitable for medium ⼩⽂ Pieces of （ Recommended range ：4KB < file_size < 500MB） Online service for carrier , For example, figure ⽚ Sharing and video sharing ⽹ standing .
 

fastdfs framework

Storage server：

Group by group （ volume ,group or volume） Organize... For units ,⼀ individual group It contains several storage machine , Data backup to each other , Storage space to group Most content ⼩ Of storage Subject to , Suggest group Multiple internal storage Try to configure the same , To avoid waste of storage space .

advantage ：⽅ I will go into ⾏ Should be ⽤ Data isolation 、 Load balancing according to application characteristics 、 Number of copies customized （group Inside storage server The quantity is the group Number of copies ）;

shortcoming ：group The capacity of is limited by the storage capacity of a single machine , At the same time when group When there is a broken machine inside , Data recovery can only rely on group Other machines in the Mainland , Make the recovery time very ⻓.

group Each inside storage Storage of depends on local ⽂ Piece system ,storage Multiple data stores can be configured ⽬ record ,⽐ if there be 10 Block disk , They are respectively mounted on /data/disk1-/data/disk10, Then you can put this 10 individual ⽬ All records are configured as storage Data storage ⽬ record .
storage Accept to write ⽂ When requesting , According to the configured rules , Choose one ⼀ Storage ⽬ Record to store ⽂ Pieces of . To avoid a single ⽬ Recorded ⽂ Too many pieces , stay storage The first ⼀ The next time you start , In every data store ⽬ record ⾥ establish 2 level ⼦⽬ record , Each level 256 individual , in total 65536 individual ⽂ Pieces of , It's new ⽂ The piece will be in hash Of ⽅ Formula is routed to one of them ⼦⽬ Record , And then ⽂ Piece data is directly used as ⼀ A local ⽂ Pieces are stored in this ⽬ Recording .

Tracker server

Tracker yes FastDFS Responsible for tracking and managing all storage and group, Every storage Connect after startup Tracker, inform ⾃⼰ Of group Etc , And keep it periodic ⼼ jump ,tracker according to storage Of ⼼ Jump information , build ⽴
group->[storage server list] Mapping table .
Tracker There is little meta information to manage , It's all stored in memory ; in addition tracker The meta information on is all from storage Reported information ⽣ Yes , You don't need to persist any data on your own , This makes tracker⾮ It's easy to expand , Directly increase tracker The machine can be extended to tracker cluster To serve ,cluster⾥ Every tracker There is complete equivalence between , be-all tracker All accepted stroage Of ⼼ Jump information ,⽣ Metadata information to provide read-write Services

client 

FastDFS To make ⽤ The provider provides basic ⽂ Piece access connection ⼝,⽐ Such as monitor、upload、download、append、delete etc. , In the client library ⽅ Provide to ⽤ Household emissary ⽤.

fastdfs The logic of uploading files

Client connection tracker： You can choose any number tracker One of them ,traker It's exactly equal to each other

traker The job of

One 、 Select assignment group The rules ：

1. Round robin, be-all group Inter polling
2. Specified group, Designate a ⼀ A definite group
3. Load balance, Choose the most ⼤ Remaining empty Group upload between ⽂ Pieces of
Two 、 Select assignment storage The rules of
1. Round robin, stay group In all of the storage Inter polling
2. First server ordered by ip, Press ip Sort
3. First server ordered by priority, Sort by priority （ The priority is storage On the configuration ）

3、 ... and 、 choice storage path The rules

Allocate data storage directory , Follow these principles

1. Round robin, Multiple storage ⽬ Record polling
2. The one with the most storage space left is preferred
Four 、 Generate field

Select storage ⽬ After recording ,storage Would be ⽂ Pieces of ⽣⼀ individual Fileid, from ：storage server ip、⽂ Piece creation time 、⽂ Pieces of ⼤⼩、⽂ Pieces of crc32 and ⼀ A random number ⽽ become , And then put this ⼆ Hexadecimal concatenation ⾏base64 code , Convert to printable string .

5、 ... and 、 Generate subdirectories

according to field Conduct hash, Route to the corresponding subdirectory , The file is then field The file name is stored in a subdirectory .

6、 ... and 、⽣ become ⽂ Piece name
When ⽂ Pieces are stored in a ⼦⽬ After recording , That is to say, we should ⽂ Pieces stored successfully , Next, we'll talk about this ⽂ Pieces of ⽣ become ⼀ individual ⽂ Piece name ,⽂ The piece name is from ：group、 Storage ⽬ record 、 Level two ⼦⽬ record 、fileid、⽂ Piece suffix （ Specified by client , The main ⽤ To distinguish ⽂ Piece type ） Splicing ⽽ become .

fastdfs The logic of downloading files

tracke from ⽂ The file name resolves ⽂ Piece group、⼤⼩、 Creation time and other information , Then select ⼀ individual storage⽤ To serve read requests .group Internal storage Weak consistency in synchronization , To avoid access to unsynchronized storage, The following rules

 1. The ⽂ Upload to the source storage - The source of the storage As long as you live , It must contain this ⽂ Pieces of , The address of the source is encoded in ⽂ In the piece name .
2. ⽂ Piece creation timestamp ==storage Time stamp synchronized to And ( current time -⽂ Piece creation timestamp ) > ⽂ Pieces are synchronized most ⼤ Time （ Such as 5 minute ) - ⽂ After the piece is created , Think that after the most ⼤ After the synchronization time , Must have synced to something else storage 了 .
3. ⽂ Piece creation timestamp < storage Time stamp synchronized to . - Before synchronizing the timestamp ⽂ Pieces are sure to be synchronized
4. ( current time -⽂ Piece creation timestamp ) > Synchronization delay threshold （ Such as ⼀ God ）. - After synchronization delay threshold time , Think ⽂ Pieces must have been synchronized