nifi ListSFTP Wait for proxy settings

Preface

Use nifi When collecting in large quantities , In the protection of the cluster, generally, only the interface function can communicate with the data source provided by the other party .
At this time nifi Cluster collection is likely nifi Network interworking cannot be carried out directly , You need to use an agent To connect to the other server to collect data .

about ListSFTP The essence of , You can see ：ListSFTP Work:

see nifi Collection configuration

As shown in the figure below ,nifi There are three proxy settings for collection , I recommend using Proxy Configuration Service To configure （ The reason is that you can set the verified account password , Not the rest .）

nifi Agent optional HTTP HTTPS and socks5 Acting as agent .
socks5 and https/http What's the difference , It's very simple , We act as agents, such as web agents https/http Different protocols are adopted , Similar agreements include ftp/sftp wait .
So to put it simply socks Namely socks The agreement ,https Namely https The agreement . You can use whatever agreement you want
Here we use socks5 To represent SFTP agreement （ You can see the agreement Big man's explanation socks)

Environmental preparation

nifi colony
Proxy node AAA
Data nodes BBB

Set up in the proxy node socks5 service

Because most company servers are not connected to the Internet , Therefore, an offline installation package is prepared for everyone to use .
Resources to address , Such as 404 Please comment on
ss5 Official address

Upload rpm.tar.gz ss5-3.8.9-8.tar.gz To the proxy node AAA Of /data1/app/ Under the table of contents

rpm.tar.gz by ss5 Related dependence of , contain

automake-1.13.4-3.el7.noarch.rpm
gcc-c+±4.8.5-44.el7.x86_64.rpm
openssl-devel-1.0.2k-21.el7_9.x86_64.rpm
cyrus-sasl-devel-2.1.26-23.el7.x86_64.rpm
make-3.82-24.el7.x86_64.rpm
pam-devel-1.1.8-23.el7.x86_64.rpm
gcc-4.8.5-44.el7.x86_64.rpm
openldap-devel-2.4.44-22.el7.x86_64.rpm

ss5-3.8.9-8.tar.gz by ss5 Latest installation package

Installation dependency

Unzip two tar.gz

cd /data1/app/
tar -zxcf rpm.tar.gz
tar -zxcf ss5-3.8.9-8.tar.gz

install ss5 rely on

cd rpm; rpm -Uvh --force --nodeps *.rpm;

install ss5

Entry directory , Use ss5 Built in check command to generate make Documents required .

cd ss5-3.8.9;
./configure

If the execution is successful, it will display

Failure display
If it fails, the dependency is not installed , Or the installation version is wrong .

go back to ss5-3.8.9 The catalog goes on make operation

cd /data1/app/ss5-3.8.9
make
make install #  There is no output 

Screenshot after completion

ss5 To configure

Follow start （ Start when the system starts ss5）

    chmod +x /etc/init.d/ss5
    chkconfig --add ss5
    chkconfig --level 345 ss5 on

Configure connectable ip And port

vim /etc/opt/ss5/ss5.conf
# find the line, and change it 
auth      0.0.0.0/0       -         u
permit  u        0.0.0.0/0       -       0.0.0.0/0       -       -       -       -       -

Then create ss5 Users of

vim /etc/opt/ss5/ss5.passwd
# add to   Space separated key value pairs 
 user name   password 
userName password

Restart the server

 service ss5 restart

If something goes wrong Permission denied

That is to ignore the above steps of forgetting to add permissions .

SS5 verification

If there is QQ perhaps TIM, That's easy Use QQ Login interface SS5 The agent can verify , Such as

Click test to view SS5 The success of .

nifi Agent configuration

Create a nifi ListFtp

Click on “→” Conduct Proxy Configuration Service Configuration of


Then click OK , Click on “able” Button on . Then start listSFTP, Pay attention when using listSFTP and FetchSFTP when , Both have to be configured Proxy Configuration Service The operation of .

follow-up

If there is a problem , Please ask questions in the comment area .