Spring Boot集成SFTP快速入门Demo

1.什么是SFTP？

SFTP（SSH File Transfer Protocol，也称 Secret File Transfer Protocol），是一种基于SSH（安全外壳）的安全的文件传输协议。使用SFTP协议可以在文件传输过程中提供一种安全的加密算法，从而保证数据的安全传输，所以SFTP是非常安全的。但是，由于这种传输方式使用了加密/解密技术，所以传输效率比普通的FTP要低。 SFTP是SSH的一部分，SFTP没有单独的守护进程，它必须使用SSHD守护进程（端口号默认是22）来完成相应的连接操作，sftp服务作为ssh的一个子服务，是通过 /etc/ssh/sshd_config 配置文件中的 Subsystem 实现的，如果没有配置 Subsystem 参数，则系统是不能进行sftp访问的。所以，要分离ssh和sftp服务的话，基本的思路是创建两个sshd进程，分别监听在不同的端口，一个作为ssh服务的deamon，另一个作为sftp服务的deamon。

Spring Integration核心组件

SftpSessionFactory: sftp 客户端与服务端的会话工厂。客户端每次访问服务器时都会创建一个 session 对象，且可以通过 SftpSessionCaching 将 session 对象缓存起来，支持 session 共享，即可以在一个会话上进行多个 channel 的操作。如果 session 被重置，则在最后一次 channel 关闭之后，将断开连接。isSharedSession 为 true 时 session 将共享。
SftpSessionCaching: sftp 会话缓存工厂。通过 poolSize 和 sessionWaiteTimeout 来设置缓存池大小和会话等待超时时间。缓存池默认是无限大，超时时间默认是 Integer.MAX_VALUE。
SftpRemoteFileTemplate: 基于 SftpSessionFactory 创建的 sftp 文件操作模板类。其父类是 RemoteFileTemplate。支持上传、下载、追加、删除、重命名、列表、是否存在。基于输入输出流实现。
SftpInboundChannelAdapter: sftp 入站通道适配器。可同步远程目录到本地，且可监听远程文件的操作，可实现下载。
SftpOutboundChannelAdapter: sftp 出站通道适配器。实际是一个 sftp 消息处理器，将在服务器与客户端之间创建一个消息传输通道。此处的消息指的是 Message 的 payload，其支持 File、byte[]、String。其支持 ls、nlst、get、rm、mget、mv、put、mput 操作。
Channel Adapter: 通道适配器，实际上是适配消息在客户端和服务器之间的传输。inbound adapter 是接收其它系统的消息，outbound adapter 是发送消息到其它系统。
@ServiceActivator: 将注解作用的方法注册为处理消息的站点，inputChannel 表示接收消息的通道。

2.环境搭建

docker run -p 22:22 -d atmoz/sftp foo:pass:::upload

验证环境

说明已经可以连接上去了

3.代码工程

实验目标

实现文件上传和下载

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><parent><artifactId>springboot-demo</artifactId><groupId>com.et</groupId><version>1.0-SNAPSHOT</version></parent><modelVersion>4.0.0</modelVersion><artifactId>sftp</artifactId><properties><maven.compiler.source>8</maven.compiler.source><maven.compiler.target>8</maven.compiler.target></properties><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-autoconfigure</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency><dependency><groupId>org.springframework.integration</groupId><artifactId>spring-integration-sftp</artifactId><!--            <version>5.4.1</version>--><version>5.2.8.RELEASE</version></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-configuration-processor</artifactId><optional>true</optional></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><optional>true</optional></dependency></dependencies>
</project>

service

package com.et.sftp.service.impl;import com.et.sftp.config.SftpConfiguration;
import com.et.sftp.service.SftpService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.integration.file.remote.FileInfo;
import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;import javax.annotation.Resource;
import java.io.*;
import java.util.List;@Slf4j
@Component
public class SftpServiceImpl implements SftpService {@Resourceprivate SftpRemoteFileTemplate remoteFileTemplate;@Resourceprivate SftpConfiguration.SftpGateway gateway;/*** single file upload** @param file File*/@Overridepublic void uploadFile(File file) {gateway.upload(file);}/*** single file upload by byte[]** @param bytes bytes*/@Overridepublic void uploadFile(byte[] bytes, String name) {try {gateway.upload(bytes, name);} catch (Exception e) {log.error("error:", e);}}/*** uopload by path** @param bytes* @param filename* @param path*/@Overridepublic void upload(byte[] bytes, String filename, String path) {try {gateway.upload(bytes, filename, path);} catch (Exception e) {log.error("error:", e);}}/*** list files by path** @param path* @return List<String>*/@Overridepublic String[] listFile(String path) {try {return remoteFileTemplate.execute(session -> {return session.listNames(path);});} catch (Exception e) {log.error("error:", e);}return null;}/*** list file and directory by path** @param path* @return List<String>*/@Overridepublic List<FileInfo> listALLFile(String path) {return gateway.listFile(path);}/*** download** @param fileName * @param savePath * @return File*/@Overridepublic File downloadFile(String fileName, String savePath) {try {return remoteFileTemplate.execute(session -> {remoteFileTemplate.setAutoCreateDirectory(true);boolean existFile = session.exists(fileName);if (existFile) {InputStream is = session.readRaw(fileName);return convertInputStreamToFile(is, savePath);} else {return null;}});} catch (Exception e) {log.error("error:", e);}return null;}/*** read file** @param fileName* @return InputStream*/@Overridepublic InputStream readFile(String fileName) {return remoteFileTemplate.execute(session -> {return session.readRaw(fileName);});}/*** files is exists** @param filePath * @return boolean*/@Overridepublic boolean existFile(String filePath) {try {return remoteFileTemplate.execute(session ->session.exists(filePath));} catch (Exception e) {log.error("error:", e);}return false;}public void renameFile(String file1, String file2) {try {remoteFileTemplate.execute(session -> {session.rename(file1, file2);return true;});} catch (Exception e) {log.error("error:", e);}}/*** create directory** @param dirName* @return*/@Overridepublic boolean mkdir(String dirName) {return remoteFileTemplate.execute(session -> {if (!existFile(dirName)) {return session.mkdir(dirName);} else {return false;}});}/*** delete file** @param fileName * @return boolean*/@Overridepublic boolean deleteFile(String fileName) {return remoteFileTemplate.execute(session -> {boolean existFile = session.exists(fileName);if (existFile) {return session.remove(fileName);} else {log.info("file : {} not exist", fileName);return false;}});}/*** batch upload (MultipartFile)** @param files List<MultipartFile>* @throws IOException*/@Overridepublic void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException {try {for (MultipartFile multipartFile : files) {if (multipartFile.isEmpty()) {continue;}File file = convert(multipartFile);gateway.upload(file);if (deleteSource) {file.delete();}}} catch (Exception e) {log.error("error:", e);}}/*** batch upload (MultipartFile)** @param files List<MultipartFile>* @throws IOException*/@Overridepublic void uploadFiles(List<MultipartFile> files) throws IOException {uploadFiles(files, true);}/*** single file upload (MultipartFile)** @param multipartFile MultipartFile* @throws IOException*/@Overridepublic void uploadFile(MultipartFile multipartFile) throws IOException {gateway.upload(convert(multipartFile));}@Overridepublic String listFileNames(String dir) {return gateway.nlstFile(dir);}@Overridepublic File getFile(String dir) {return null;}@Overridepublic List<File> mgetFile(String dir) {return null;}@Overridepublic boolean rmFile(String file) {return false;}@Overridepublic boolean mv(String sourceFile, String targetFile) {return false;}@Overridepublic File putFile(String dir) {return null;}@Overridepublic List<File> mputFile(String dir) {return null;}@Overridepublic String nlstFile(String dir) {return gateway.nlstFile(dir);}private static File convertInputStreamToFile(InputStream inputStream, String savePath) {OutputStream outputStream = null;File file = new File(savePath);try {outputStream = new FileOutputStream(file);int read;byte[] bytes = new byte[1024];while ((read = inputStream.read(bytes)) != -1) {outputStream.write(bytes, 0, read);}log.info("convert InputStream to file done, savePath is : {}", savePath);} catch (IOException e) {log.error("error:", e);} finally {if (inputStream != null) {try {inputStream.close();} catch (IOException e) {e.printStackTrace();}}if (outputStream != null) {try {outputStream.close();} catch (IOException e) {log.error("error:", e);}}}return file;}private static File convert(MultipartFile file) throws IOException {File convertFile = new File(file.getOriginalFilename());convertFile.createNewFile();FileOutputStream fos = new FileOutputStream(convertFile);fos.write(file.getBytes());fos.close();return convertFile;}
}

package com.et.sftp.service;import org.springframework.integration.file.remote.FileInfo;
import org.springframework.web.multipart.MultipartFile;import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;public interface SftpService {void uploadFile(File file);void uploadFile(byte[] bytes, String name);void upload(byte[] bytes, String filename, String path);String[] listFile(String path);List<FileInfo> listALLFile(String path);File downloadFile(String fileName, String savePath);InputStream readFile(String fileName);boolean existFile(String filePath);boolean mkdir(String dirName);boolean deleteFile(String fileName);void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException;void uploadFiles(List<MultipartFile> files) throws IOException;void uploadFile(MultipartFile multipartFile) throws IOException;String listFileNames(String dir);File getFile(String dir);List<File> mgetFile(String dir);boolean rmFile(String file);boolean mv(String sourceFile, String targetFile);File putFile(String dir);List<File> mputFile(String dir);//void upload(File file);//void upload(byte[] inputStream, String name);//List<File> downloadFiles(String dir);String nlstFile(String dir);
}

config

在配置SFTP adapters之前，需要配置SFTP Session Factory；Spring Integration提供了如下xml和spring boot的定义方式。每次使用 SFTP adapter，都需要Session Factory会话对象，一般情况，都会创建一个新的SFTP会话。同时还提供了Session的缓存功能。Spring integration中的Session Factory是依赖于JSch库来提供。 JSch支持在一个连接配置上多个channel的操作。原生的JSch技术开发，在打开一个channel操作之前，需要建立Session的连接。同样的，默认情况，Spring Integration为每一个channel操作使用单独的物理连接。在3.0版本发布之后，Cache Session Factory 出现 (CachingSessionFactory)，将Session Factory包装在缓存中，支持Session共享，可以在一个连接上支持多个JSch Channel的操作。如果缓存被重置，在最后一次channel关闭之后，才会断开连接。

package com.et.sftp.config;import com.jcraft.jsch.ChannelSftp;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.expression.common.LiteralExpression;
import org.springframework.expression.spel.standard.SpelExpressionParser;
import org.springframework.integration.annotation.Gateway;
import org.springframework.integration.annotation.MessagingGateway;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.core.MessagingTemplate;
import org.springframework.integration.file.FileNameGenerator;
import org.springframework.integration.file.remote.FileInfo;
import org.springframework.integration.file.remote.session.CachingSessionFactory;
import org.springframework.integration.file.remote.session.SessionFactory;
import org.springframework.integration.file.support.FileExistsMode;
import org.springframework.integration.sftp.gateway.SftpOutboundGateway;
import org.springframework.integration.sftp.outbound.SftpMessageHandler;
import org.springframework.integration.sftp.session.DefaultSftpSessionFactory;
import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.messaging.handler.annotation.Payload;import javax.annotation.Resource;
import java.io.File;
import java.util.List;@Configuration
@EnableConfigurationProperties(SftpProperties.class)
public class SftpConfiguration {@Resourceprivate SftpProperties properties;@Beanpublic MessagingTemplate messagingTemplate(BeanFactory beanFactory) {MessagingTemplate messagingTemplate = new MessagingTemplate();messagingTemplate.setBeanFactory(beanFactory);return messagingTemplate;}@Beanpublic SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);factory.setHost(properties.getHost());factory.setPort(properties.getPort());factory.setUser(properties.getUsername());factory.setPassword(properties.getPassword());factory.setAllowUnknownKeys(true);
//        factory.setTestSession(true);
//        return factory;return new CachingSessionFactory<ChannelSftp.LsEntry>(factory);}@Beanpublic SftpRemoteFileTemplate remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {return new SftpRemoteFileTemplate(sftpSessionFactory);}@Bean@ServiceActivator(inputChannel = "downloadChannel")public MessageHandler downloadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "mget", "payload");sftpOutboundGateway.setOptions("-R");sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));sftpOutboundGateway.setAutoCreateLocalDirectory(true);return sftpOutboundGateway;}@Bean@ServiceActivator(inputChannel = "uploadChannel", outputChannel = "testChannel")public MessageHandler uploadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
//        handler.setChmod();handler.setFileNameGenerator(message -> {if (message.getPayload() instanceof File) {return ((File) message.getPayload()).getName();} else {throw new IllegalArgumentException("File expected as payload.");}});return handler;}@Bean@ServiceActivator(inputChannel = "uploadByteChannel")public MessageHandler multiTypeHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));handler.setFileNameGenerator(message -> {if (message.getPayload() instanceof byte[]) {return (String) message.getHeaders().get("name");} else {throw new IllegalArgumentException("byte[] expected as payload.");}});return handler;}@Bean@ServiceActivator(inputChannel = "lsChannel")public MessageHandler lsHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "ls", "payload");sftpOutboundGateway.setOptions("-R"); return sftpOutboundGateway;}@Bean@ServiceActivator(inputChannel = "nlstChannel")public MessageHandler listFileNamesHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "nlst", "payload");return sftpOutboundGateway;}@Bean@ServiceActivator(inputChannel = "getChannel")public MessageHandler getFileHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "get", "payload");sftpOutboundGateway.setOptions("-R");sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));sftpOutboundGateway.setAutoCreateLocalDirectory(true);return sftpOutboundGateway;}/*** create by: qiushicai* create time: 2020/11/20** @return*/@Bean@ServiceActivator(inputChannel = "abc")public MessageHandler abcHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));handler.setFileNameGenerator(message -> {if (message.getPayload() instanceof byte[]) {System.out.println("receive message:" + new String((byte[]) message.getPayload()));message.getHeaders().forEach((k, v) -> System.out.println("\t\t|---" + k + "=" + v));return "ok";} else {throw new IllegalArgumentException("byte[] expected as payload.");}});return handler;}/****  the #root object is the Message, which has two properties (headers and payload) that allow such expressions as payload, payload.thing, headers['my.header'], and so on**  link{ https://stackoverflow.com/questions/46650004/spring-integration-ftp-create-dynamic-directory-with-remote-directory-expressi}*  link{ https://docs.spring.io/spring-integration/reference/html/spel.html}* @return*/@Bean@ServiceActivator(inputChannel = "toPathChannel")public MessageHandler pathHandler() {SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());// automatically create the remote directoryhandler.setAutoCreateDirectory(true);handler.setRemoteDirectoryExpression(new SpelExpressionParser().parseExpression("headers[path]"));handler.setFileNameGenerator(new FileNameGenerator() {@Overridepublic String generateFileName(Message<?> message) {return (String) message.getHeaders().get("filename");}});return handler;}/*** <ul>* <li>ls (list files)* <li> nlst (list file names)* <li> get (retrieve a file)* <li> mget (retrieve multiple files)* <li> rm (remove file(s))* <li> mv (move and rename file)* <li> put (send a file)* <li> mput (send multiple files)* </ul>** @author ：qiushicai* @date ：Created in 2020/11/20* @description： outbound gateway API* @version:*/@MessagingGatewaypublic interface SftpGateway {//ls (list files)@Gateway(requestChannel = "lsChannel")List<FileInfo> listFile(String dir);@Gateway(requestChannel = "nlstChannel")String nlstFile(String dir);@Gateway(requestChannel = "getChannel")File getFile(String dir);@Gateway(requestChannel = "mgetChannel")List<File> mgetFile(String dir);@Gateway(replyChannel = "rmChannel")boolean rmFile(String file);@Gateway(replyChannel = "mvChannel")boolean mv(String sourceFile, String targetFile);@Gateway(requestChannel = "putChannel")File putFile(String dir);@Gateway(requestChannel = "mputChannel")List<File> mputFile(String dir);@Gateway(requestChannel = "uploadChannel")void upload(File file);@Gateway(requestChannel = "uploadByteChannel")void upload(byte[] inputStream, String name);@Gateway(requestChannel = "toPathChannel")void upload(@Payload byte[] file, @Header("filename") String filename, @Header("path") String path);@Gateway(requestChannel = "downloadChannel")List<File> downloadFiles(String dir);}
}

从Spring Integration 3.0开始，通过SftpSession对象提供了一个新类Remote File Template。提供了Sftp文件发送、检索、删除和重命名文件的方法。此外，还提供了一个执行方法，允许调用者在会话上执行多个操作。在所有情况下，模板负责可靠地关闭会话。 SftpRemoteFileTemplate继承Remote File Template,可以很容易的实现对SFTP文件的发送(包括文件追加，替换等)，检索，删除和重命名。

package com.et.sftp.config;import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;@Data
@ConfigurationProperties(prefix = "sftp")
public class SftpProperties {private String host;private Integer port;private String password;private String username;private String remoteDir;private String localDir;
}

`application.properties`

##sftp properties
sftp.host=127.0.0.1
sftp.port=22
sftp.username=foo
sftp.password=pass
sftp.remoteDir=/upload
sftp.localDir=D:\\tmp\\sync-files

以上只是一些关键代码，所有代码请参见下面代码仓库

代码仓库

https://github.com/Harries/springboot-demo

4.测试

在测试包里面有专门的测试类，具体可以查看源代码

文件是否存在

@Test
void testExistFile() {boolean existFile = sftpService.existFile("/upload/home222.js");System.out.println(existFile);
}

列出目录下的文件

@Test
void listFileTest() {sftpService.listALLFile("/upload").stream().forEach(System.out::println);
}

下载文件

 @Testvoid testDownLoad() throws Exception {sftpService.downloadFile("/upload/home.js", "D:\\tmp\\c222c.js");
//
//        sftpService.uploadFile(new File("D:\\tmp\\cc.js"));//        InputStream inputStream = sftpService.readFile("/upload/cc.js");
//
//        IOUtils.copy(inputStream, new FileOutputStream(new File("D:\\tmp\\" + UUID.randomUUID() + ".js")));}

上传文件

@Test
void uploadFile() {sftpService.uploadFile(new File("D:\\tmp\\cc.js"));
}

5.引用

SFTP Adapters
Spring Boot集成SFTP快速入门Demo | Harries Blog™