原创

Spring Boot集成SFTP快速入门Demo

1.什么是SFTP?

SFTP(SSH File Transfer Protocol,也称 Secret File Transfer Protocol),是一种基于SSH(安全外壳)的安全的文件传输协议。使用SFTP协议可以在文件传输过程中提供一种安全的加密算法,从而保证数据的安全传输,所以SFTP是非常安全的。但是,由于这种传输方式使用了加密/解密技术,所以传输效率比普通的FTP要低。 SFTP是SSH的一部分,SFTP没有单独的守护进程,它必须使用SSHD守护进程(端口号默认是22)来完成相应的连接操作,sftp服务作为ssh的一个子服务,是通过 /etc/ssh/sshd_config 配置文件中的 Subsystem 实现的,如果没有配置 Subsystem 参数,则系统是不能进行sftp访问的。所以,要分离ssh和sftp服务的话,基本的思路是创建两个sshd进程,分别监听在不同的端口,一个作为ssh服务的deamon,另一个作为sftp服务的deamon。

Spring Integration核心组件

  • SftpSessionFactory: sftp 客户端与服务端的会话工厂。客户端每次访问服务器时都会创建一个 session 对象,且可以通过 SftpSessionCaching 将 session 对象缓存起来,支持 session 共享,即可以在一个会话上进行多个 channel 的操作。如果 session 被重置,则在最后一次 channel 关闭之后,将断开连接。isSharedSession 为 true 时 session 将共享。
  • SftpSessionCaching: sftp 会话缓存工厂。通过 poolSize 和 sessionWaiteTimeout 来设置缓存池大小和会话等待超时时间。缓存池默认是无限大,超时时间默认是 Integer.MAX_VALUE。
  • SftpRemoteFileTemplate: 基于 SftpSessionFactory 创建的 sftp 文件操作模板类。其父类是 RemoteFileTemplate。支持上传、下载、追加、删除、重命名、列表、是否存在。基于输入输出流实现。
  • SftpInboundChannelAdapter: sftp 入站通道适配器。可同步远程目录到本地,且可监听远程文件的操作,可实现下载。
  • SftpOutboundChannelAdapter: sftp 出站通道适配器。实际是一个 sftp 消息处理器,将在服务器与客户端之间创建一个消息传输通道。此处的消息指的是 Message 的 payload,其支持 File、byte[]、String。其支持 ls、nlst、get、rm、mget、mv、put、mput 操作。
  • Channel Adapter: 通道适配器,实际上是适配消息在客户端和服务器之间的传输。inbound adapter 是接收其它系统的消息,outbound adapter 是发送消息到其它系统。
  • @ServiceActivator: 将注解作用的方法注册为处理消息的站点,inputChannel 表示接收消息的通道。

2.环境搭建

docker run -p 22:22 -d atmoz/sftp foo:pass:::upload

验证环境

11说明已经可以连接上去了

3.代码工程

实验目标

实现文件上传和下载

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>springboot-demo</artifactId>
        <groupId>com.et</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>sftp</artifactId>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-autoconfigure</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-sftp</artifactId>
            <!--            <version>5.4.1</version>-->
            <version>5.2.8.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>
</project>

service

package com.et.sftp.service.impl;

import com.et.sftp.config.SftpConfiguration;
import com.et.sftp.service.SftpService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.integration.file.remote.FileInfo;
import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;

import javax.annotation.Resource;
import java.io.*;
import java.util.List;


@Slf4j
@Component
public class SftpServiceImpl implements SftpService {

    @Resource
    private SftpRemoteFileTemplate remoteFileTemplate;

    @Resource
    private SftpConfiguration.SftpGateway gateway;

    /**
     * single file upload
     *
     * @param file File
     */
    @Override
    public void uploadFile(File file) {
        gateway.upload(file);
    }

    /**
     * single file upload by byte[]
     *
     * @param bytes bytes
     */
    @Override
    public void uploadFile(byte[] bytes, String name) {
        try {
            gateway.upload(bytes, name);
        } catch (Exception e) {
            log.error("error:", e);
        }
    }

    /**
     * uopload by path
     *
     * @param bytes
     * @param filename
     * @param path
     */
    @Override
    public void upload(byte[] bytes, String filename, String path) {
        try {
            gateway.upload(bytes, filename, path);
        } catch (Exception e) {
            log.error("error:", e);
        }
    }

    /**
     * list files by path
     *
     * @param path
     * @return List<String>
     */
    @Override
    public String[] listFile(String path) {
        try {
            return remoteFileTemplate.execute(session -> {
                return session.listNames(path);
            });
        } catch (Exception e) {
            log.error("error:", e);
        }
        return null;
    }


    /**
     * list file and directory by path
     *
     * @param path
     * @return List<String>
     */
    @Override
    public List<FileInfo> listALLFile(String path) {
        return gateway.listFile(path);
    }

    /**
     * download
     *
     * @param fileName 
     * @param savePath 
     * @return File
     */
    @Override
    public File downloadFile(String fileName, String savePath) {
        try {
            return remoteFileTemplate.execute(session -> {
                remoteFileTemplate.setAutoCreateDirectory(true);
                boolean existFile = session.exists(fileName);
                if (existFile) {
                    InputStream is = session.readRaw(fileName);
                    return convertInputStreamToFile(is, savePath);
                } else {
                    return null;
                }
            });
        } catch (Exception e) {
            log.error("error:", e);
        }
        return null;
    }

    /**
     * read file
     *
     * @param fileName
     * @return InputStream
     */

    @Override
    public InputStream readFile(String fileName) {
        return remoteFileTemplate.execute(session -> {
            return session.readRaw(fileName);
        });
    }

    /**
     * files is exists
     *
     * @param filePath 
     * @return boolean
     */
    @Override
    public boolean existFile(String filePath) {
        try {
            return remoteFileTemplate.execute(session ->
                    session.exists(filePath));
        } catch (Exception e) {
            log.error("error:", e);
        }
        return false;
    }

    public void renameFile(String file1, String file2) {
        try {
            remoteFileTemplate.execute(session -> {
                session.rename(file1, file2);
                return true;
            });
        } catch (Exception e) {
            log.error("error:", e);
        }
    }

    /**
     * create directory
     *
     * @param dirName
     * @return
     */

    @Override
    public boolean mkdir(String dirName) {
        return remoteFileTemplate.execute(session -> {
            if (!existFile(dirName)) {

                return session.mkdir(dirName);
            } else {
                return false;
            }
        });
    }

    /**
     * delete file
     *
     * @param fileName 
     * @return boolean
     */
    @Override
    public boolean deleteFile(String fileName) {
        return remoteFileTemplate.execute(session -> {
            boolean existFile = session.exists(fileName);
            if (existFile) {
                return session.remove(fileName);
            } else {
                log.info("file : {} not exist", fileName);
                return false;
            }
        });
    }

    /**
     * batch upload (MultipartFile)
     *
     * @param files List<MultipartFile>
     * @throws IOException
     */
    @Override
    public void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException {
        try {
            for (MultipartFile multipartFile : files) {
                if (multipartFile.isEmpty()) {
                    continue;
                }
                File file = convert(multipartFile);
                gateway.upload(file);
                if (deleteSource) {
                    file.delete();
                }
            }
        } catch (Exception e) {
            log.error("error:", e);
        }
    }

    /**
     * batch upload (MultipartFile)
     *
     * @param files List<MultipartFile>
     * @throws IOException
     */
    @Override
    public void uploadFiles(List<MultipartFile> files) throws IOException {
        uploadFiles(files, true);
    }

    /**
     * single file upload (MultipartFile)
     *
     * @param multipartFile MultipartFile
     * @throws IOException
     */
    @Override
    public void uploadFile(MultipartFile multipartFile) throws IOException {
        gateway.upload(convert(multipartFile));
    }

    @Override
    public String listFileNames(String dir) {
        return gateway.nlstFile(dir);
    }

    @Override
    public File getFile(String dir) {
        return null;
    }

    @Override
    public List<File> mgetFile(String dir) {
        return null;
    }

    @Override
    public boolean rmFile(String file) {
        return false;
    }

    @Override
    public boolean mv(String sourceFile, String targetFile) {
        return false;
    }

    @Override
    public File putFile(String dir) {
        return null;
    }

    @Override
    public List<File> mputFile(String dir) {
        return null;
    }

    @Override
    public String nlstFile(String dir) {
        return gateway.nlstFile(dir);
    }

    private static File convertInputStreamToFile(InputStream inputStream, String savePath) {
        OutputStream outputStream = null;
        File file = new File(savePath);
        try {
            outputStream = new FileOutputStream(file);
            int read;
            byte[] bytes = new byte[1024];
            while ((read = inputStream.read(bytes)) != -1) {
                outputStream.write(bytes, 0, read);
            }
            log.info("convert InputStream to file done, savePath is : {}", savePath);
        } catch (IOException e) {
            log.error("error:", e);
        } finally {
            if (inputStream != null) {
                try {
                    inputStream.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            if (outputStream != null) {
                try {
                    outputStream.close();
                } catch (IOException e) {
                    log.error("error:", e);
                }
            }
        }
        return file;
    }

    private static File convert(MultipartFile file) throws IOException {
        File convertFile = new File(file.getOriginalFilename());
        convertFile.createNewFile();
        FileOutputStream fos = new FileOutputStream(convertFile);
        fos.write(file.getBytes());
        fos.close();
        return convertFile;
    }
}
package com.et.sftp.service;

import org.springframework.integration.file.remote.FileInfo;
import org.springframework.web.multipart.MultipartFile;

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;


public interface SftpService {


    void uploadFile(File file);

    void uploadFile(byte[] bytes, String name);

    
    void upload(byte[] bytes, String filename, String path);

    
    String[] listFile(String path);

    List<FileInfo> listALLFile(String path);

    
    File downloadFile(String fileName, String savePath);


    InputStream readFile(String fileName);

    
    boolean existFile(String filePath);


    boolean mkdir(String dirName);

    boolean deleteFile(String fileName);

    
    void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException;

    
    void uploadFiles(List<MultipartFile> files) throws IOException;

    
    void uploadFile(MultipartFile multipartFile) throws IOException;


   
    String listFileNames(String dir);


    File getFile(String dir);

   
    List<File> mgetFile(String dir);

  
    boolean rmFile(String file);

    
    boolean mv(String sourceFile, String targetFile);

   
    File putFile(String dir);

    
    List<File> mputFile(String dir);


    //void upload(File file);

    //void upload(byte[] inputStream, String name);

    //List<File> downloadFiles(String dir);


    String nlstFile(String dir);
}

config

在配置SFTP adapters之前,需要配置SFTP Session Factory;Spring Integration提供了如下xml和spring boot的定义方式。 每次使用 SFTP adapter,都需要Session Factory会话对象,一般情况,都会创建一个新的SFTP会话。同时还提供了Session的缓存功能。Spring integration中的Session Factory是依赖于JSch库来提供。 JSch支持在一个连接配置上多个channel的操作。原生的JSch技术开发,在打开一个channel操作之前,需要建立Session的连接。同样的,默认情况,Spring Integration为每一个channel操作使用单独的物理连接。在3.0版本发布之后,Cache Session Factory 出现 (CachingSessionFactory),将Session Factory包装在缓存中,支持Session共享,可以在一个连接上支持多个JSch Channel的操作。如果缓存被重置,在最后一次channel关闭之后,才会断开连接。
package com.et.sftp.config;

import com.jcraft.jsch.ChannelSftp;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.expression.common.LiteralExpression;
import org.springframework.expression.spel.standard.SpelExpressionParser;
import org.springframework.integration.annotation.Gateway;
import org.springframework.integration.annotation.MessagingGateway;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.core.MessagingTemplate;
import org.springframework.integration.file.FileNameGenerator;
import org.springframework.integration.file.remote.FileInfo;
import org.springframework.integration.file.remote.session.CachingSessionFactory;
import org.springframework.integration.file.remote.session.SessionFactory;
import org.springframework.integration.file.support.FileExistsMode;
import org.springframework.integration.sftp.gateway.SftpOutboundGateway;
import org.springframework.integration.sftp.outbound.SftpMessageHandler;
import org.springframework.integration.sftp.session.DefaultSftpSessionFactory;
import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.messaging.handler.annotation.Payload;

import javax.annotation.Resource;
import java.io.File;
import java.util.List;



@Configuration
@EnableConfigurationProperties(SftpProperties.class)
public class SftpConfiguration {

    @Resource
    private SftpProperties properties;


    @Bean
    public MessagingTemplate messagingTemplate(BeanFactory beanFactory) {
        MessagingTemplate messagingTemplate = new MessagingTemplate();
        messagingTemplate.setBeanFactory(beanFactory);
        return messagingTemplate;
    }

    @Bean
    public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
        DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
        factory.setHost(properties.getHost());
        factory.setPort(properties.getPort());
        factory.setUser(properties.getUsername());
        factory.setPassword(properties.getPassword());
        factory.setAllowUnknownKeys(true);
//        factory.setTestSession(true);
//        return factory;
        return new CachingSessionFactory<ChannelSftp.LsEntry>(factory);
    }

    @Bean
    public SftpRemoteFileTemplate remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        return new SftpRemoteFileTemplate(sftpSessionFactory);
    }


    @Bean
    @ServiceActivator(inputChannel = "downloadChannel")
    public MessageHandler downloadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "mget", "payload");
        sftpOutboundGateway.setOptions("-R");
        sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
        sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));
        sftpOutboundGateway.setAutoCreateLocalDirectory(true);
        return sftpOutboundGateway;
    }


    @Bean
    @ServiceActivator(inputChannel = "uploadChannel", outputChannel = "testChannel")
    public MessageHandler uploadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
        handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
//        handler.setChmod();
        handler.setFileNameGenerator(message -> {
            if (message.getPayload() instanceof File) {
                return ((File) message.getPayload()).getName();
            } else {
                throw new IllegalArgumentException("File expected as payload.");
            }
        });
        return handler;
    }

    @Bean
    @ServiceActivator(inputChannel = "uploadByteChannel")
    public MessageHandler multiTypeHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
        handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
        handler.setFileNameGenerator(message -> {
            if (message.getPayload() instanceof byte[]) {
                return (String) message.getHeaders().get("name");
            } else {
                throw new IllegalArgumentException("byte[] expected as payload.");
            }
        });
        return handler;
    }


    @Bean
    @ServiceActivator(inputChannel = "lsChannel")
    public MessageHandler lsHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "ls", "payload");
        sftpOutboundGateway.setOptions("-R"); 
        return sftpOutboundGateway;
    }


    @Bean
    @ServiceActivator(inputChannel = "nlstChannel")
    public MessageHandler listFileNamesHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "nlst", "payload");
        return sftpOutboundGateway;
    }

    @Bean
    @ServiceActivator(inputChannel = "getChannel")
    public MessageHandler getFileHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "get", "payload");
        sftpOutboundGateway.setOptions("-R");
        sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
        sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));
        sftpOutboundGateway.setAutoCreateLocalDirectory(true);
        return sftpOutboundGateway;
    }

    /**
     * create by: qiushicai
     * create time: 2020/11/20
     *
     * @return
     */
    @Bean
    @ServiceActivator(inputChannel = "abc")
    public MessageHandler abcHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
        SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
        handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
        handler.setFileNameGenerator(message -> {
            if (message.getPayload() instanceof byte[]) {
                System.out.println("receive message:" + new String((byte[]) message.getPayload()));
                message.getHeaders().forEach((k, v) -> System.out.println("\t\t|---" + k + "=" + v));
                return "ok";
            } else {
                throw new IllegalArgumentException("byte[] expected as payload.");
            }
        });
        return handler;
    }

    /**
     *
     *  the #root object is the Message, which has two properties (headers and payload) that allow such expressions as payload, payload.thing, headers['my.header'], and so on
     *
     *  link{ https://stackoverflow.com/questions/46650004/spring-integration-ftp-create-dynamic-directory-with-remote-directory-expressi}
     *  link{ https://docs.spring.io/spring-integration/reference/html/spel.html}
     * @return
     */
    @Bean
    @ServiceActivator(inputChannel = "toPathChannel")
    public MessageHandler pathHandler() {
        SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
        // automatically create the remote directory
        handler.setAutoCreateDirectory(true);
        handler.setRemoteDirectoryExpression(new SpelExpressionParser().parseExpression("headers[path]"));
        handler.setFileNameGenerator(new FileNameGenerator() {
            @Override
            public String generateFileName(Message<?> message) {
                return (String) message.getHeaders().get("filename");
            }
        });
        return handler;
    }


    /**
     * <ul>
     * <li>ls (list files)
     * <li> nlst (list file names)
     * <li> get (retrieve a file)
     * <li> mget (retrieve multiple files)
     * <li> rm (remove file(s))
     * <li> mv (move and rename file)
     * <li> put (send a file)
     * <li> mput (send multiple files)
     * </ul>
     *
     * @author :qiushicai
     * @date :Created in 2020/11/20
     * @description: outbound gateway API
     * @version:
     */
    @MessagingGateway
    public interface SftpGateway {

        //ls (list files)
        @Gateway(requestChannel = "lsChannel")
        List<FileInfo> listFile(String dir);


        @Gateway(requestChannel = "nlstChannel")
        String nlstFile(String dir);



        @Gateway(requestChannel = "getChannel")
        File getFile(String dir);

        @Gateway(requestChannel = "mgetChannel")
        List<File> mgetFile(String dir);

        @Gateway(replyChannel = "rmChannel")
        boolean rmFile(String file);

        @Gateway(replyChannel = "mvChannel")
        boolean mv(String sourceFile, String targetFile);

        @Gateway(requestChannel = "putChannel")
        File putFile(String dir);

        @Gateway(requestChannel = "mputChannel")
        List<File> mputFile(String dir);

        @Gateway(requestChannel = "uploadChannel")
        void upload(File file);

        @Gateway(requestChannel = "uploadByteChannel")
        void upload(byte[] inputStream, String name);

        @Gateway(requestChannel = "toPathChannel")
        void upload(@Payload byte[] file, @Header("filename") String filename, @Header("path") String path);

        @Gateway(requestChannel = "downloadChannel")
        List<File> downloadFiles(String dir);

    }
}
从Spring Integration 3.0开始,通过SftpSession对象提供了一个新类Remote File Template。提供了Sftp文件发送、检索、删除和重命名文件的方法。此外,还提供了一个执行方法,允许调用者在会话上执行多个操作。在所有情况下,模板负责可靠地关闭会话。 SftpRemoteFileTemplate继承Remote File Template,可以很容易的实现对SFTP文件的发送(包括文件追加,替换等),检索,删除和重命名。
package com.et.sftp.config;


import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;

@Data
@ConfigurationProperties(prefix = "sftp")
public class SftpProperties {
    private String host;
    private Integer port;
    private String password;
    private String username;
    private String remoteDir;
    private String localDir;
}

application.properties

##sftp properties
sftp.host=127.0.0.1
sftp.port=22
sftp.username=foo
sftp.password=pass
sftp.remoteDir=/upload
sftp.localDir=D:\\tmp\\sync-files

4.测试

在测试包里面有专门的测试类,具体可以查看源代码

文件是否存在

@Test
void testExistFile() {
    boolean existFile = sftpService.existFile("/upload/home222.js");
    System.out.println(existFile);
}

列出目录下的文件

@Test
void listFileTest() {
    sftpService.listALLFile("/upload").stream().forEach(System.out::println);
}

下载文件

 @Test
    void testDownLoad() throws Exception {
        sftpService.downloadFile("/upload/home.js", "D:\\tmp\\c222c.js");
//
//        sftpService.uploadFile(new File("D:\\tmp\\cc.js"));


//        InputStream inputStream = sftpService.readFile("/upload/cc.js");
//
//        IOUtils.copy(inputStream, new FileOutputStream(new File("D:\\tmp\\" + UUID.randomUUID() + ".js")));

    }

上传文件

@Test
void uploadFile() {
    sftpService.uploadFile(new File("D:\\tmp\\cc.js"));
}

5.引用

正文到此结束
Loading...