Spring Boot应用Shutdown的正确姿势

Spring Boot应用Shutdown的正确姿势

基于Spring Boot的应用，程序在发布新版本时，如何正确地停服升级呢？直接 kill -9 pid 肯定是不合适的，可能导致被处理的数据处在灰色的中间状态。

我们项目分两类应用，一种是基于Spring Boot的Web项目，涉及Servlet容器，一种是基于Spring Cloud Stream的Consumer应用，来监听消息队列处理event。下面我针对这两类应用，添加各自的gracefully shutdown机制。

版本说明

  • Spring Boot 2.0.3.RELEASE
  • Spring Cloud Stream 2.0.0.RELEASE

Web应用

此类应用在停服时，应该先终止Servlet容器响应新的请求，然后待已接收的请求处理完后，再停止服务。项目中使用的是内嵌Tomcat，使用 wilkinsona 的方案来实现。

/**
 * Gracefully shut down tomcat server
 *
 * Ref: https://dzone.com/articles/graceful-shutdown-spring-boot-applications
 */
@Configuration
public class WebConfig {

    @Bean
    public GracefulShutdown gracefulShutdown() {
        return new GracefulShutdown();
    }

    @Bean
    public ConfigurableServletWebServerFactory webServerFactory(GracefulShutdown gracefulShutdown) {
        TomcatServletWebServerFactory factory = new TomcatServletWebServerFactory();
        factory.addConnectorCustomizers(gracefulShutdown);
        return factory;
    }

}


public class GracefulShutdown implements ApplicationListener<ContextClosedEvent>, TomcatConnectorCustomizer {

    private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

    private volatile Connector connector;

    @Value("${app.shutdownTimeout}")
    int shutdownTimeout;

    @Override
    public void onApplicationEvent(ContextClosedEvent event) {
        this.connector.pause();

        Executor executor = this.connector.getProtocolHandler().getExecutor();

        if (executor instanceof ThreadPoolExecutor) {
            ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
            threadPoolExecutor.shutdown();

            try {
                if (threadPoolExecutor.awaitTermination(shutdownTimeout, TimeUnit.SECONDS)) {
                    log.warn("Tomcat thread pool did not shutdown gracefully within " + shutdownTimeout
                            + " seconds. Proceeding with forceful shutdown.");
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }
    }

    @Override
    public void customize(Connector connector) {
        this.connector = connector;
    }
}

Consumer应用

项目中使用Spring Cloud Stream来监听RabbitMQ的消息，配置监听比较方便，基本在yml中声明下，然后再 @EnableBinding @StreamListener 就完了，比如下面专门发邮件的例子：

spring:
  main:
    web-application-type: none
  cloud:
    stream:
      bindings:
        input:
          consumer:
            maxAttempts: 1
            concurrency: 5
            shutdownTimeout: 40000 # 添加consumer shutdown timeout
          destination: sendMailTopic
          content-type: application/json
          group: mailGroup

@EnableBinding(Sink.class)
public class MailHandler {
    private Logger log = LoggerFactory.getLogger(MailHandler.class);

    @Value("${spring.mail.username}")
    String sendFrom;

    @Value("${spring.profiles.active}")
    String activeProfile;

    @Autowired
    Tracer tracer;

    @Autowired
    JavaMailSender mailSender;

    @Autowired
    LogService logService;

    /**
     * Listen to sendMailTopic of RabbitMQ
     *
     * @param vo MailVO
     */
    @StreamListener(Sink.INPUT)
    public void sendMail(MailVO vo) {
        try {
            String subject = vo.getSubject() + ", Env - " + activeProfile;

            if ("admin".equals(vo.getEmailTo())) {
                this.sendMailToAdmin(vo, subject);
            } else {
                SimpleMailMessage smm = new SimpleMailMessage();
                smm.setFrom(sendFrom);
                smm.setTo(vo.getEmailTo());
                smm.setSubject(subject);
                smm.setText(vo.getContent());
                mailSender.send(smm);
            }
            log.info("Send mail success. mail content - {}", vo);
        } catch (Exception e) {
            log.error("Send mail failed. mail content - {}", vo, e);
        }
    }
}

那针对此类非Web应用如何正确停服呢？

查看 org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer#doShutdown 可以看到框架已经考虑了此类问题，先停止消费消息，然后执行 cancellationLock.await 去等待woker线程完成，如果超时就强制停止了。

    @Override
    protected void doShutdown() {
        Thread thread = this.containerStoppingForAbort.get();
        if (thread != null && !thread.equals(Thread.currentThread())) {
            logger.info("Shutdown ignored - container is stopping due to an aborted consumer");
            return;
        }

        try {
            List<BlockingQueueConsumer> canceledConsumers = new ArrayList<>();
            synchronized (this.consumersMonitor) {
                if (this.consumers != null) {
                    Iterator<BlockingQueueConsumer> consumerIterator = this.consumers.iterator();
                    while (consumerIterator.hasNext()) {
                        BlockingQueueConsumer consumer = consumerIterator.next();
                        consumer.basicCancel(true);
                        canceledConsumers.add(consumer);
                        consumerIterator.remove();
                        if (consumer.declaring) {
                            consumer.thread.interrupt();
                        }
                    }
                }
                else {
                    logger.info("Shutdown ignored - container is already stopped");
                    return;
                }
            }
            logger.info("Waiting for workers to finish.");
            boolean finished = this.cancellationLock.await(getShutdownTimeout(), TimeUnit.MILLISECONDS);
            if (finished) {
                logger.info("Successfully waited for workers to finish.");
            }
            else {
                logger.info("Workers not finished.");
                if (isForceCloseChannel()) {
                    canceledConsumers.forEach(consumer -> {
                        if (logger.isWarnEnabled()) {
                            logger.warn("Closing channel for unresponsive consumer: " + consumer);
                        }
                        consumer.stop();
                    });
                }
            }
        }
        catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            logger.warn("Interrupted waiting for workers.  Continuing with shutdown.");
        }

        synchronized (this.consumersMonitor) {
            this.consumers = null;
            this.cancellationLock.deactivate();
        }

    }

但是通过调试代码发现，Spring Cloud Stream在创建MessageListenerContainer时并没有设置shutdownTimeout，所以永远是默认的5秒，而且在yml的配置中也没有shutdown超时时间的相关设置。

此时，最简单直接的办法就是修改源码，添加shutdown超时时间的配置以及在创建MessageListenerContainer时使用该配置。修改如下：

org.springframework.cloud.stream.binder.rabbit.RabbitMessageChannelBinder#createConsumerEndpoint

    @Override
    protected MessageProducer createConsumerEndpoint(ConsumerDestination consumerDestination, String group,
            ExtendedConsumerProperties<RabbitConsumerProperties> properties) {
        Assert.state(!HeaderMode.embeddedHeaders.equals(properties.getHeaderMode()),
                "the RabbitMQ binder does not support embedded headers since RabbitMQ supports headers natively");
        String destination = consumerDestination.getName();
        SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer(
                this.connectionFactory);
        // 这里加上超时时间设置
        listenerContainer.setShutdownTimeout(properties.getShutdownTimeout());
        listenerContainer.setAcknowledgeMode(properties.getExtension().getAcknowledgeMode());
        listenerContainer.setChannelTransacted(properties.getExtension().isTransacted());
        listenerContainer.setDefaultRequeueRejected(properties.getExtension().isRequeueRejected());
        int concurrency = properties.getConcurrency();
        concurrency = concurrency > 0 ? concurrency : 1;
        listenerContainer.setConcurrentConsumers(concurrency);
        int maxConcurrency = properties.getExtension().getMaxConcurrency();
        if (maxConcurrency > concurrency) {
            listenerContainer.setMaxConcurrentConsumers(maxConcurrency);
        }
        listenerContainer.setPrefetchCount(properties.getExtension().getPrefetch());
        listenerContainer.setRecoveryInterval(properties.getExtension().getRecoveryInterval());
        listenerContainer.setTxSize(properties.getExtension().getTxSize());
        listenerContainer.setTaskExecutor(new SimpleAsyncTaskExecutor(consumerDestination.getName() + "-"));
        listenerContainer.setQueueNames(destination);
        listenerContainer.setAfterReceivePostProcessors(this.decompressingPostProcessor);
        listenerContainer.setMessagePropertiesConverter(
                RabbitMessageChannelBinder.inboundMessagePropertiesConverter);
        listenerContainer.setExclusive(properties.getExtension().isExclusive());
        listenerContainer.setMissingQueuesFatal(properties.getExtension().getMissingQueuesFatal());
        if (properties.getExtension().getQueueDeclarationRetries() != null) {
            listenerContainer.setDeclarationRetries(properties.getExtension().getQueueDeclarationRetries());
        }
        if (properties.getExtension().getFailedDeclarationRetryInterval() != null) {
            listenerContainer.setFailedDeclarationRetryInterval(
                    properties.getExtension().getFailedDeclarationRetryInterval());
        }
        if (getApplicationEventPublisher() != null) {
            listenerContainer.setApplicationEventPublisher(getApplicationEventPublisher());
        }
        else if (getApplicationContext() != null) {
            listenerContainer.setApplicationEventPublisher(getApplicationContext());
        }
        listenerContainer.afterPropertiesSet();

        AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(listenerContainer);
        adapter.setBeanFactory(this.getBeanFactory());
        adapter.setBeanName("inbound." + destination);
        DefaultAmqpHeaderMapper mapper = DefaultAmqpHeaderMapper.inboundMapper();
        mapper.setRequestHeaderNames(properties.getExtension().getHeaderPatterns());
        adapter.setHeaderMapper(mapper);
        ErrorInfrastructure errorInfrastructure = registerErrorInfrastructure(consumerDestination, group, properties);
        if (properties.getMaxAttempts() > 1) {
            adapter.setRetryTemplate(buildRetryTemplate(properties));
            adapter.setRecoveryCallback(errorInfrastructure.getRecoverer());
        }
        else {
            adapter.setErrorMessageStrategy(errorMessageStrategy);
            adapter.setErrorChannel(errorInfrastructure.getErrorChannel());
        }
        return adapter;
    }

package org.springframework.cloud.stream.binder;

@JsonInclude(JsonInclude.Include.NON_DEFAULT)
public class ConsumerProperties {

    /**
     * 添加shutdown超时时间的相关配置
     * Message container shutdown timeout, default is 30s
     */
    private int shutdownTimeout = 30000;

    /**
     * The concurrency setting of the consumer. Default: 1.
     */
    private int concurrency = 1;

    /**
     * Whether the consumer receives data from a partitioned producer. Default: 'false'.
     */
    private boolean partitioned;
    
    // set get ...
}

然后将修改后的spring-cloud-stream-2.0.0.RELEASE和spring-cloud-stream-binder-rabbit-2.0.0.RELEASE发布到Maven私服，替换官方的版本即可。

最后替换shutdown.sh的脚本，将kill -9 改为kill 或 kill -15

#!/bin/sh

. ./common-env.sh

shutdownService(){
    # Find the service process id
    sid=`ps -ef | grep $1 | grep -v grep | awk '{print $2}'`

    if [ -n "${sid}" ]; then
        echo "Shutdown $1"
        kill ${sid}
    else
        echo "$1 not found"
    fi
}

# Shut down all services
for service in ${oxx_services[*]}
do
   shutdownService ${service}
done

水平有限，能力一般，文中如有错误，欢迎大家留言指正。

https://github.com/spring-projects/spring-boot/issues/4657#issuecomment-161354811

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 216,039评论 6赞 498
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 92,223评论 3赞 392
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 161,916评论 0赞 351
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 58,009评论 1赞 291
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 67,030评论 6赞 388
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 51,011评论 1赞 295
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 39,934评论 3赞 416
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 38,754评论 0赞 271
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 45,202评论 1赞 309
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 37,433评论 2赞 331
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 39,590评论 1赞 346
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 35,321评论 5赞 342
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 40,917评论 3赞 325
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 31,568评论 0赞 21
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 32,738评论 1赞 268
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 47,583评论 2赞 368
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 44,482评论 2赞 352