rocketmq_过期文件清理

Q、broker清理过期文件的机制是什么?

A:broker在启动的时候会开启一个定时任务,默认延迟60s执行,间隔10s执行一次;

      // org.apache.rocketmq.store.DefaultMessageStore 1346
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                DefaultMessageStore.this.cleanFilesPeriodically();
            }
        }, 1000 * 60, this.messageStoreConfig.getCleanResourceInterval(), TimeUnit.MILLISECONDS);
Q、broker会清理哪些过期的文件?

A:commitLog和consumeQueue

// org.apache.rocketmq.store.DefaultMessageStore 1389
private void cleanFilesPeriodically() {
        // 删除commitlog
        this.cleanCommitLogService.run();
        // 删除consumeQueue
        this.cleanConsumeQueueService.run();
    }

Q、broker是如何判定一个文件已经过期?

A:当前时间-文件最后一次更新时间>3天

 // org.apache.rocketmq.store.MappedFileQueue 354
 long liveMaxTimestamp = mappedFile.getLastModifiedTimestamp() + expiredTime;
 if (System.currentTimeMillis() >= liveMaxTimestamp || cleanImmediately) {
       if (mappedFile.destroy(intervalForcibly)) {
       }
}
Q、哪些情况会触发broker删除过期文件?

A:(1) 到了删除的时间点(可配置),默认为凌晨4点;

// org.apache.rocketmq.store.DefaultMessageStore.CleanCommitLogService 1674
private boolean isTimeToDelete() {
   String when = DefaultMessageStore.this.getMessageStoreConfig().getDeleteWhen();
      if (UtilAll.isItTimeToDo(when)) {
              DefaultMessageStore.log.info("it's time to reclaim disk space, " + when);
               return true;
      }
       return false;
}

(2) 磁盘空间不足(这个方法的逻辑很清晰,不再多啰嗦);

private boolean isSpaceToDelete() {
            double ratio = DefaultMessageStore.this.getMessageStoreConfig().getDiskMaxUsedSpaceRatio() / 100.0;

            cleanImmediately = false;

            {
                String storePathPhysic = DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog();
                // 当前磁盘分区(比如c盘、d盘)的空间使用率
                double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic);
                if (physicRatio > diskSpaceWarningLevelRatio) {  // 大于90%
                    // 如果磁盘分区使用率超过该阔值,将设置磁盘不可写,此时会拒绝新消息的写入。
                    boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskFull();
                    if (diskok) {
                        DefaultMessageStore.log.error("physic disk maybe full soon " + physicRatio + ", so mark disk full");
                    }

                    cleanImmediately = true;
                } else if (physicRatio > diskSpaceCleanForciblyRatio) { // 大于85%
                    cleanImmediately = true;
                } else {
                    boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskOK();
                    if (!diskok) {
                        DefaultMessageStore.log.info("physic disk space OK " + physicRatio + ", so mark disk ok");
                    }
                }

                if (physicRatio < 0 || physicRatio > ratio) {  // 如果ratio配置太大,不会删除过期文件,但是runningFlags改了,消息便不能发送成功
                    DefaultMessageStore.log.info("physic disk maybe full soon, so reclaim space, " + physicRatio);
                    return true;
                }
            }

            {
                String storePathLogics = StorePathConfigHelper
                    .getStorePathConsumeQueue(DefaultMessageStore.this.getMessageStoreConfig().getStorePathRootDir());
                double logicsRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathLogics);
                if (logicsRatio > diskSpaceWarningLevelRatio) {
                    boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskFull();
                    if (diskok) {
                        DefaultMessageStore.log.error("logics disk maybe full soon " + logicsRatio + ", so mark disk full");
                    }

                    cleanImmediately = true;
                } else if (logicsRatio > diskSpaceCleanForciblyRatio) {
                    cleanImmediately = true;
                } else {
                    boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskOK();
                    if (!diskok) {
                        DefaultMessageStore.log.info("logics disk space OK " + logicsRatio + ", so mark disk ok");
                    }
                }

                if (logicsRatio < 0 || logicsRatio > ratio) {  //
                    DefaultMessageStore.log.info("logics disk maybe full soon, so reclaim space, " + logicsRatio);
                    return true;
                }
            }

            return false;
        }

(3)预留,手工触发

boolean manualDelete = this.manualDeleteFileSeveralTimes > 0;
最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

推荐阅读更多精彩内容