ceph 分布式存储-文件存储(CephFS)搭建

1. 创建元数据服务器

1.1 安装mds

PG数量的预估 集群中单个池的PG数计算公式如下：PG 总数 = (OSD 数 * 100) / 最大副本数 / 池数 (结果必须舍入到最接近2的N次幂的值)

#ceph-deploy install mds_server
 
$ ceph-deploy install ceph-xxx-osd03.gz01

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /bin/ceph-deploy install ceph-xxx-osd03.gz01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  testing                       : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xcaa830>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  dev_commit                    : None
[ceph_deploy.cli][INFO  ]  install_mds                   : False
[ceph_deploy.cli][INFO  ]  stable                        : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  adjust_repos                  : True
[ceph_deploy.cli][INFO  ]  func                          : <function install at 0xc22050>
[ceph_deploy.cli][INFO  ]  install_mgr                   : False
[ceph_deploy.cli][INFO  ]  install_all                   : False
[ceph_deploy.cli][INFO  ]  repo                          : False
[ceph_deploy.cli][INFO  ]  host                          : ['ceph-xxx-osd03.gz01']
[ceph_deploy.cli][INFO  ]  install_rgw                   : False
[ceph_deploy.cli][INFO  ]  install_tests                 : False
[ceph_deploy.cli][INFO  ]  repo_url                      : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  install_osd                   : False
[ceph_deploy.cli][INFO  ]  version_kind                  : stable
[ceph_deploy.cli][INFO  ]  install_common                : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  dev                           : master
[ceph_deploy.cli][INFO  ]  nogpgcheck                    : False
[ceph_deploy.cli][INFO  ]  local_mirror                  : None
[ceph_deploy.cli][INFO  ]  release                       : None
[ceph_deploy.cli][INFO  ]  install_mon                   : False
[ceph_deploy.cli][INFO  ]  gpg_url                       : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts ceph-xxx-osd03.gz01
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-xxx-osd03.gz01 ...
[ceph-xxx-osd03.gz01][DEBUG ] connected to host: ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] detect platform information from remote host
[ceph-xxx-osd03.gz01][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: CentOS Linux 7.2.1511 Core
[ceph-xxx-osd03.gz01][INFO  ] installing Ceph on ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][INFO  ] Running command: yum clean all
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Cleaning repos: base test_update epel extras tmprepo updates
[ceph-xxx-osd03.gz01][DEBUG ] Cleaning up everything
[ceph-xxx-osd03.gz01][INFO  ] Running command: yum -y install epel-release
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Determining fastest mirrors
[ceph-xxx-osd03.gz01][DEBUG ] 132 packages excluded due to repository priority protections
[ceph-xxx-osd03.gz01][DEBUG ] Package matching epel-release-7-9.noarch already installed. Checking for update.
[ceph-xxx-osd03.gz01][DEBUG ] Nothing to do
[ceph-xxx-osd03.gz01][INFO  ] Running command: yum -y install yum-plugin-priorities
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Loading mirror speeds from cached hostfile
[ceph-xxx-osd03.gz01][DEBUG ] 132 packages excluded due to repository priority protections
[ceph-xxx-osd03.gz01][DEBUG ] Package yum-plugin-priorities-1.1.31-42.el7.noarch already installed and latest version
[ceph-xxx-osd03.gz01][DEBUG ] Nothing to do
[ceph-xxx-osd03.gz01][DEBUG ] Configure Yum priorities to include obsoletes
[ceph-xxx-osd03.gz01][WARNIN] check_obsoletes has been enabled for Yum priorities plugin
[ceph-xxx-osd03.gz01][INFO  ] Running command: rpm --import https://download.ceph.com/keys/release.asc
[ceph-xxx-osd03.gz01][INFO  ] Running command: yum remove -y ceph-release
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository ceph is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Resolving Dependencies
[ceph-xxx-osd03.gz01][DEBUG ] --> Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] ---> Package ceph-release.noarch 0:1-1.el7 will be erased
[ceph-xxx-osd03.gz01][DEBUG ] --> Finished Dependency Resolution
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Dependencies Resolved
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ]  Package              Arch           Version            Repository         Size
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Removing:
[ceph-xxx-osd03.gz01][DEBUG ]  ceph-release         noarch         1-1.el7            installed         535
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Transaction Summary
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Remove  1 Package
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Installed size: 535
[ceph-xxx-osd03.gz01][DEBUG ] Downloading packages:
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction test
[ceph-xxx-osd03.gz01][DEBUG ] Transaction test succeeded
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction
[ceph-xxx-osd03.gz01][DEBUG ]   Erasing    : ceph-release-1-1.el7.noarch                                  1/1
[ceph-xxx-osd03.gz01][DEBUG ] warning: /etc/yum.repos.d/ceph.repo saved as /etc/yum.repos.d/ceph.repo.rpmsave
[ceph-xxx-osd03.gz01][DEBUG ]   Verifying  : ceph-release-1-1.el7.noarch                                  1/1
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Removed:
[ceph-xxx-osd03.gz01][DEBUG ]   ceph-release.noarch 0:1-1.el7
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Complete!
[ceph-xxx-osd03.gz01][INFO  ] Running command: yum install -y https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Examining /var/tmp/yum-root-Y2Avp6/ceph-release-1-0.el7.noarch.rpm: ceph-release-1-1.el7.noarch
[ceph-xxx-osd03.gz01][DEBUG ] Marking /var/tmp/yum-root-Y2Avp6/ceph-release-1-0.el7.noarch.rpm to be installed
[ceph-xxx-osd03.gz01][DEBUG ] Resolving Dependencies
[ceph-xxx-osd03.gz01][DEBUG ] --> Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] ---> Package ceph-release.noarch 0:1-1.el7 will be installed
[ceph-xxx-osd03.gz01][DEBUG ] --> Finished Dependency Resolution
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Dependencies Resolved
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ]  Package          Arch       Version     Repository                        Size
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Installing:
[ceph-xxx-osd03.gz01][DEBUG ]  ceph-release     noarch     1-1.el7     /ceph-release-1-0.el7.noarch     535
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Transaction Summary
[ceph-xxx-osd03.gz01][DEBUG ] ================================================================================
[ceph-xxx-osd03.gz01][DEBUG ] Install  1 Package
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Total size: 535
[ceph-xxx-osd03.gz01][DEBUG ] Installed size: 535
[ceph-xxx-osd03.gz01][DEBUG ] Downloading packages:
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction check
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction test
[ceph-xxx-osd03.gz01][DEBUG ] Transaction test succeeded
[ceph-xxx-osd03.gz01][DEBUG ] Running transaction
[ceph-xxx-osd03.gz01][DEBUG ]   Installing : ceph-release-1-1.el7.noarch                                  1/1
[ceph-xxx-osd03.gz01][DEBUG ]   Verifying  : ceph-release-1-1.el7.noarch                                  1/1
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Installed:
[ceph-xxx-osd03.gz01][DEBUG ]   ceph-release.noarch 0:1-1.el7
[ceph-xxx-osd03.gz01][DEBUG ]
[ceph-xxx-osd03.gz01][DEBUG ] Complete!
[ceph-xxx-osd03.gz01][WARNIN] ensuring that /etc/yum.repos.d/ceph.repo contains a high priority
[ceph-xxx-osd03.gz01][WARNIN] altered ceph.repo priorities to contain: priority=1
[ceph-xxx-osd03.gz01][INFO  ] Running command: yum -y install ceph ceph-radosgw
[ceph-xxx-osd03.gz01][DEBUG ] Loaded plugins: fastestmirror, langpacks, priorities
[ceph-xxx-osd03.gz01][WARNIN] Repository epel is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-debuginfo is listed more than once in the configuration
[ceph-xxx-osd03.gz01][WARNIN] Repository epel-source is listed more than once in the configuration
[ceph-xxx-osd03.gz01][DEBUG ] Loading mirror speeds from cached hostfile
[ceph-xxx-osd03.gz01][DEBUG ] 133 packages excluded due to repository priority protections
[ceph-xxx-osd03.gz01][DEBUG ] Package matching 1:ceph-10.2.10-0.el7.x86_64 already installed. Checking for update.
[ceph-xxx-osd03.gz01][DEBUG ] Package matching 1:ceph-radosgw-10.2.10-0.el7.x86_64 already installed. Checking for update.
[ceph-xxx-osd03.gz01][DEBUG ] Nothing to do
[ceph-xxx-osd03.gz01][INFO  ] Running command: ceph --version
[ceph-xxx-osd03.gz01][DEBUG ] ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)

1.2 拷贝ceph配置及秘钥

$ cd /etc/ceph/
$ ceph-deploy admin ceph-xxx-osd03.gz01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /bin/ceph-deploy admin ceph-xxx-osd03.gz01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2ac9b90>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['ceph-xxx-osd03.gz01']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x2a24c08>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] connected to host: ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] detect platform information from remote host
[ceph-xxx-osd03.gz01][DEBUG ] detect machine type
[ceph-xxx-osd03.gz01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

1.3 修改秘钥文件权限，保证有可读权限

sudo chmod +r /etc/ceph/ceph.client.admin.keyring

1.4 创建元数据服务器

$ ceph-deploy --overwrite-conf mds create ceph-xxx-osd03.gz01

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf mds create ceph-xxx-osd03.gz01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xc61518>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0xc428c0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('ceph-xxx-osd03.gz01', 'ceph-xxx-osd03.gz01')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts ceph-xxx-osd03.gz01:ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] connected to host: ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] detect platform information from remote host
[ceph-xxx-osd03.gz01][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.2.1511 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-xxx-osd03.gz01][DEBUG ] create path if it doesn't exist
[ceph-xxx-osd03.gz01][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-xxx-osd03.gz01 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-xxx-osd03.gz01/keyring
[ceph-xxx-osd03.gz01][INFO  ] Running command: systemctl enable ceph-mds@ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-xxx-osd03.gz01.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-xxx-osd03.gz01][INFO  ] Running command: systemctl start ceph-mds@ceph-xxx-osd03.gz01
[ceph-xxx-osd03.gz01][INFO  ] Running command: systemctl enable ceph.target

2. 存储池

2.1 创建存储池数据

#ceph osd pool create cephfs_data <pg_num>
 
$ ceph osd pool create cephfs_data 128
pool 'cephfs_data' created

2.2 创建存储池元数据

#ceph osd pool create cephfs_metadata <pg_num>
 
$ ceph osd pool create cephfs_data 128
pool 'cephfs_metadata' created

2.3 查看存储池

$ ceph osd lspools
1 rbd,2 test_data,3 test_metadata,5 test,6 benmark_test,7 .rgw.root,8 default.rgw.control,9 default.rgw.meta,10 default.rgw.log,11 default.rgw.buckets.index,12 web-services,13 test_pool,15 cephfs_data,16 cephfs_metadata

3. 文件系统

3.1 创建文件系统

#ceph fs new <fs_name> cephfs_metadata cephfs_data
 
$ ceph fs new test_fs cephfs_metadata cephfs_data
new fs with metadata pool 16 and data pool 15

3.2 查看文件系统

$ ceph fs ls
name: test_fs, metadata pool: test_metadata, data pools: [test_data ]

4. mds状态

4.1 查看mds状态

$ ceph mds stat
test_fs-1/1/1 up [test_fs:0]=ceph-bench-osd00=up:active}

说明：

  • e8 : e标识epoch，8是epoch号
  • tstfs-1/1/1 up : tstfs是cephfs名字，后面的三个1分别是[mds_map.in/mds_map.up/mds_map.max_mds](http://mds_map.in/mds_map.up/mds_map.max_mds)，up是cephfs状态
  • {[tstfs:0]=mds-daemon-1=up:active} : [tstfs:0]指tstfs的rank 0，mds-daemon-1是服务tstfs的mds daemon name，up:active是cephfs的状态为 up & active
  • test_fs-1是active的，它的mds daemon为ceph-xxx-osd03.gz01
  • 又添加一个新的mds daemon后，它会处于standby状态，若前两个mds daemon出问题，它会顶替上去

5. 内核驱动挂载ceph文件系统

5.1 查看秘钥

$ cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQBpxadZjXLcBxAArU8dYn75aAZYfdGdogYRYg==

5.2 创建目录

$ sudo mkdir /mnt/mycephfs

5.3 挂载文件系统

#指定用户名、密钥
#sudo mount -t {ip-address-of-monitor}:6789 /mycephfs -o name=xxxx,secret=xxxx
 
$ sudo mount -t ceph 10.1.178.32:6789:/ /mnt/mycephfs -o name=admin,secret=AQBpxadZjXLcBxAArU8dYn75aAZYfdGdogYRYg==
$ df -h
10.1.178.32:6789:/  201T  1.9T  199T   1% /mnt/mycephfs

6. 用户空间挂载ceph文件系统

6.1 安装ceph-fuse

$ yum install ceph-fuse

6.2 创建目录

$ sudo mkdir /mnt/mycephfs-fuse

6.3 创建用户

#ceph auth get-or-create client.<client-name/id> mon 'allow r' mds 'allow r, allow rw path=<directory>' osd 'allow rw pool=<pool>' -o <file_name>

$ ceph auth get-or-create client.1 mon 'allow r' mds 'allow r, allow
rw path=/' osd 'allow rw pool=data' -o ceph.client.1.keyring
[client.1]
key = AQACNoZXhrzqIRAABPKHTach4x03JeNadeQ9Uw==

6.4 挂载文件系统

说明:

  • ceph-fuse 识别不了的选项将传递给 libfuse 。
  • -d启动后将脱离终端、进入守护状态。
  • -c ceph.conf, --conf=ceph.conf用指定的 ceph.conf 而非默认的
    /etc/ceph/ceph.conf 来查找启动时需要的监视器地址。
  • -m monaddress[:port]连接到指定监视器，而不是从 ceph.conf 里找。
  • -r root_directory把文件系统内的 root_directory 作为根挂载，而不是整个 Ceph 文件系统树。
#Ceph 存储集群默认要求认证，需指定相应的密钥环文件
#sudo ceph-fuse -n client.<client-name> - -k keyring -m {ip-address-of-monitor}:6789 /mycephfs
 
$ sudo ceph-fuse -n client.1  -k /etc/ceph/ceph.client.admin.keyring -m 10.1.178.32:6789 /mnt/mycephfs-fuse
2017-11-15 14:26:45.323695 7fef1e1e2ec0 -1 init, newargv = 0x7fef29cb68a0 newargc=11ceph-fuse[3300741]:
starting ceph client

$ df -h
ceph-fuse 201T 1.9T 199T 1% /mnt/mycephfs-fuse
最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 214,658评论 6赞 496
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 91,482评论 3赞 389
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 160,213评论 0赞 350
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 57,395评论 1赞 288
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 66,487评论 6赞 386
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 50,523评论 1赞 293
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 39,525评论 3赞 414
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 38,300评论 0赞 270
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 44,753评论 1赞 307
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 37,048评论 2赞 330
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 39,223评论 1赞 343
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 34,905评论 5赞 338
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 40,541评论 3赞 322
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 31,168评论 0赞 21
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 32,417评论 1赞 268
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 47,094评论 2赞 365
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 44,088评论 2赞 352

推荐阅读更多精彩内容