「ceph-deploy」CentOS7部署Ceph-nautilus 14.2.18版本集群学习

主机名

系统

IP

安装软件

ceph-admin

192.168.168.200

ceph-deploy

ceph-node1

192.168.168.201

ceph
mon

mgr
mds

ceph-node2

192.168.168.202

ceph
mon

ceph-node3

192.168.168.203

ceph
mon

版本信息

[root@ceph-adminmy-cluster]1SMPFriOct1817:15:30UTC2019[root@ceph-adminmy-cluster]cat/etc/[root@ceph-admin~]cat/etc/hosts|

关闭防火墙并设置开启不启动systemctlstopfirewalldsystemctldisablefirewalld所有节点设置时区timedatectlset-timezoneAsia/Shanghaicp/etc//etc/新增以自己为serversed-i'/^配置允许访问的client网段sed-i'/设置即使不同步其他时钟源，该server依然可以作为时钟源sed-i'/^\(.*\)$/\1/g'/etc/检查时间同步是否正常，client端出现*号表示时间同步正常chronycsourcescat/etc/().ablekernelsynchronizationofthereal-timeclock(RTC).rtcsynchwtimestamp*/etc/

注释原有的时间同步serversed-i'/^server/s/^\(.*\)$/新增以ceph-admin为serversed-i'/^所有节点重启chrony，使其生效systemctlrestartchronydchronycsources210Numberofsources=1MSName/IPaddressStratumPollReachLastRxLastsample===============================================================================^*192.168.168.200361738+478ns[-21us]+/-21ms

由于使用root用户安装，所以不创建新账号将公钥分发到各个node节点foriin123dossh-copy-idroot@ceph-node$idone

[root@ceph-admin~]ceph-deploy--versionTraceback(mostrecentcalllast):File"/usr/bin/ceph-deploy",line18,inmodulefromceph_"/usr/lib//site-packages/ceph_deploy/",line1,inmoduleimportpkg_resourcesImportError:Nomodulenamedpkg_resources[root@ceph-admin~]yuminstall-ypython2-pip已安装::8.1.2-14.el7作为依赖被安装:_640:1.0-8.el7python-backports-ssl_match_:3.5.0.1-1.:1.0.16-2.:0.9.8-7.el7完毕！[root@ceph-admin~]pwd/root[root@ceph-admin~]cdmy-cluster/[root@ceph-adminmy-cluster]创建一个集群，--/24用于集群内部同步/24--/24ceph-node1[root@ceph-adminmy-cluster][root@ceph-adminmy-cluster][mon.]key=AQCJFldgAAAAABAAP/KUN+lSmq04mFsIkXUzcA==capsmon=allow*

[root@ceph-adminmy-cluster]。。。。。。。[ceph-node1][DEBUG]正在检查/var/tmp/yum-root-7DB79r/:[ceph-node1][DEBUG]/var/tmp/yum-root-7DB79r/将被安装[ceph-node1][DEBUG]正在解决依赖关系[ceph-node1][DEBUG]--正在检查事务[ceph-node1][DEBUG]---软件包将被安装[ceph-node1][DEBUG]--解决依赖关系完成[ceph-node1][DEBUG][ceph-node1][DEBUG]依赖关系解决[ceph-node1][DEBUG][ceph-node1][DEBUG]================================================================================[ceph-node1][DEBUG]Package架构版本源大小[ceph-node1][DEBUG]================================================================================[ceph-node1][DEBUG]正在安装:[ceph-node1][DEBUG]/[ceph-node1][DEBUG][ceph-node1][DEBUG]事务概要[ceph-node1][DEBUG]================================================================================[ceph-node1][DEBUG]安装1软件包[ceph-node1][DEBUG][ceph-node1][DEBUG]总计：535[ceph-node1][DEBUG]安装大小：535[ceph-node1][DEBUG]Downloadingpackages:[ceph-node1][DEBUG]Runningtransactioncheck[ceph-node1][DEBUG]Runningtransactiontest[ceph-node1][DEBUG]Transactiontestsucceeded[ceph-node1][DEBUG]Runningtransaction[ceph-node1][DEBUG]正在安装:/1[ceph-node1][DEBUG]警告：/etc//已建立为/etc//[ceph-node1][DEBUG]验证中:/1[ceph-node1][DEBUG][ceph-node1][DEBUG]已安装:[ceph-node1][DEBUG]:1-1.el7[ceph-node1][DEBUG][ceph-node1][DEBUG]完毕！[ceph-node1][WARNIN]ensuringthat/etc//[ceph_deploy][ERROR]RuntimeError:NoSectionError:Nosection:'ceph'

解决办法：在失败的节点上执行yumremoveceph-release
然后在admin节点上继续执行

[root@ceph-adminmy-cluster](564bdc4ae87418a232fc901524470e1a0f76d641)mimic(stable)

先通过yumlistceph查询是否是需要的版本，如果查不到任何内容，先检查ceph源是否存在，再次查询。主要要添加好ceph对应版本的源。

[root@ceph-node1~]node1节点需要安装monitor[root@ceph-node1~]。。。已安装:_642:14.2.18-0._642:14.2.18-0.el7作为依赖被安装:_642:14.2.18-0._642:14.2.18-0._642:14.2.18-0._642:14.2.18-0._642:14.2.18-0._642:14.2.18-0._640:1.4.9-5._640:0.11.5-1._640:1.8.1-2.el7_9:1.8.1-2.el7_9_640:1.8.1-2.el7_9:1.0.4-1._640:2.0.4-4.el7完毕！(befbc92f3c11eedd8626487211d200c0b44786d9)nautilus(stable)[root@ceph-node1~]node2、node3节点仅按章ceph[root@ceph-node2~]yuminstall-yceph

ceph-deploymoncreate-initial：将部署为在“moninitialmembers”中定义的监视器，等待它们形成仲裁，然后进将为“moninitialmembers”中定义的监控器部署，等待它们形成仲裁，然后收集密钥，并在进程中报告监控器状态。如果监视器没有形成仲裁，命令最终将超时。

ceph-scluster:id:5f844a3b-1257-4089-bff5-836def275bf0health:HEALTH_OKservices:mon:1daemons,quorumceph-node1(age75s)mgr:nodaemonsactiveosd:0osds:0up,0indata:pools:0pools,0pgsobjects:0objects,0Busage:0Bused,0B/0Bavailpgs:

ceph-deploymgrcreateceph-node1

在node节点上查看集群状态，可以看到mgr已经添加到ceph集群，此处的HEALTH_WARN代表还没添加OSD

[root@ceph-node1~]lsblkNAMEMAJ:MINRMSIZEROTYPEMOUNTPOINTsda8:0010G0disk├─sda18:101G0part/boot└─sda28:209G0part├─centos-root253:008G0lvm/└─centos-swap253:101G0lvm[SWAP]sdb8:16010G0disksr011:011024M0rom

将三个节点的/dev/sdb加入OSD

将node节点的/dev/sdb加入OSD[root@ceph-adminmy-cluster]。。。[ceph-node1][INFO]checkingOSDstatus[ceph-node1][DEBUG]findthelocationofanexecutable[ceph-node1][INFO]Runningcommand:/bin/ceph--cluster=cephosdstat--format=json[ceph_][DEBUG]:id:5f844a3b-1257-4089-bff5-836def275bf0health:HEALTH_WARNOSDcount1osd_pool_default_size3services:mon:1daemons,quorumceph-node1(age6m)mgr:ceph-node1(active,since6m)osd:1osds:1up(since35s),1in(since35s)data:pools:0pools,0pgsobjects:0objects,0Busage:1.0GiBused,9.0GiB/10GiBavailpgs:

继续添加剩余的OSD

[root@ceph-adminmy-cluster]ceph-deployosdcreate--data/dev/sdbceph-node3

最后再来查看所有OSD和集群状态

:id:5f844a3b-1257-4089-bff5-836def275bf0health:HEALTH_OKservices:mon:1daemons,quorumceph-node1(age10m)mgr:ceph-node1(active,since10m)osd:3osds:3up(since35s),3in(since35s)data:pools:0pools,0pgsobjects:0objects,0Busage:3.0GiBused,27GiB/30GiBavailpgs:

CephMetadataServer元数据服务器添加到node1节点。然后将CephMonitor和CephManager添加到node2和node3节点，以提高可靠性和可用性。

要使用Cephfs，您需要至少一个元数据服务器。执行以下内容以创建元数据服务器：

[root@ceph-adminmy-cluster]在node节点上查看mds[root@ceph-node1~]先在node节点上查看mon状态[root@ceph-node1~]admin节点操作，添加node2和node3为monitor[root@ceph-adminmy-cluster]执行报错[ceph-node2][INFO]Runningcommand:ceph--cluster=ceph--admin-daemon/var/run/ceph/_status[ceph-node2][ERROR]admin_socket:exceptiongettingcommanddescriptions:[Errno2]Nosuchfileordirectory[ceph-node2][WARNIN]ceph-node2isnotdefinedin`moninitialmembers`[ceph-node2][WARNIN]monitorceph-node2doesnotexistinmonmap[ceph-node2][WARNIN]neither`public_addr`nor`public_network`keysaredefinedformonitors[ceph-node2][WARNIN]monitorsmaynotbeabletoformquorum[ceph-node2][INFO]Runningcommand:ceph--cluster=ceph--admin-daemon/var/run/ceph/_status[ceph-node2][ERROR]admin_socket:exceptiongettingcommanddescriptions:[Errno2]Nosuchfileordirectory[ceph-node2][WARNIN]monitor:,mightnotberunningyet

查阅资料，要在定义一下public_network，一定要添加到[global]段

[root@ceph-adminmy-cluster]推送到node节点[root@ceph-adminmy-cluster]admin节点操作，添加node2为monitor[root@ceph-adminmy-cluster]admin节点操作，添加node3为monitor[root@ceph-adminmy-cluster]其中有个[ceph-node2][WARNIN]ceph-node2isnotdefinedin`moninitialmembers`的错误警告[global]fsid=5f844a3b-1257-4089-bff5-836def275bf0mon_initial_members=ceph-node1,ceph-node2,ceph-node3mon_host=192.168.168.201auth_cluster_required=cephxauth_service_required=cephxauth_client_required=cephxpublicnetwork=192.168.168.0/24

查看monitor状态

cephquorum_status--formatjson-pretty

{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["ceph-node1","ceph-node2","ceph-node3"],"quorum_leader_name":"ceph-node1","quorum_age":257,"monmap":{"epoch":3,"fsid":"5f844a3b-1257-4089-bff5-836def275bf0","modified":"2021-03-2122:10:22.992987","created":"2021-03-2118:53:14.599473","min_mon_release":14,"min_mon_release_name":"nautilus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus"],"optional":[]},"mons":[{"rank":0,"name":"ceph-node1","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.201:3300","nonce":0},{"type":"v1","addr":"192.168.168.201:6789","nonce":0}]},"addr":"192.168.168.201:6789/0","public_addr":"192.168.168.201:6789/0"},{"rank":1,"name":"ceph-node2","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.202:3300","nonce":0},{"type":"v1","addr":"192.168.168.202:6789","nonce":0}]},"addr":"192.168.168.202:6789/0","public_addr":"192.168.168.202:6789/0"},{"rank":2,"name":"ceph-node3","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.203:3300","nonce":0},{"type":"v1","addr":"192.168.168.203:6789","nonce":0}]},"addr":"192.168.168.203:6789/0","public_addr":"192.168.168.203:6789/0"}]}}

cephmonstate3:3monsat{ceph-node1=[v2:192.168.168.201:3300/0,v1:192.168.168.201:6789/0],ceph-node2=[v2:192.168.168.202:3300/0,v1:192.168.168.202:6789/0],ceph-node3=[v2:192.168.168.203:3300/0,v1:192.168.168.203:6789/0]},electionepoch14,leader0ceph-node1,quorum0,1,2ceph-node1,ceph-node2,ceph-node3cephmondumpdumpedmonmapepoch3epoch3fsid5f844a3b-1257-4089-bff5-836def275bf0last_changed2021-03-2122:10:22.992987created2021-03-2118:53:14.599473min_mon_release14(nautilus)0:[v2:192.168.168.201:3300/0,v1:192.168.168.201:6789/0]:[v2:192.168.168.202:3300/0,v1:192.168.168.202:6789/0]:[v2:192.168.168.203:3300/0,v1:192.168.168.203:6789/0]:id:5f844a3b-1257-4089-bff5-836def275bf0health:HEALTH_OKservices:mon:3daemons,quorumceph-node1,ceph-node2,ceph-node3(age6m)mgr:ceph-node1(active,since48m)mds:1up:standbyosd:3osds:3up(since53m),3in(since2h)data:pools:0pools,0pgsobjects:0objects,0Busage:3.0GiBused,27GiB/30GiBavailpgs:

CephManager守护进程以活动/待机模式运行。部署其他管理器守护程序可确保如果一个守护程序或主机发生故障，则可以在不中断服务的情况下接管另一个。
mgr默认是主备模式，同一时间只有一个是运行的。

ceph-deploymgrcreateceph-node2ceph-node3

ceph-scluster:id:5f844a3b-1257-4089-bff5-836def275bf0health:HEALTH_OKservices:mon:3daemons,quorumceph-node1,ceph-node2,ceph-node3(age7m)mgr:ceph-node1(active,since95m),standbys:ceph-node2,ceph-node3mds:1up:standbyosd:3osds:3up(since12m),3in(since85m)data:pools:0pools,0pgsobjects:0objects,0Busage:3.0GiBused,27GiB/30GiBavailpgs:

要使用Ceph的Ceph对象网关组件，必须部署RGW的实例。执行以下内容以创建RGW的新实例：

yumlistceph-radosgw已加载插件：fastestmirror,prioritiesLoadingmirrorspeedsfromcachedhostfile*base:*extras:*updates:可安装的软件包_642:14.2.18-0.el7ceph-x86_64[root@ceph-node1~]admin节点执行[root@ceph-adminmy-cluster]。。。[ceph_][INFO]TheCephObjectGateway(RGW)isnowrunningonhostceph-node1anddefaultport7480========[client]rgwfronts=civetwebport=80[client]rgwfronts=civetwebport=[::]:80ceph-scluster:id:5f844a3b-1257-4089-bff5-836def275bf0health:HEALTH_OKservices:mon:3daemons,quorumceph-node1,ceph-node2,ceph-node3(age22m)mgr:ceph-node1(active,since2h),standbys:ceph-node2,ceph-node3mds:1up:standbyosd:3osds:3up(since68m),3in(since2h)rgw:1daemonactive(ceph-node1)taskstatus:data:pools:4pools,128pgsobjects:187objects,1.2KiBusage:3.0GiBused,27GiB/30GiBavailpgs:128active+clean

[root@centos7~]修改主机名[root@ceph-client~]yum-yinstallcephceph-radosgw-y[root@ceph-client~]配置主机名免密登录[root@ceph-admin~]ssh-copy-idceph-clientcdmy-cluster/[root@ceph-adminmy-cluster]ls/etc/ceph/

cephosdpoolcreateceph-demo6464pool'ceph-demo'_numpg_num:64[root@ceph-node1~]cephosdpoolgetceph-demosizesize:3[root@ceph-node1~]cephosdpoolgetceph-democrush_rulecrush_rule:replicated_rule

命令：rbdcreate--size{megabytes}{pool-name}/{image-name}

要把块设备加入某节点，必须在集群中创建一个映像。指定存储池必须是已创建好的。

/

如果创建映像时不指定存储池名称，默认使用rbd存储池，同时rbd存储池要先自行创建好。
rbdcreate--size1024foo表示在默认的rbd存储池中创建一个大小为1GB，名为foo的映像。

命令：rbdls{poolname}

如果不指定存储池，rbdls则列出rbd存储池中的块设备。

[root@ceph-client~]查看rbd详细信息[root@ceph-client~]rbdinfoceph-demo/'':size10GiBin2560objectsorder22(4MiBobjects)snapshot_count:0id:2332d4832d85cblock_name_prefix:rbd_:2features:layering,exclusive-lock,object-map,fast-diff,deep-flattenop_features:flags:create_timestamp:TueMar3023:03:222021access_timestamp:TueMar3023:03:222021modify_timestamp:TueMar3023:03:222021

命令：rbdrm{image-name}

命令：rbdrm{pool-name}/{image-name}，从某存储池中删除一个块设备，用要删除的映像名字替换{image-name}、用存储池名字替换{pool-name}

:100%completedone.

用rbd把映像名映射为内核模块。必须指定映像名、存储池名、和用户名。若RBD内核模块尚未加载，rbd命令会自动加载。

命令：rbdmap{pool-name}/{image-name}--id{user-name}

命令：rbdmap{pool-name}/{image-name}--id{user-name}--keyring/path/to/keyring，如果启用了cephx认证，还必须提供密钥，可以用密钥环或密钥文件指定密钥。

[root@ceph-client~]rbdmapceph-demo/:warning:imagealreadymappedas/dev/rbd0/dev/rbd1[root@ceph-client~]取消块设备映射[root@ceph-client~]查看rbd信息[root@ceph-client~]或者[root@ceph-client~]ls/dev/rbd/ceph-demo[root@ceph-client~]多出了一个裸设备/dev/rdb0[root@ceph-client~]格式化设备，挂载使用[root@ceph-client~]挂载[root@ceph-client~]mount/dev/rbd0/mnt/rbd-demo/[root@ceph-client~]扩容rbd[root@ceph-client~]df-h|greprbd/dev/%/mnt/rbd-demoresize2fs/dev/(28-Dec-2013)Filesystemat/dev/rbd0ismountedon/mnt/rbd-demo;on-lineresizingrequiredold_desc_blocks=2,new_desc_blocks=2Thefilesystemon/dev/|greprbd/dev/rbd015G41M14G1%/mnt/rbd-demo

[root@ceph-client~]mount/dev/rbd0/mnt/rbd-demo/[root@ceph-client~]cephmdsstat1up:standby

创建存储池，但结果不符合预期测试删除

[root@ceph-node1~]cephosdpoolcreatecephfs_data64ErrorERANGE:pg_num64size3wouldmean768totalpgs,whichexceedsmax750(mon_max_pg_per_osd250*num_in_osds3)[root@ceph-node1~]删除该pool[root@ceph-node1~]提示需要在配置文件中加入下一行配置[root@ceph-node1~]重启ceph-mon服务[root@ceph-node1~]重新删除就成功了[root@ceph-node1~]获取自己已创建的[root@ceph-node1~]获取现有的PG数和PGP数值，之前创建的是64个[root@ceph-node1~]cephosdpoolgetceph-demopgp_numpgp_num:64cephosddump|grepceph|grep-isizepool5'ceph-demo'replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num64pgp_num64autoscale_modewarnlast_change84flagshashpspool,selfmanaged_snapsstripe_width0applicationrbdcephhealthHEALTH_OK[root@ceph-node1~]pg调整期间使用ceph-w监测状态，可以看到集群状态的详细信息，可以看到数据的再平衡过程。等状态再次恢复正常后（[INF]overallHEALTH_OK），再调整pgp[root@ceph-node1~]cephosdpoolcreatecephfs_data32pool'cephfs_data'created[root@ceph-node1~]cephfsnewcephfscephfs_metadatacephfs_datanewfswithmetadatapool8anddatapool7[root@ceph-node1~]cephmdsstatcephfs:1{0=ceph-node1=up:active}直接挂在会失败[root@ceph-client~]查看密钥内容[root@ceph-client~]要挂载启用了cephx认证的Ceph文件系统，你必须指定用户名、密钥。[root@ceph-client~]df-h|:6789:/8.3%/mnt/mycephfsumount/mnt/mycephfs/cat/etc/ceph/|grepkey|awk-F"="'{print$2}'/etc/ceph/:6789://mnt/mycephfs/-oname=admin,secretfile=/etc/ceph/[root@ceph-client~]启动mon进程启动osd进程

集群健康状态：cephhealth

[root@ceph-node2~]ceph-scluster:id:5f844a3b-1257-4089-bff5-836def275bf0health:HEALTH_OKservices:mon:3daemons,quorumceph-node1,ceph-node2,ceph-node3(age14m)mgr:ceph-node1(active,since14m),standbys:ceph-node2,ceph-node3mds:1up:standbyosd:3osds:3up(since14m),3in(since46h)rgw:1daemonactive(ceph-node1)taskstatus:data:pools:4pools,128pgsobjects:187objects,1.2KiBusage:3.0GiBused,27GiB/30GiBavailpgs:128active+clean

集群运行实时状态：ceph-w、watchceph-s

ceph-w

集群健康状态细节：cephhealthdetail

[root@ceph-node2~]cephdfRAWSTORAGE:CLASSSIZEAVAILUSEDRAWUSED%:POOLIDPGSSTOREDOBJECTSUSED%

[root@ceph-node2~]|more

查看cephlog日志所在的目录：_file

[root@ceph-node1~]_file/var/log/ceph/:3monsat{ceph-node1=[v2:192.168.168.201:3300/0,v1:192.168.168.201:6789/0],ceph-node2=[v2:192.168.168.202:3300/0,v1:192.168.168.202:6789/0],ceph-node3=[v2:192.168.168.203:3300/0,v1:192.168.168.203:6789/0]},electionepoch62,leader0ceph-node1,quorum0,1,2ceph-node1,ceph-node2,ceph-node3[root@ceph-node1~]可以用json格式显示

{"epoch":3,"min_mon_release_name":"14","num_mons":3,"leader":"ceph-node1","quorum":[{"rank":0,"name":"ceph-node1"},{"rank":1,"name":"ceph-node2"},{"rank":2,"name":"ceph-node3"}]}

查看状态：cephmon_status

[root@ceph-node1~]cephquorum_status{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["ceph-node1","ceph-node2","ceph-node3"],"quorum_leader_name":"ceph-node1","quorum_age":2313,"monmap":{"epoch":3,"fsid":"5f844a3b-1257-4089-bff5-836def275bf0","modified":"2021-03-2122:10:22.992987","created":"2021-03-2118:53:14.599473","min_mon_release":14,"min_mon_release_name":"nautilus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus"],"optional":[]},"mons":[{"rank":0,"name":"ceph-node1","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.201:3300","nonce":0},{"type":"v1","addr":"192.168.168.201:6789","nonce":0}]},"addr":"192.168.168.201:6789/0","public_addr":"192.168.168.201:6789/0"},{"rank":1,"name":"ceph-node2","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.202:3300","nonce":0},{"type":"v1","addr":"192.168.168.202:6789","nonce":0}]},"addr":"192.168.168.202:6789/0","public_addr":"192.168.168.202:6789/0"},{"rank":2,"name":"ceph-node3","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.203:3300","nonce":0},{"type":"v1","addr":"192.168.168.203:6789","nonce":0}]},"addr":"192.168.168.203:6789/0","public_addr":"192.168.168.203:6789/0"}]}}[root@ceph-node1~]json显示

{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["ceph-node1","ceph-node2","ceph-node3"],"quorum_leader_name":"ceph-node1","quorum_age":2330,"monmap":{"epoch":3,"fsid":"5f844a3b-1257-4089-bff5-836def275bf0","modified":"2021-03-2122:10:22.992987","created":"2021-03-2118:53:14.599473","min_mon_release":14,"min_mon_release_name":"nautilus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus"],"optional":[]},"mons":[{"rank":0,"name":"ceph-node1","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.201:3300","nonce":0},{"type":"v1","addr":"192.168.168.201:6789","nonce":0}]},"addr":"192.168.168.201:6789/0","public_addr":"192.168.168.201:6789/0"},{"rank":1,"name":"ceph-node2","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.202:3300","nonce":0},{"type":"v1","addr":"192.168.168.202:6789","nonce":0}]},"addr":"192.168.168.202:6789/0","public_addr":"192.168.168.202:6789/0"},{"rank":2,"name":"ceph-node3","public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.168.203:3300","nonce":0},{"type":"v1","addr":"192.168.168.203:6789","nonce":0}]},"addr":"192.168.168.203:6789/0","public_addr":"192.168.168.203:6789/0"}]}}

映射信息：cephmondump

[root@ceph-node1~]_statuscephmongetmap-omon_

查看上面获得的map：monmaptool–printmon_和cephmondump返回值一样

[root@ceph-node1~]_socket/var/run/ceph/

删除一个mon节点：cephmonremoveceph-node2

[root@ceph-node2~]通过ceph-w实时监控到的状态变化2021-03-2422:05:21.524555[INF]:05:23.499174[INF]:05:23.506510[INF],monsceph-node1,ceph-node3inquorum(ranks0,1)

将删除的节点添加进来：monaddnameIPaddr[:port]:addnewmonitornamednameataddr

[root@ceph-node2~]ceph-w显示的状态变化2021-03-2422:07:53.803089[WRN]Healthcheckfailed:1/3monsdown,quorumceph-node1,ceph-node3(MON_DOWN)2021-03-2422:07:53.814976[WRN]Healthdetail:HEALTH_WARN1/3monsdown,quorumceph-node1,ceph-node32021-03-2422:07:53.815005[WRN]MON_DOWN1/3monsdown,quorumceph-node1,ceph-node32021-03-2422:07:53.815015[WRN](rank2)addr[v2:192.168.168.202:3300/0,v1:192.168.168.202:6789/0]isdown(outofquorum)2021-03-2422:10:00.002661[WRN]overallHEALTH_WARN1/3monsdown,quorumceph-node1,ceph-node3cephhealthdetailHEALTH_WARN1/3monsdown,quorumceph-node1,ceph-node3MON_DOWN1/3monsdown,quorumceph-node1,(rank2)addr[v2:192.168.168.202:3300/0,v1:192.168.168.202:6789/0]isdown(outofquorum)[root@ceph-node2~]:6789[v2:192.168.168.202:3300/0,v1:192.168.168.202:6789/0][root@ceph-node2~]cephmdsstat1up:standby

查看msd的映射信息：cephmdsdump

cephmdsdump

删除一个mds节点：

查看cephosd运行状态：cephosdstat

[root@ceph-node1~]cephosddumpepoch286fsid5f844a3b-1257-4089-bff5-836def275bf0created2021-03-2118:53:15.026826modified2021-04-0515:58:21.728012flagssortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimitcrush_version7full____min_compat_clientjewelmin_compat_clientjewelrequire_osd_releasenautiluspool1'.'replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num32pgp_num32autoscale_modewarnlast_change18flagshashpspoolstripe_width0applicationrgwpool2''replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num32pgp_num32autoscale_modewarnlast_change20flagshashpspoolstripe_width0applicationrgwpool3''replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num32pgp_num32autoscale_modewarnlast_change22flagshashpspoolstripe_width0applicationrgwpool4''replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num32pgp_num32autoscale_modewarnlast_change24flagshashpspoolstripe_width0applicationrgwpool5'ceph-demo'replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num32pgp_num32autoscale_modewarnlast_change270lfor0/268/266flagshashpspool,selfmanaged_snapsstripe_width0applicationrbdremoved_snaps[1~3]pool7'cephfs_data'replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num32pgp_num32autoscale_modewarnlast_change286flagshashpspool,pool_snapsstripe_width0applicationcephfspool8'cephfs_metadata'replicatedsize3min_size2crush_rule0object_hashrjenkinspg_num32pgp_num32autoscale_modewarnlast_change277flagshashpspoolstripe_width0pg_autoscale_bias4pg_num_min16recovery_priority5applicationcephfsmax__from121up_thru274down_at120last_clean_interval[108,115)[v2:192.168.168.201:6802/1239,v1:192.168.168.201:6803/1239][v2:192.168.168.201:6804/1239,v1:192.168.168.201:6805/1239]exists,_from120up_thru274down_at116last_clean_interval[108,115)[v2:192.168.168.202:6800/1243,v1:192.168.168.202:6801/1243][v2:192.168.168.202:6802/1243,v1:192.168.168.202:6803/1243]exists,_from118up_thru274down_at117last_clean_interval[111,115)[v2:192.168.168.203:6800/1182,v1:192.168.168.203:6801/1182][v2:192.168.168.203:6802/1182,v1:192.168.168.203:6803/1182]exists,upb22b12d0-3751-49bb-ac27-133dd3793886

查看osd的目录树：cephosdtree

[root@ceph-node1~]默认最大是4个osd节点

置最大的osd的个数（当扩大osd节点的时候必须扩大这个值）：cephosdsetmaxosd10

cephosdsetmaxosd10

设置osdcrush的权重为1.0：cephosdcrushset{id}{weight}[{loc1}[{loc2}…]]

=

设置osd的权重：

把一个osd节点逐出集群：

_data8cephfs_metadata

[root@ceph-node1~]删除该pool[root@ceph-node1~]提示需要在配置文件中加入下一行配置[root@ceph-node1~]重启ceph-mon服务[root@ceph-node1~]重新删除就成功了[root@ceph-node1~]radosdfPOOL_NAMEUSEDOBJECTSCLONESCOPIESMISSING_ON_PRIMARYUNFOUNDDEGRADEDRD_OPSRDWR__data0B00000000B00B0B0Bcephfs__objects287total__avail26GiBtotal_space30GiB

[root@ceph-node1~]设置允许最大object数量为10[root@ceph-node1~]设置允许容量限制为10GB[root@ceph-node1~]限制存储数据大小为10M[root@ceph-node1~]查看配额[root@ceph-node1~]cephosdpoolset-quotacephfs_datamax_objects0set-quotamax_objects=0forpoolcephfs_data[root@ceph-node1~]cephosdpoolget-quotacephfs_dataquotasforpool'cephfs_data':maxobjects:N/Amaxbytes:N/A

cephosdpoolmksnapcephfs_datacephfs_data_snap_2021.04.05createdpoolcephfs_datasnapcephfs_data_snap_2021.04.05[root@ceph-node1~]和;为注释，ceph集群在启动的时候会按照顺序加载所有的conf配置文件。配置文件分为以下几大块配置。
global：全局配置。osd：osd专用配置，可以使用，来表示某一个OSD专用配置，N为osd的编号，如0、1、1等。mon：mon专用配置，也可以使用来为某一个monitor节点做专用配置，其中A为该节点的名称，ceph-monitor-2、ceph-monitor-1等。使用命令cephmondump可以获取节点的名称。client：客户端专用配置。
配置文件可以从多个地方进行顺序加载，如果冲突将使用最新加载的配置，其加载顺序为。
$CEPH_CONF环境变量-c指定的位置/etc/ceph/~/.ceph//
配置文件还可以使用一些元变量应用到配置文件，如。
$cluster：当前集群名。$type：当前服务类型。$id：进程的标识符。$host：守护进程所在的主机名。$name：值为$type.$id。
详细参数
[global]集群标识IDmonhost=10.0.1.1,10.0.1.2,10.0.1.3集群认证authservicerequired=cephx客户端认证osdpooldefaultsize=3PG处于degraded状态不影响其IO能力,min_size是一个PG能接受IO的最小副本数publicnetwork=10.0.1.0/24集群网络maxopenfiles=131072如果设置了该选项，Ceph会设置系统的maxopenfdsmoninitialmembers=node1,node2,node3默认值0.05默认值1默认值300格式化系统类型osdmaxwritesize=512OSD一次可写入的最大值(MB)osdclientmessagesizecap=2147483648客户端允许在内存中的最大数据(bytes)osddeepscrubstride=131072在DeepScrub时候允许读取的字节数(bytes)osdopthreads=16并发文件系统操作数osddiskthreads=4OSD密集型操作例如恢复和Scrubbing时的线程osdmapcachesize=1024保留OSDMap的缓存(MB)osdmapcacheblsize=128OSD进程在内存中的OSDMap缓存(MB)osdmountoptionsxfs="rw,noexec,nodev,noatime,nodiratime,nobarrier"CephOSDxfsMount选项osdrecoveryoppriority=2恢复操作优先级，取值1-63，值越高占用资源越高osdrecoverymaxactive=10同一时间内活跃的恢复请求数osdmaxbackfills=4一个OSD允许的最大backfills数osdminpglogentries=30000修建PGLog是保留的最大PGLog数osdmaxpglogentries=100000修建PGLog是保留的最大PGLog数osdmonheartbeatinterval=40OSDping一个monitor的时间间隔（默认30s）msdispatchthrottlebytes=1048576000等待派遣的最大消息数objecterinflightops=819200客户端流控，允许的最大未发送io请求数，超过阀值会堵塞应用io，为0表示不受限osdoplogthreshold=50一次显示多少操作的logosdcrushchooseleaftype=0CRUSH规则用到chooseleaf时的bucket的类型[client]rbdcache=trueRBD缓存rbdcachesize=335544320RBD缓存大小(bytes)rbdcachemaxdirty=134217728缓存为write-back时允许的最大dirty字节数(bytes)，如果为0，使用write-throughrbdcachemaxdirtyage=30在被刷新到存储盘前dirty数据存在缓存的时间(seconds)rbdcachewritethroughuntilflush=false该选项是为了兼容之前的virtio驱动，避免因为不发送flush请求，数据不回写默认值0每个chunk对象抽象为一个Object；librbd中以Object为单位来管理缓存，增大该值可以提升性能rbdcachetargetdirty=235544320开始执行回写过程的脏数据大小，不能超过rbd_cache_max_dirty