问题解决记录 - KRBD Map失败

来源：互联网发布：曙光大数据学院是几本编辑：程序博客网时间：2024/06/05 04:04

问题：

centos6.5，升级kernel到3.10.84

rbd create test1无报错

rbd info test1无报错显示正常

rbd map test1出现问题

开始是没有响应，等待一段时间后，出现：

rbd: sysfs write failed

rbd: map failed: (5) Input/output error

解决过程：

应该是内核rbd模块无法读取rbd信息所致。

怀疑可能是image 数据损坏，重新建了两个image，都出现一样的问题，排除这个可能。

查看dmesg发现有错误输出：

mon1 xxxxxxx:6789 feature set mismatch, my XXXXXX < server's XXXXXX, missing 1000000000000

经过一番搜索，在下面这个文章找到了解药：

http://cephnotes.ksperis.com/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client/

对照feature表，应该是当前内核版本缺少 CEPH_FEATURE_CRUSH_V4 **** 这个特性支持。

而这个特性在4.1版本内核才被支持，升级内核不现实，只能在服务端关闭这个特性。

通过ceph osd crush dump看到：

"tunables": {
"choose_local_tries": 0,
"choose_local_fallback_tries": 0,
"choose_total_tries": 50,
"chooseleaf_descend_once": 1,
"chooseleaf_vary_r": 0,
"straw_calc_version": 1,
"allowed_bucket_algs": 22,
"profile": "unknown",
"optimal_tunables": 0,
"legacy_tunables": 0,
"require_feature_tunables": 1,
"require_feature_tunables2": 1,
"require_feature_tunables3": 0,
"has_v2_rules": 0,
"has_v3_rules": 0,
"has_v4_buckets": 1
}

最后这个特性显得比较诡异，为什么被打开了呢？

尝试把crushmap导出，用crushtool反编译成文本，加上一行：

tunbale has_v4_buckets 0

重新编译发现不能编译，提示这并不是一个合法的配置项。

于是连蒙带猜，认为这个配置项不是手工配置的，而是由其它配置自动产生的。

straw2是一个跟crush相关的新特性，很有可能与之有关。

检查crushmap，发现其中的bucket的alg全部配置成了straw2(不知道哪个哥们干的)

于是将其改回straw，重新导入crushmap。

发现has_v4_buckets特性的值已经变为0了。

尝试重新map image: rbd map test1

终于成功了~~

收获：

以后用KRBD时，可以通过查表清楚的知道哪些版本支持什么样的特性啦。

为防止老外删除文章，我还是把表格拷贝过来比较保险：

FeatureBITOCT3.83.93.103.143.153.184.1CEPH_FEATURE_UID01 CEPH_FEATURE_NOSRCADDR12RRRRRRRCEPH_FEATURE_MONCLOCKCHECK24 CEPH_FEATURE_FLOCK38 CEPH_FEATURE_SUBSCRIBE2410 CEPH_FEATURE_MONNAMES520 CEPH_FEATURE_RECONNECT_SEQ640 -R-RRRRCEPH_FEATURE_DIRLAYOUTHASH780 CEPH_FEATURE_OBJECTLOCATOR8100 CEPH_FEATURE_PGID649200 RRRRRRCEPH_FEATURE_INCSUBOSDMAP10400 CEPH_FEATURE_PGPOOL311800 RRRRRRCEPH_FEATURE_OSDREPLYMUX121000 CEPH_FEATURE_OSDENC132000 RRRRRRCEPH_FEATURE_OMAP144000 CEPH_FEATURE_MONENC158000 CEPH_FEATURE_QUERY_T1610000 CEPH_FEATURE_INDEP_PG_MAP1720000 CEPH_FEATURE_CRUSH_TUNABLES1840000SSSSSSSCEPH_FEATURE_CHUNKY_SCRUB1980000 CEPH_FEATURE_MON_NULLROUTE20100000 CEPH_FEATURE_MON_GV21200000 CEPH_FEATURE_BACKFILL_RESERVATION22400000 CEPH_FEATURE_MSG_AUTH23800000 -S-SCEPH_FEATURE_RECOVERY_RESERVATION241000000 CEPH_FEATURE_CRUSH_TUNABLES2252000000 SSSSSSCEPH_FEATURE_CREATEPOOLID264000000 CEPH_FEATURE_REPLY_CREATE_INODE278000000 SSSSSSCEPH_FEATURE_OSD_HBMSGS2810000000 CEPH_FEATURE_MDSENC2920000000 CEPH_FEATURE_OSDHASHPSPOOL3040000000 SSSSSSCEPH_FEATURE_MON_SINGLE_PAXOS3180000000 CEPH_FEATURE_OSD_SNAPMAPPER32100000000 CEPH_FEATURE_MON_SCRUB33200000000 CEPH_FEATURE_OSD_PACKED_RECOVERY34400000000 CEPH_FEATURE_OSD_CACHEPOOL35800000000 -S-SSSCEPH_FEATURE_CRUSH_V2361000000000 -S-SSSCEPH_FEATURE_EXPORT_PEER372000000000 -S-SSSCEPH_FEATURE_OSD_ERASURE_CODES***384000000000 CEPH_FEATURE_OSD_TMAP2OMAP38*4000000000 CEPH_FEATURE_OSDMAP_ENC398000000000 -S-SSCEPH_FEATURE_MDS_INLINE_DATA4010000000000 CEPH_FEATURE_CRUSH_TUNABLES34120000000000 -S-SSCEPH_FEATURE_OSD_PRIMARY_AFFINITY41*20000000000 -S-SSCEPH_FEATURE_MSGR_KEEPALIVE24240000000000 CEPH_FEATURE_OSD_POOLRESEND4380000000000 CEPH_FEATURE_ERASURE_CODE_PLUGINS_V244100000000000 CEPH_FEATURE_OSD_SET_ALLOC_HINT45200000000000 CEPH_FEATURE_OSD_FADVISE_FLAGS46400000000000 CEPH_FEATURE_OSD_REPOP46*400000000000 CEPH_FEATURE_OSD_OBJECT_DIGEST46*400000000000 CEPH_FEATURE_OSD_TRANSACTION_MAY_LAY46*400000000000 CEPH_FEATURE_MDS_QUOTA47800000000000 CEPH_FEATURE_CRUSH_V4 ****481000000000000 -S-CEPH_FEATURE_OSD_MIN_SIZE_RECOVERY492000000000000 CEPH_FEATURE_OSD_PROXY_FEATURES49*4000000000000

0 0