Bluestore filestore + non-collocated scenario failed

Hi,

EDIT: this issue is not caused by the scenario non-collocated. It is caused by a missing udev rule. Read the comment https://github.com/ceph/ceph-ansible/issues/2209#issuecomment-350828925

We tried to deploy Ceph Luminous in 6 “Ubuntu Xenial” servers with:

• ceph-ansible v3.0.14
• scenario: non-collocated
• filestore: bluestore

But the following command failed because directories like ‘/var/lib/ceph/tmp/mnt.XXXXXX’ were not found:

/usr/bin/ceph-osd --cluster ceph --mkfs -i 0 --monmap /var/lib/ceph/tmp/mnt.LA4YuK/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.LA4YuK --osd-uuid 9f87c093-5566-4c72-a805-8c44eac0ef27 --setuser ceph --setgroup ceph

(I created the directory ‘/var/lib/ceph/tmp/mnt.LA4YuK’ manually, executed the command “ceph-osd --cluster ceph …etc.” and the command returned 0 = success)

One of the Ansible tasks output:

{
	"changed": false,
	"cmd": [
		"ceph-disk",
		"activate",
		"/dev/sdb1"
	],
	"delta": "0:00:00.686214",
	"end": "2017-11-21 22:49:23.881782",
	"failed": true,
	"item": "/dev/sdb",
	"msg": "non-zero return code",
	"rc": 1,
	"start": "2017-11-21 22:49:23.195568",
	"stderr": "got monmap epoch 1\n2017-11-21 22:49:23.781382 7f179d85be00 -1 bluestore(/var/lib/ceph/tmp/mnt.LA4YuK) _open_db /var/lib/ceph/tmp/mnt.LA4YuK/block.db link target doesn't exist\n2017-11-21 22:49:23.821937 7f179d85be00 -1 bluestore(/var/lib/ceph/tmp/mnt.LA4YuK) mkfs failed, (2) No such file or directory\n2017-11-21 22:49:23.821963 7f179d85be00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (2) No such file or directory\n2017-11-21 22:49:23.822065 7f179d85be00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.LA4YuK: (2) No such file or directory\u001b[0m\nmount_activate: Failed to activate\nTraceback (most recent call last):\n  File \"/usr/sbin/ceph-disk\", line 9, in <module>\n    load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 5704, in run\n    main(sys.argv[1:])\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 5657, in main\n    main_catch(args.func, args)\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 5682, in main_catch\n    func(args)\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3759, in main_activate\n    reactivate=args.reactivate,\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3522, in mount_activate\n    (osd_id, cluster) = activate(path, activate_key_template, init)\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3699, in activate\n    keyring=keyring,\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3151, in mkfs\n    '--setgroup', get_ceph_group(),\n  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 558, in command_check_call\n    return subprocess.check_call(arguments)\n  File \"/usr/lib/python2.7/subprocess.py\", line 541, in check_call\n    raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '-i', u'0', '--monmap', '/var/lib/ceph/tmp/mnt.LA4YuK/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.LA4YuK', '--osd-uuid', u'9f87c093-5566-4c72-a805-8c44eac0ef27', '--setuser', 'ceph', '--setgroup', 'ceph']' returned non-zero exit status 1",
	"stderr_lines": [
		"got monmap epoch 1",
		"2017-11-21 22:49:23.781382 7f179d85be00 -1 bluestore(/var/lib/ceph/tmp/mnt.LA4YuK) _open_db /var/lib/ceph/tmp/mnt.LA4YuK/block.db link target doesn't exist",
		"2017-11-21 22:49:23.821937 7f179d85be00 -1 bluestore(/var/lib/ceph/tmp/mnt.LA4YuK) mkfs failed, (2) No such file or directory",
		"2017-11-21 22:49:23.821963 7f179d85be00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (2) No such file or directory",
		"2017-11-21 22:49:23.822065 7f179d85be00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.LA4YuK: (2) No such file or directory\u001b[0m",
		"mount_activate: Failed to activate",
		"Traceback (most recent call last):",
		"  File \"/usr/sbin/ceph-disk\", line 9, in <module>",
		"    load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 5704, in run",
		"    main(sys.argv[1:])",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 5657, in main",
		"    main_catch(args.func, args)",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 5682, in main_catch",
		"    func(args)",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3759, in main_activate",
		"    reactivate=args.reactivate,",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3522, in mount_activate",
		"    (osd_id, cluster) = activate(path, activate_key_template, init)",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3699, in activate",
		"    keyring=keyring,",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 3151, in mkfs",
		"    '--setgroup', get_ceph_group(),",
		"  File \"/usr/lib/python2.7/dist-packages/ceph_disk/main.py\", line 558, in command_check_call",
		"    return subprocess.check_call(arguments)",
		"  File \"/usr/lib/python2.7/subprocess.py\", line 541, in check_call",
		"    raise CalledProcessError(retcode, cmd)",
		"subprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '-i', u'0', '--monmap', '/var/lib/ceph/tmp/mnt.LA4YuK/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.LA4YuK', '--osd-uuid', u'9f87c093-5566-4c72-a805-8c44eac0ef27', '--setuser', 'ceph', '--setgroup', 'ceph']' returned non-zero exit status 1"
	],
	"stdout": "",
	"stdout_lines": []
}

osd.yml configuration file:

osd_scenario: non-collocated
devices:
  - /dev/sdb
  - /dev/sdc
  - /dev/sdd
  - /dev/sde
  - /dev/sdf
  - /dev/sdg
  - /dev/sdh
  - /dev/sdi
  - /dev/sdj
  - /dev/sdk
  - /dev/sdl
  - /dev/sdm
  - /dev/sdn
  - /dev/sdo
  - /dev/sdp
  - /dev/sdq
  - /dev/sdr
  - /dev/sds
  - /dev/sdt
  - /dev/sdu
  - /dev/sdv
  - /dev/sdw
  - /dev/sdx
  - /dev/sdy

dedicated_devices:
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssda
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb
  - /dev/rssdb

(The devices /dev/rssda and /dev/rssdb are PCIe SSD)

FYI: the same ceph-ansible configuration worked perfectly with scenario=collocated. It looks like it is a non-collocated scenario issue.

Is it a ceph-ansible bug? Are you able to reproduce the same issue in your test environment? Any ideas?

Thank you in advance.

[guits]

@Asher256 Our CI cover this scenario and doesn’t hit this issue though. I will try to reproduce this in my testing env.

[alfredodeza]

@Asher256 you are experiencing known issues with udev and ceph-disk. This is why we have worked on introducing new tooling that will not rely on udev. The new tool (ceph-volume) will handle this better, and ceph-ansible is capable of deploying OSDs using it.

I am closing this as this isn’t a ceph-ansible issue, but rather a well-known ceph-disk limitation