Installing/Upgrade to Splunk v8.0 with Splunkenizer

Describe the bug The installation of Splunk v8.0.0 with Splunkenizer was unsuccessful.

1.- When we tried a fresh installation with ansible/deploy_site.yml, the playbook finished successfully however the Indexers were not registered.

2.- When we tried an upgrade, with ansible/ugrade_splunk.yml, the playbook failed because it couldn’t restart splunkd.

Expected behavior In Case 1 we expected this output:

[splunk@splp0cm000 ~]$ splunk show cluster-status

 Replication factor met
 Search factor met
 All data is searchable
 Indexing Ready YES

 splp0ix000.splunk.sbb.ch 	 024D3D82-2928-4DE0-8B9C-63265B58A66C	 site1
	 Searchable YES
	 Status  Up
	 Bucket Count=124

 splp0ix002.splunk.sbb.ch 	 6DA75D2C-6156-4FEC-B79F-63D18C3AC313	 site1
	 Searchable YES
	 Status  Up
	 Bucket Count=128

 splp0ix003.splunk.sbb.ch 	 92E8BE2A-1AD0-4B43-B392-4323E236F497	 site2
	 Searchable YES
	 Status  Up
	 Bucket Count=126

 splp0ix001.splunk.sbb.ch 	 B5E84637-93EA-4D0C-91B1-78B496326A52	 site2
	 Searchable YES
	 Status  Up
	 Bucket Count=132

However we receive this other one:

[splunk@splp0cm000 ~]$ splunk show cluster-status

 Replication factor not met
 Search factor not met
 All data is not searchable
 Indexing Ready NO

In Case 2, the playbook is interrupted with this error on all target servers:

...
TASK [splunk_software : start splunk] 
********************************************
2019-10-29 16:06:34,951 p=828 u=aleoliva |  fatal: [splp0cm000.splunk.sbb.ch]: FAILED! => {"changed": false, "msg": "Unable to start service splunk: Job for splunk.service failed because the control process exited with error code. See \"systemctl status splunk.service\" and \"journalctl -xe\" for details.\n"}
...

Inside the servers, we can get this information:

[linux@splp0cm000 ~]$ sudo systemctl status splunk
· splunk.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
   Loaded: loaded (/etc/systemd/system/splunk.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2019-10-30 09:32:10 CET; 1min 25s ago
  Process: 30167 ExecStartPost=/bin/bash -c chown -R 1001:1001 /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
  Process: 30165 ExecStartPost=/bin/bash -c chown -R 1001:1001 /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
  Process: 30164 ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd --accept-license --answer-yes --no-prompt (code=exited, status=1/FAILURE)
 Main PID: 30164 (code=exited, status=1/FAILURE)

Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Unit splunk.service entered failed state.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: splunk.service failed.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: splunk.service holdoff time over, scheduling restart.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: start request repeated too quickly for splunk.service
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Unit splunk.service entered failed state.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: splunk.service failed.

[linux@splp0cm000 ~]$ sudo journalctl -xe
-- Unit splunk.service has failed.
--
-- The result is failed.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Unit splunk.service entered failed state.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: splunk.service failed.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: splunk.service holdoff time over, scheduling restart.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.
-- Subject: Unit splunk.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit splunk.service has finished shutting down.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: start request repeated too quickly for splunk.service
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
-- Subject: Unit splunk.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit splunk.service has failed.
--
-- The result is failed.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: Unit splunk.service entered failed state.
Oct 30 09:32:10 splp0cm000.splunk.sbb.ch systemd[1]: splunk.service failed.
Oct 30 09:32:15 splp0cm000.splunk.sbb.ch sudo[30170]:    linux : TTY=pts/0 ; PWD=/home/linux ; USER=root ; COMMAND=/bin/journalctl -xe
Oct 30 09:32:15 splp0cm000.splunk.sbb.ch sudo[30170]: pam_unix(sudo:session): session opened for user root by linux(uid=0)

Workaround The Case 2 can be solved and get Splunk running without issues, after running these commands:

$ splunk version --accept-license --answer-yes
$ sudo /opt/splunk/bin/splunk disable boot-start
$ sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 1 -systemd-unit-file-name splunk
$ sudo systemctl start splunk

Desktop (please complete the following information):

• OS: CentOS Linux release 7.6.1810 (Core)
• Ansible Version [ansible --version]:

ansible 2.8.4
  config file = /home/aleoliva/Vagrant/Splunkenizer/ansible.cfg
  configured module search path = ['/home/aleoliva/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ansible
  executable location = /opt/rh/rh-python36/root/usr/bin/ansible
  python version = 3.6.3 (default, May 31 2019, 13:05:43) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

Additional context For Case 1 next logs have been attached:

• ansible-splunk8-deploy_site.log.gz -> Logs of ansible-playbook in verbose mode.
• splunkd-splp0cm000.log.gz -> Splunkd logs on ClusterMater
• splunkd-splp0ix000.log.gz -> Splunkd logs on one Indexer of the cluster

For Case 2 next logs have been attached:

• ansible-splunk8-ugrade_splunk.log.gz -> Logs of ansible-playbook in verbose mode.

[aleoliva]

After downloading the new org_all_indexes, a fresh installation of Splunk v8.0 looks successful. The point 1 looks solved.

[aleoliva]

The error, reported on previous #issuecomment-552940073, never reappeared.

We consider that it was an isolate case and we suggest to close this issue, since works good with version 8.0 now