Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Getting error "Failed to bootstrap cluster" when trying to restart patroni service

See original GitHub issue

Hey! I’m getting error “Failed to bootstrap cluster” on every patroni node. I can see, that he is failing to resolve host with % symbol. Where is that symbol coming from?

-- The start-up result is RESULT.
May 04 14:57:10 abiv-psql-a.prd.itarchive.home polkitd(authority=local)[924]: Unregistered Authentication Agent for unix-process:8148:218845 (system bus name :1.65, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,457 INFO: Failed to import patroni.dcs.consul
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,467 INFO: Selected new etcd server http:// http://192.168.0.10/.122:2379
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,468 WARNING: failed to resolve host %20192.168.0.10.122
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,469 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTT
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,469 WARNING: failed to resolve host %20192.168.0.10.122
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,469 ERROR: Failed to get list of machines from http:// 192.168.0.10.122:2379/v2: MaxRetryError("HTTPConnectionPool(host='%20192.168.0.10.122', port=2379): Max retries e
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,470 INFO: Retrying on http://192.168.0.10.120:2379
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,477 INFO: Lock owner: None; I am abiv-psql-a.test.itarchive.home
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:10,480 INFO: trying to bootstrap a new cluster
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: The files belonging to this database system will be owned by user "postgres".
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: This user must also own the server process.
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: The database cluster will be initialized with locale "en_US.UTF-8".
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: The default text search configuration will be set to "english".
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: Data page checksums are enabled.
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: creating directory /data/patroni ... ok
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: creating subdirectories ... ok
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: selecting default max_connections ... 100
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: selecting default shared_buffers ... 128MB
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: selecting default timezone ... Europe/Tallinn
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: selecting dynamic shared memory implementation ... posix
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: creating configuration files ... ok
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: running bootstrap script ... ok
May 04 14:57:10 abiv-psql-a.prd.itarchive.home patroni[8160]: performing post-bootstrap initialization ... ok
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: syncing data to disk ... ok
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: WARNING: enabling "trust" authentication for local connections
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: You can change this by editing pg_hba.conf or using the option -A, or
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: --auth-local and --auth-host, the next time you run initdb.
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: Success. You can now start the database server using:
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]:     /usr/lib/postgresql/10/bin/pg_ctl -D /data/patroni -l logfile start
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:11,173 INFO: postmaster pid=8187
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:11.179 EEST [8187] LOG:  listening on IPv4 address "192.168.0.10.123", port 5432
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:11.180 EEST [8187] FATAL:  lock file "/var/run/postgresql/.s.PGSQL.5432.lock" already exists
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:11.180 EEST [8187] HINT:  Is another postmaster (PID 943) using socket file "/var/run/postgresql/.s.PGSQL.5432"?
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:11.180 EEST [8187] LOG:  database system is shut down
May 04 14:57:11 abiv-psql-a.prd.itarchive.home patroni[8160]: 192.168.0.10.123:5432 - no response
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:12,185 ERROR: postmaster is not running
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:12,190 INFO: removing initialize key after failed attempt to bootstrap the cluster
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:12,192 INFO: renaming data directory to /data/patroni_2020-05-04-14-57-12
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]: 2020-05-04 14:57:12,477 INFO: Lock owner: None; I am abiv-psql-a.test.itarchive.home
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]: Traceback (most recent call last):
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/bin/patroni", line 11, in <module>
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     load_entry_point('patroni==1.4.2', 'console_scripts', 'patroni')()
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/lib/python3/dist-packages/patroni/__init__.py", line 174, in main
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     return patroni_main()
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/lib/python3/dist-packages/patroni/__init__.py", line 143, in patroni_main
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     patroni.run()
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/lib/python3/dist-packages/patroni/__init__.py", line 114, in run
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     logger.info(self.ha.run_cycle())
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/lib/python3/dist-packages/patroni/ha.py", line 1134, in run_cycle
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     info = self._run_cycle()
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/lib/python3/dist-packages/patroni/ha.py", line 1058, in _run_cycle
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     return self.post_bootstrap()
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/lib/python3/dist-packages/patroni/ha.py", line 962, in post_bootstrap
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     self.cancel_initialization()
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:   File "/usr/lib/python3/dist-packages/patroni/ha.py", line 957, in cancel_initialization
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]:     raise PatroniException('Failed to bootstrap cluster')
May 04 14:57:12 abiv-psql-a.prd.itarchive.home patroni[8160]: patroni.exceptions.PatroniException: 'Failed to bootstrap cluster'
May 04 14:57:12 abiv-psql-a.prd.itarchive.home systemd[1]: patroni.service: Main process exited, code=exited, status=1/FAILURE

Here’s my conf

scope: postgres
name: abiv-psql-a.prd.itarchive.home

restapi:
    listen: 192.168.0.10.123:8008
    connect_address: 192.168.0.10.123:8008

etcd:
    hosts: 192.168.0.120:2379, 192.168.0.121:2379, 192.168.0.122:2379

bootstrap:
    dcs:
        ttl: 30
        loop_wait: 10
        retry_timeout: 10
        maximum_lag_on_failover: 1048576
        postgresql:
            use_pg_rewind: true
            use_slots: true
            parameters:
                    wal_level: replica
                    hot_standby: "on"
                    wal_keep_segments: 8
                    max_wal_senders: 5
                    max_replication_slots: 5
                    checkpoint_timeout: 30
                    synchronous_commit: "on"
                    synchronous_standby_names: "*"

    initdb:
    - encoding: UTF8
    - data-checksums

    pg_hba:
    - host replication replicator 127.0.0.1/32 md5
    - host replication replicator 192.168.0.10/24 md5
    - host replication replicator 192.168.0.11/24 md5
    - host replication replicator 192.168.0.12/24 md5
    - host replication replicator 192.168.0.13/24 md5
    - host all all 0.0.0.0/0 md5

    users:
        admin:
            password: admin
            options:
                - createrole
                - createdb

postgresql:
    listen: 192.168.0.10:5432
    connect_address: 192.168.0.10:5432
    data_dir: /data/patroni
    pgpass: /tmp/pgpass
    authentication:
        replication:
            username: replicator
            password: replica_pass
        superuser:
            username: postgres
            password: postgres_pass
    parameters:
        unix_socket_directories: '/var/run/postgresql/'

tags:
    nofailover: false
    noloadbalance: false
    clonefrom: false
    nosync: false