Bootstrap patroni with PGbackrest

Hello,

if i get it correct, to restore whole cluster from a backup the correct way is to remove patroni cluster, then restart patroni so it would bootstrap a new one, but i must specify custom bootstrap method. I am having trouble with this. My bootstrap section of patroni.yml looks like this:

bootstrap:
    dcs:
        ttl: 30
        loop_wait: 10
        retry_timeout: 10
        maximum_lag_on_failover: 1048576
        postgresql:
            use_pg_rewind: true
            recovery_conf:
                restore_command: 'pgbackrest --stanza=testpg archive-get %f %p'
    method: pgbackrest
    pgbackrest:
        command: pgbackrest --stanza=testpg  --process-max=4 restore
        keep_existing_recovery_conf: True
        no_params: True


    initdb:
    - encoding: UTF8
    - data-checksums

< pg_hba section here>.

So i remove cluster, clear PGDATA directory. Restart patroni and it starts to bootstrap, but fails with following messages:

Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: 2019-10-11 15:07:24,104 INFO: Lock owner: None; I am testpgdb1
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: 2019-10-11 15:07:24,127 INFO: trying to bootstrap a new cluster
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: 2019-10-11 15:07:24,129 INFO: Running custom bootstrap script: pgbackrest --stanza=testpg  --process-max=4 restore
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: ERROR: [031]: invalid option '--scope=postgres'
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: 2019-10-11 15:07:24,172 INFO: removing initialize key after failed attempt to bootstrap the cluster
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: 2019-10-11 15:07:24,278 INFO: Lock owner: None; I am testpgdb1
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: Traceback (most recent call last):
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/bin/patroni", line 11, in <module>
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: load_entry_point('patroni==1.6.0', 'console_scripts', 'patroni')()
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/lib/python2.7/site-packages/patroni/__init__.py", line 196, in main
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: return patroni_main()
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/lib/python2.7/site-packages/patroni/__init__.py", line 160, in patroni_main
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: patroni.run()
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/lib/python2.7/site-packages/patroni/__init__.py", line 125, in run
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: logger.info(self.ha.run_cycle())
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/lib/python2.7/site-packages/patroni/ha.py", line 1344, in run_cycle
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: info = self._run_cycle()
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/lib/python2.7/site-packages/patroni/ha.py", line 1253, in _run_cycle
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: return self.post_bootstrap()
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/lib/python2.7/site-packages/patroni/ha.py", line 1149, in post_bootstrap
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: self.cancel_initialization()
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: File "/usr/lib/python2.7/site-packages/patroni/ha.py", line 1144, in cancel_initialization
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: raise PatroniException('Failed to bootstrap cluster')
Oct 11 15:07:24 testpgdb1.foo.bar patroni[42256]: patroni.exceptions.PatroniException: 'Failed to bootstrap cluster'

As i got from the docs (https://patroni.readthedocs.io/en/latest/replica_bootstrap.html), patroni supplies the parameter, that causes problem, as pgbackrest doesn’t have one with such name. So i added “no_params: True” line, but no luck. What am i doing wrong?

P.S.: i’ve managed to successfully restore cluster doing following:

• stopped patroni on each node
• removed contents of old pgdata data dir on each node
• manualy restored from pgbackrest on one node
• after starting patroni, it became leader
• after starting patroni on other nodes the successfully restored themselves with pgbackrest and start ed follwing the leader.

so custom ‘create_replica_methods’ using pbackrest works correctly.

But i still can’t get bootstrapping master with pgbackrest to work.

[kolbasky]

@vjtm, glad you’ve figured out. @CyberDem0n, maybe you could implement “no params” option for bootstraping in some future release. That would be neater than writing wrappers for one-line command.

[guihecheng]

@amadhaka ok

1. To config the archive stuff, you need the following:

bootstrap:
    dcs:
        ttl: 30
        loop_wait: 10
        retry_timeout: 10
        postgresql:
            use_pg_rewind: true
            parameters:
              wal_level: replica
              archive_mode: 'on'
              archive_command: pgbackrest --stanza=patroni_stanza archive-push %p
              archive_timeout: 60
           recovery_conf:
             restore_command: pgbackrest --stanza=patroni_stanza archive-get %f %p
             standby_mode: 'on'

Then, there may be 2 operations you want to do:

1. Bootstrap
2. PITR

To do the 1st, you should follow the discussion in this thread above, because this is not the case I faced. To do the 2nd, you should have the following:

    # should be commented out
    #initdb:
    #- encoding: UTF8
    #- data-checksums
    
    method: pgbackrest
    pgbackrest:
        command: /pgbackrest_bootstrap.sh
        keep_existing_recovery_conf: True

And the script pgbackrest_bootstrap.sh

#!/bin/bash
sudo -u postgres mv /data/patroni.bak/* /data/patroni/

After you stop patroni, pgbackrest restore, and restart your patroni service, do the operation as suggested by the thread as I was suggested above. etcdctl rm /db/postgres/initialize Then you have a complete PITR.

I hope it helps and don’t mess up things, good luck 😉