HDP-3.0 with postgres: Schema registry (HDF) startup fails on centos-7 (latest) because default postgresql (v9.2) is too old

from the Schema Registry Server Start log:

"migrate" option failed : org.flywaydb.core.internal.exception.FlywayEnterpriseUpgradeRequiredException: Flyway Enterprise or PostgreSQL upgrade required: PostgreSQL 9.2 is past regular support by PostgreSQL and no longer supported by Flyway Community Edition and Flyway Pro Edition, but still supported by Flyway Enterprise Edition.

The tested installation contains a (almost)full HDP-3.0.1 stack, plus NIFI and the 2 registries (REGISTRY_SERVER being the SchemaRegistry):

  - host_group: "hdp-master"
    services:
    - REGISTRY_SERVER
    - NIFI_REGISTRY_MASTER ...

Solution options

• using an external DB (to be installed+prepared outside this role), by configuring database_options.external_hostname ** con: DBs + Users needs to be populated (one could re-use the the existing database_options var tough)
• consider using an external role to install/configure the databases (which often have such flexibility builtin), like: galaxy.ansible.com/geerlingguy/postgresql
• added role support to specify an optional/custom pgdg repo ** but I’m not sure howto to best support overriding the (included) postgres_* vars depending on the choosen version ** I will attach a patch, how I made it work, but where I ‘hardcoded’ existing vars to use 9.6. Of course different ways to make this configurable… but first I went @alexandruanghel 's opinion if this could be a viable contribution

[alexandruanghel]

Always use https://supportmatrix.hortonworks.com to check for any compatibility. There is a large matrix of what’s supported or not so quite complex to do this in Ansible (at least in these playbooks, which or more “universal”).

You’re right in your assumptions, some versions might work (outside of that matrix), and for many things the difference between supported and not supported is “we haven’t tested / certified that” rather than “it won’t work”. Which is also a reason why I’ve been relaxed a bit about versions in the database role/vars folder.

For the moment, the static way of defining the version would be preferred: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/roles/database/vars/redhat-7.yml#L9 Obviously that can be moved to the main group_vars but I’d like to keep that as simple as possible. It also doesn’t help that some older db versions are not available in newer OS-es (like MySQL 5.6 in Ubuntu 16). And if really needed to overwrite that, you can use --extra-vars option.

To do these versions dynamically you would need to create some sort of internal matrix / variable file in Ansible, which will also need to be maintained, etc.

Anyway, the point of these playbooks is to provide a baseline. You can of course use other roles and prepare the database outside these playbooks. As long as database_options.external_hostname is set and the username/passwords are correct and the database is created in advance (essentially what the database role does) then the blueprint should work fine. This would actually be required if you use an external cloud service like RDS. And the reason why the entire database role is skipped if database_options.external_hostname is not empty: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/prepare_nodes.yml#L17

Another expectation that I have is people would customize this as much as possible for their “production”. Of course things can be contributed back if they are “generic” but it’s sometimes more efficient when you’re only using 1 flavour of OS and 1-2 versions of Ambari/HDP. And by customizing I also mean changing and maintaining the few variable files that might be required for that environment, including matching the Ambari version with HDP and the database version, and maybe customizing the repo url of the db: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/roles/database/vars/redhat-7.yml#L10

[lhoss]

can you integrate please this change: de71ccf

I just rebased the branch to upstream master 👍