OCP 3.11 deployment fails while creating nodes with error: "Could not find csr for nodes"

OCP 3.11 deployment fails while creating nodes with error: “Could not find csr for nodes”

OCP environment with 3 Master , 3 Node and 3 app instances

Reference link

https://github.com/openshift/openshift-ansible/tree/master/playbooks/aws Error encountered after the deployment of master nodes:

Could not find csr for nodes: ip-172-31-39-6.ec2.internal, ip-172-31-27-65.ec2.internal, ip-172-31-57-248.ec2.internal, ip-172-31-33-50.ec2.internal, ip-172-31-31-76.ec2.internal

Version 3.11

Ansible Version

ansible --version ansible 2.6.5

Git describe

git describe openshift-ansible-3.11.32-1-2-g263b6f5

##### Steps To Reproduce### Once the inventory and the provisioning_vars.yml file has been updated with the correct settings for the desired AWS account then we are ready to build an AMI. $ ansible-playbook -i inventory.yml build_ami.yml -e @provisioning_vars.yml 2. Now that we have created an AMI for our Openshift installation, there are two ways to use the AMI.

In the default behavior, the AMI id will be found and used in the last created fashion. The openshift_aws_ami option can be specified. This will allow the user to override the behavior of the role and use a custom AMI specified in the openshift_aws_ami variable. We are now ready to provision and install the cluster. This can be accomplished by calling all of the following steps at once or one-by-one. The all in one can be called like this:

$ ansible-playbook -i inventory.yml provision_install.yml -e @provisioning_vars.yml

Expected Results Once this playbook completes, it should create the compute and infra node scale groups. These nodes will attempt to register themselves to the cluster.

All the nodes should be up and running as expected

##### Observed Results######## After the master instances are up and running , in the node deployment phase , playbook fails stating the below error:

Task:     Approve node certificates when bootstrapping
     Message:  Could not find csr for nodes: ip-172-31-39-6.ec2.internal, ip-172-31-27-65.ec2.internal, ip-172-31-57-248.ec2.internal, ip-172-31-33-50.ec2.internal, ip-172-31-31-76.ec2.internal

##### Additional Information######

• cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo)

• Inventory file information is as below:

[OSEv3:children]
masters
etcd
nodes

[OSEv3:vars]
debug_level=2
#osm_etcd_image=registry.access.redhat.com/rhel7/etcd:3.2.22
ansible_user=ec2-user
ansible_become=yes
openshift_deployment_type=openshift-enterprise
openshift_release='3.11'
openshift_master_api_port=443
openshift_master_console_port=443
openshift_portal_net=172.30.0.0/16
os_sdn_network_plugin_name='redhat/openshift-ovs-networkpolicy'
openshift_master_cluster_method=native
openshift_node_local_quota_per_fsgroup=512Mi
osm_use_cockpit=true
openshift_hostname_check=false
openshift_builddefaults_nodeselectors="{'node-role.kubernetes.io/infra': 'true'}"
openshift_hosted_router_selector='node-role.kubernetes.io/infra=true'
openshift_hosted_router_replicas=2
openshift_install_examples=true
openshift_examples_modify_imagestreams=true
openshift_master_bootstrap_auto_approve=True
oreg_auth_user='---------'
oreg_auth_password='----------'
openshift_disable_check=package_version,memory_availability,disk_availability,docker_image_availability

openshift_master_default_subdomain=apps.shegde.sysdeseng.com
openshift_master_cluster_hostname=internal.api.shegde.sysdeseng.com
openshift_master_cluster_public_hostname=api.shegde.sysdeseng.com

#CLoud Provider 

openshift_cloudprovider_kind=aws
openshift_clusterid=shegde
openshift_cloudprovider_aws_access_key=--------------------------
openshift_cloudprovider_aws_secret_key=---------------------------
  
# Hosted registry
openshift_hosted_manage_registry=true
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_storage_provider=s3
openshift_hosted_registry_storage_s3_accesskey=**************
openshift_hosted_registry_storage_s3_secretkey=***************
openshift_hosted_registry_storage_s3_bucket=*******
openshift_hosted_registry_storage_s3_region=us-east-1
openshift_hosted_registry_storage_s3_chunksize=26214400
openshift_hosted_registry_storage_s3_rootdirectory=/registry
openshift_hosted_registry_pullthrough=true
openshift_hosted_registry_acceptschema2=true
openshift_hosted_registry_enforcequota=true
openshift_hosted_registry_replicas=2
#
# # Aggregated logging
openshift_logging_install_logging=True
openshift_logging_storage_kind=dynamic
openshift_logging_storage_volume_size=25Gi
openshift_logging_es_cluster_size=3
openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"}

#Metrics
openshift_metrics_install_metrics=True
openshift_metrics_storage_kind=dynamic
openshift_metrics_storage_volume_size=25Gi
openshift_metrics_hawkular_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_cassandra_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_heapster_nodeselector={"node-role.kubernetes.io/infra": "true"}

openshift_enable_service_catalog=True
template_service_broker_install=True
#    
#    #    ################################################################################
#    #    # cluster specific settings maybe be placed here
#

[masters]

[masters:vars]
openshift_node_group_name=node-config-master
openshift_node_group_name=node-config-master
openshift_node_group_name=node-config-master

[etcd]

[etcd:children]
masters

[infra]
[infra:vars]
openshift_node_group_name=node-config-infra
openshift_node_group_name=node-config-infra
openshift_node_group_name=node-config-infra

[nodes]

[nodes:children]
masters

[nodes:vars]
openshift_node_group_name=node-config-compute

oc get csr

[root@ip-172-31-59-23 ~]# oc get csr
NAME        AGE       REQUESTOR                                   CONDITION
csr-55gjb   1h        system:admin                                Approved,Issued
csr-6zzkz   1h        system:admin                                Approved,Issued
csr-8rls5   1h        system:node:ip-172-31-59-23.ec2.internal    Approved,Issued
csr-c59t9   1h        system:node:ip-172-31-29-29.ec2.internal    Approved,Issued
csr-f99ms   1h        system:admin                                Approved,Issued
csr-mc52v   1h        system:node:ip-172-31-29-29.ec2.internal    Approved,Issued
csr-phlrd   1h        system:node:ip-172-31-37-246.ec2.internal   Approved,Issued
csr-td6kc   1h        system:admin                                Approved,Issued
csr-v7872   1h        system:admin                                Approved,Issued
[root@ip-172-31-59-23 ~]# 

systemctl status atomic-openshift-node
● atomic-openshift-node.service - OpenShift Node
   Loaded: loaded (/etc/systemd/system/atomic-openshift-node.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/atomic-openshift-node.service.d
           └─override.conf
   Active: active (running) since Mon 2018-11-05 20:45:40 UTC; 40min ago
     Docs: https://github.com/openshift/origin
 Main PID: 14959 (hyperkube)
   Memory: 63.2M
   CGroup: /system.slice/atomic-openshift-node.service
           └─14959 /usr/bin/hyperkube kubelet --v=2 --address=0.0.0.0 --allow-privileged=true --an...

Nov 05 21:25:31 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:25:31.462830  ...
Nov 05 21:25:40 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:25:40.686218  ...
Nov 05 21:25:41 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:25:41.463068  ...
Nov 05 21:25:41 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:25:41.466850  ...
Nov 05 21:25:51 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:25:51.467081  ...
Nov 05 21:25:51 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:25:51.471016  ...
Nov 05 21:26:01 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:26:01.471260  ...
Nov 05 21:26:01 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:26:01.476492  ...
Nov 05 21:26:11 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:26:11.476689  ...
Nov 05 21:26:11 ip-172-31-59-23.ec2.internal atomic-openshift-node[14959]: I1105 21:26:11.480569  ...
Hint: Some lines were ellipsized, use -l to show in full.

oc version

[root@ip-172-31-59-23 ~]# oc version
oc v3.11.16
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://internal.api.shegde.sysdeseng.com:443
openshift v3.11.16
kubernetes v1.11.0+d4cacc0

oc get nodes

[root@ip-172-31-59-23 ~]# oc get nodes
NAME                            STATUS    ROLES     AGE       VERSION
ip-172-31-29-29.ec2.internal    Ready     master    1h        v1.11.0+d4cacc0
ip-172-31-37-246.ec2.internal   Ready     master    1h        v1.11.0+d4cacc0
ip-172-31-59-23.ec2.internal    Ready     master    1h        v1.11.0+d4cacc0

[kahootali]

I was able to fix this with a workaround that i exposed my nodes(the ones for which it was giving csr error) with a load balancer and gave them a public ip

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.