Function Pod doesn't come up when requirements.txt is used for dependencies

If we use requirements.txt with dependencies defined in it, ‘serverless deploy’ gets stuck with pod at Init state.

Dependency mentioned in requirements.txt is not even used in my handler.py. And if I just delete requirements.txt file from service folder, everything starts working, function gets deployed without any issues.

mongo-connect-7dfb4868f-crt96 0/1 Init:CrashLoopBackOff 4 5m

handler.py

def mongoConnect():
    return 'Connected to MongoDB'

requirements.txt

pymongo==2.7

serverless.yml

service: mongo-utils

provider:
  name: kubeless
  runtime: python2.7

plugins:
  - serverless-python-requirements
  - serverless-kubeless
functions:
  mongo-connect:
    handler: handler.mongoConnect

custom:
  pythonRequirements:
    pythonBin: C:\Python27\python.exe
    invalidateCaches: true

Serverless Commands:

C:\Users\myusername\workspace\devopsinsight\functions\mongo-utils>sls deploy
Serverless: Installing required Python packages with C:\Python27\python.exe...
Serverless: Linking required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Unlinking required Python packages...
Serverless: Deploying function mongo-connect...

Pod logs:

C:\>kubectl logs mongo-connect-7dfb4868f-crt96
Error from server (BadRequest): container "mongo-connect" in pod "mongo-connect-7dfb4868f-crt96" is waiting to start: PodInitializing

Function pod describe:

C:\>kubectl describe po mongo-connect-7dfb4868f-crt96
Name:           mongo-connect-7dfb4868f-crt96
Namespace:      default
Node:           cmbu-devops-services-kube-node2/10.197.10.228
Start Time:     Thu, 28 Dec 2017 21:38:38 +0530
Labels:         function=mongo-connect
                pod-template-hash=389604249
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"mongo-connect-7dfb4868f","uid":"96ba80eb-ebe9-11e7-b07f-005056ac...
                prometheus.io/path=/metrics
                prometheus.io/port=8080
                prometheus.io/scrape=true
Status:         Pending
IP:             10.42.0.9
Created By:     ReplicaSet/mongo-connect-7dfb4868f
Controlled By:  ReplicaSet/mongo-connect-7dfb4868f
Init Containers:
  prepare:
    Container ID:  docker://947ac94729615804397bc6bd0d707262e9d166a799be6c78dfe424bd56dd8950
    Image:         kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
    Image ID:      docker-pullable://kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697
    Port:          <none>
    Command:
      sh
      -c
    Args:
      cp /src/handler.py /kubeless/handler.py && cp /src/requirements.txt /kubeless
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 28 Dec 2017 21:38:39 +0530
      Finished:     Thu, 28 Dec 2017 21:38:39 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /kubeless from mongo-connect (rw)
      /src from mongo-connect-deps (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qkfws (ro)
  install:
    Container ID:  docker://6f70b3837f8ae4379c8ce7bc63f2f0590759dc038818d56e595d25a61d18bd11
    Image:         tuna/python-pillow:2.7.11-alpine
    Image ID:      docker-pullable://tuna/python-pillow@sha256:6e39c3a1382e0666eac1e435b8dd78e202d1cf791166ba2a77ad52a6584e04ac
    Port:          <none>
    Command:
      sh
      -c
    Args:
      pip install --prefix=/kubeless -r /kubeless/requirements.txt
    State:          Running
      Started:      Thu, 28 Dec 2017 21:42:27 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 28 Dec 2017 21:40:56 +0530
      Finished:     Thu, 28 Dec 2017 21:42:01 +0530
    Ready:          False
    Restart Count:  3
    Environment:
      FUNC_HANDLER:  mongoConnect
      MOD_NAME:      handler
      FUNC_TIMEOUT:  180
      TOPIC_NAME:
    Mounts:
      /kubeless from mongo-connect (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qkfws (ro)
Containers:
  mongo-connect:
    Container ID:
    Image:          kubeless/python@sha256:ba948a6783b93d75037b7b1806a3925d441401ae6fba18282f712a1b1a786899
    Image ID:
    Port:           8080/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8080/healthz delay=3s timeout=1s period=30s #success=1 #failure=3
    Environment:
      FUNC_HANDLER:  mongoConnect
      MOD_NAME:      handler
      FUNC_TIMEOUT:  180
      TOPIC_NAME:
      PYTHONPATH:    /kubeless/lib/python2.7/site-packages
    Mounts:
      /kubeless from mongo-connect (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qkfws (ro)
Conditions:
  Type           Status
  Initialized    False
  Ready          False
  PodScheduled   True
Volumes:
  mongo-connect:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  mongo-connect-deps:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongo-connect
    Optional:  false
  default-token-qkfws:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qkfws
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                     From                                      Message
  ----     ------                 ----                    ----                                      -------
  Normal   SuccessfulMountVolume  3m                      kubelet, cmbu-devops-services-kube-node2  MountVolume.SetUp succeeded for volume "mongo-connect"
  Normal   SuccessfulMountVolume  3m                      kubelet, cmbu-devops-services-kube-node2  MountVolume.SetUp succeeded for volume "mongo-connect-deps"
  Normal   SuccessfulMountVolume  3m                      kubelet, cmbu-devops-services-kube-node2  MountVolume.SetUp succeeded for volume "default-token-qkfws"
  Normal   Pulled                 3m                      kubelet, cmbu-devops-services-kube-node2  Container image "kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697" already present on machine
  Normal   Created                3m                      kubelet, cmbu-devops-services-kube-node2  Created container
  Normal   Started                3m                      kubelet, cmbu-devops-services-kube-node2  Started container
  Normal   Scheduled              2m                      default-scheduler                         Successfully assigned mongo-connect-7dfb4868f-crt96 to cmbu-devops-services-kube-node2
  Warning  BackOff                6s (x3 over 1m)         kubelet, cmbu-devops-services-kube-node2  Back-off restarting failed container
  Warning  FailedSync             6s (x3 over 1m)         kubelet, cmbu-devops-services-kube-node2  Error syncing pod
  Normal   Pulled                 <invalid> (x4 over 3m)  kubelet, cmbu-devops-services-kube-node2  Container image "tuna/python-pillow:2.7.11-alpine" already present on machine
  Normal   Created                <invalid> (x4 over 3m)  kubelet, cmbu-devops-services-kube-node2  Created container
  Normal   Started                <invalid> (x4 over 3m)  kubelet, cmbu-devops-services-kube-node2  Started container

[vishweshwarp]

@andresmgot init container has access to internet but I noticed a problem with pod network which is causing kube-dns restart each time function pod tries to resolve python depenencies. After I solved network issue, function pod came up successfully.

[andresmgot]

Hi @vishweshwarp,

Actually the error entry in the controller should not affect the deployment (since monitoring is handled at the very end).

My guess is that the initContainer does not have access to the internet or pypi.python.org so the connection hangs. In which platform/cloud provider are you running your example? What are the logs of the initContainer? You can retrieve them executing:

kubectl logs mongo-connect-7dfb4868f-crt96 -c prepare

Also note that you don’t need the serverless-python-requirements plugin, that one is not used with Kubeless, we handle the installation so your serverless.yaml should look something like:

service: mongo-utils

provider:
  name: kubeless
  runtime: python2.7

plugins:
  - serverless-kubeless
functions:
  mongo-connect:
    handler: handler.mongoConnect