Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Tye fails to deploy

See original GitHub issue

Describe the bug

Tye deployment nominally succeeds, and I can run docker pull mtman/backend:1.0.0 on the server as well, but the images deployed by Tye have the ImagePullBackOff status.

I have a Docker Hub paid subscription and have run docker login and kubectl create secret generic regcred --from-file=.dockerconfigjson=snap/docker/471/.docker/config.json --type=kubernetes.io/dockerconfigjson as well.

I’m using micro k8s.

To Reproduce

Server

Create a Ubuntu machine (2GB RAM, 20GB SSD)
Run snap install microk8s --classic
Run snap install docker
Get the token by running microk8s.kubectl config view on the server
Run docker login and use your credentials to Docker Hub
Run microk8s.kubectl create secret generic regcred --from-file=.dockerconfigjson=snap/docker/471/.docker/config.json --type=kubernetes.io/dockerconfigjson

Client

Modify ~/.kube/config so the cluster has the server IP (https://YourIP:16443) and the current context is set to that cluster name
Add your user token (name: context, token: <get it by running microk8s.kubectl config view on the server>)
Follow the official tutorial to the deployment point
Run kubectl apply -f https://raw.githubusercontent.com/dotnet/tye/master/docs/tutorials/hello-tye/redis.yaml
Run tye deploy --interactive
Enter your Docker Hub nickname when asked
Enter redis:6379 as the Redis connection string
Run kubectl get pods Result:

PS C:\doc\src\tmp\tyepoc> kubectl get pods
NAME                        READY   STATUS             RESTARTS   AGE
backend-b565b4b59-wkz6k     0/1     ImagePullBackOff   0          3m37s
frontend-857c95f597-kb5pr   0/1     ImagePullBackOff   0          3m37s
redis-57455cbdbf-xggd5      1/1     Running            0          4m35s

Further technical details

Tye version: 0.4.0-alpha.20371.1+d7623120d66b85bfeda8ab69eb5ff792df6b7243
Result of tye deploy -i -v debug:

PS C:\doc\src\tmp\tyepoc> tye deploy -i -v debug
Loading Application Details...
Locating .NET SDK...
Found .NET SDK at: C:\Program Files\dotnet\sdk\3.1.401\
Registered .NET SDK.Loading project 'C:\doc\src\tmp\tyepoc\frontend\frontend.csproj'.
Loaded project 'C:\doc\src\tmp\tyepoc\frontend\frontend.csproj'.
Restoring project 'C:\doc\src\tmp\tyepoc\frontend\frontend.csproj'.
Build started 2020-08-22 00:41:01.
__________________________________________________
Project "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" (Restore target(s)):

Target _FilterRestoreGraphProjectInputItems:
    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" is building "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" (_IsProjectRestoreSupported target(s)):

Target _GetAllRestoreProjectPathItems:
    Determining projects to restore...
    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" is building "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" (_GenerateRestoreProjectPathWalk target(s)):

    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" is building "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" (_IsProjectRestoreSupported target(s)):

Target _GenerateRestoreGraph:
    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" is building "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" (_GenerateRestoreGraphProjectEntry target(s)):

    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" is building "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" (_GenerateProjectRestoreGraph target(s)):

Target Restore:
    Committing restore...
    Assets file has not changed. Skipping assets file writing. Path: C:\doc\src\tmp\tyepoc\frontend\obj\project.assets.json
    Restored C:\doc\src\tmp\tyepoc\frontend\frontend.csproj (in 56 ms).

    NuGet Config files used:
        C:\Users\lingw\AppData\Roaming\NuGet\NuGet.Config
        C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config

    Feeds used:
        https://api.nuget.org/v3/index.json
        C:\lib\nuget
        C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\
    All projects are up-to-date for restore.

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.55
Restored project 'C:\doc\src\tmp\tyepoc\frontend\frontend.csproj'.
Build started 2020-08-22 00:41:02.
__________________________________________________
Project "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" (ResolveReferences;ResolvePackageDependenciesDesignTime;PrepareResources;GetAssemblyAttributes target(s)):


Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.27
Found application version: 1.0.0
RunCommand=C:\doc\src\tmp\tyepoc\frontend\bin\Debug\netcoreapp3.1\frontend.exe
RunArguments=
TargetPath=C:\doc\src\tmp\tyepoc\frontend\bin\Debug\netcoreapp3.1\frontend.dll
PublishDir=bin\Debug\netcoreapp3.1\publish\
AssemblyName=frontend
IntermediateOutputPath=obj\Debug\netcoreapp3.1\
Found target framework: netcoreapp3.1
Found shared frameworks: Microsoft.NETCore.App, Microsoft.AspNetCore.App
IsAspNet=True
Evaluation Took: 1308.8379ms
Loading project 'C:\doc\src\tmp\tyepoc\backend\backend.csproj'.
Loaded project 'C:\doc\src\tmp\tyepoc\backend\backend.csproj'.
Restoring project 'C:\doc\src\tmp\tyepoc\backend\backend.csproj'.
Build started 2020-08-22 00:41:02.
__________________________________________________
Project "C:\doc\src\tmp\tyepoc\backend\backend.csproj" (Restore target(s)):

Target _FilterRestoreGraphProjectInputItems:
    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\backend\backend.csproj" is building "C:\doc\src\tmp\tyepoc\backend\backend.csproj" (_IsProjectRestoreSupported target(s)):

Target _GetAllRestoreProjectPathItems:
    Determining projects to restore...
    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\backend\backend.csproj" is building "C:\doc\src\tmp\tyepoc\backend\backend.csproj" (_GenerateRestoreProjectPathWalk target(s)):

    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\backend\backend.csproj" is building "C:\doc\src\tmp\tyepoc\backend\backend.csproj" (_IsProjectRestoreSupported target(s)):

Target _GenerateRestoreGraph:
    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\backend\backend.csproj" is building "C:\doc\src\tmp\tyepoc\backend\backend.csproj" (_GenerateRestoreGraphProjectEntry target(s)):

    __________________________________________________
    Project "C:\doc\src\tmp\tyepoc\backend\backend.csproj" is building "C:\doc\src\tmp\tyepoc\backend\backend.csproj" (_GenerateProjectRestoreGraph target(s)):

Target Restore:
    Committing restore...
    Assets file has not changed. Skipping assets file writing. Path: C:\doc\src\tmp\tyepoc\backend\obj\project.assets.json
    Restored C:\doc\src\tmp\tyepoc\backend\backend.csproj (in 2 ms).

    NuGet Config files used:
        C:\Users\lingw\AppData\Roaming\NuGet\NuGet.Config
        C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config

    Feeds used:
        https://api.nuget.org/v3/index.json
        C:\lib\nuget
        C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\
    All projects are up-to-date for restore.

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.11
Restored project 'C:\doc\src\tmp\tyepoc\backend\backend.csproj'.
Build started 2020-08-22 00:41:02.
__________________________________________________
Project "C:\doc\src\tmp\tyepoc\backend\backend.csproj" (ResolveReferences;ResolvePackageDependenciesDesignTime;PrepareResources;GetAssemblyAttributes target(s)):


Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.14
Found application version: 1.0.0
RunCommand=C:\doc\src\tmp\tyepoc\backend\bin\Debug\netcoreapp3.1\backend.exe
RunArguments=
TargetPath=C:\doc\src\tmp\tyepoc\backend\bin\Debug\netcoreapp3.1\backend.dll
PublishDir=bin\Debug\netcoreapp3.1\publish\
AssemblyName=backend
IntermediateOutputPath=obj\Debug\netcoreapp3.1\
Found target framework: netcoreapp3.1
Found shared frameworks: Microsoft.NETCore.App, Microsoft.AspNetCore.App
IsAspNet=True
Evaluation Took: 493.2275ms
Verifying kubectl installation...
Verifying kubectl connection to cluster...
Enter the Container Registry (ex: 'example.azurecr.io' for Azure or 'example' for dockerhub): mtman
Processing Service 'frontend'...
    Applying container defaults...
    Done Applying container defaults...
    Compiling Services...
    Done Compiling Services...
    Publishing Project...
        Running 'dotnet publish'.
        > dotnet publish "C:\doc\src\tmp\tyepoc\frontend\frontend.csproj" -c Release -o "C:\Users\lingw\AppData\Local\Temp\wub2fve1.hpn"
        Done running 'dotnet publish' exit code: 0
        Created Publish Output: 'C:\Users\lingw\AppData\Local\Temp\wub2fve1.hpn'
    Done Publishing Project...
    Building Docker Image...
        Writing Dockerfile to 'C:\Users\lingw\AppData\Local\Temp\wub2fve1.hpn\Dockerfile'.
        Done writing Dockerfile.
        Running 'docker build'.
        > docker build "C:\Users\lingw\AppData\Local\Temp\wub2fve1.hpn" -t mtman/frontend:1.0.0 -f "C:\Users\lingw\AppData\Local\Temp\wub2fve1.hpn\Dockerfile"
            Sending build context to Docker daemon  4.795MB

            Step 1/4 : FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
             ---> bdca989bc8d3
            Step 2/4 : WORKDIR /app
             ---> Using cache
             ---> 5996961b8873
            Step 3/4 : COPY . /app
             ---> Using cache
             ---> 9fdebbfed5bf
            Step 4/4 : ENTRYPOINT ["dotnet", "frontend.dll"]
             ---> Using cache
             ---> e59496705e9c
            Successfully built e59496705e9c
            Successfully tagged mtman/frontend:1.0.0
            SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
        Done running 'docker build' exit code: 0
        Created Docker Image: 'mtman/frontend:1.0.0'
    Done Building Docker Image...
    Pushing Docker Image...
        Running 'docker push'.
        > docker push mtman/frontend:1.0.0
            The push refers to repository [docker.io/mtman/frontend]
            0610d72c5ed5: Preparing
            91249b65073e: Preparing
            2e849d361dc1: Preparing
            c370033eb984: Preparing
            2135da4d457a: Preparing
            b24b2d7a1887: Preparing
            d0f104dc0a1f: Preparing
            b24b2d7a1887: Waiting
            d0f104dc0a1f: Waiting
            91249b65073e: Layer already exists
            2135da4d457a: Layer already exists
            c370033eb984: Layer already exists
            0610d72c5ed5: Layer already exists
            2e849d361dc1: Layer already exists
            b24b2d7a1887: Layer already exists
            d0f104dc0a1f: Layer already exists
            1.0.0: digest: sha256:8d51dd3b8c51aaae03dfeb309fb82d269f3035fdf013cbb15d8dd6a54474fbd3 size: 1793
        Done running 'docker push' exit code: 0
        Pushed docker image: 'mtman/frontend:1.0.0'
    Done Pushing Docker Image...
    Validating Secrets...
        Validating secret 'binding-production-redis-secret'.
        Found existing secret 'binding-production-redis-secret'.
    Done Validating Secrets...
    Generating Manifests...
    Done Generating Manifests...
Done Processing Service 'frontend'...
Processing Service 'backend'...
    Applying container defaults...
    Done Applying container defaults...
    Compiling Services...
    Done Compiling Services...
    Publishing Project...
        Running 'dotnet publish'.
        > dotnet publish "C:\doc\src\tmp\tyepoc\backend\backend.csproj" -c Release -o "C:\Users\lingw\AppData\Local\Temp\xcaiwfat.ivu"
        Done running 'dotnet publish' exit code: 0
        Created Publish Output: 'C:\Users\lingw\AppData\Local\Temp\xcaiwfat.ivu'
    Done Publishing Project...
    Building Docker Image...
        Writing Dockerfile to 'C:\Users\lingw\AppData\Local\Temp\xcaiwfat.ivu\Dockerfile'.
        Done writing Dockerfile.
        Running 'docker build'.
        > docker build "C:\Users\lingw\AppData\Local\Temp\xcaiwfat.ivu" -t mtman/backend:1.0.0 -f "C:\Users\lingw\AppData\Local\Temp\xcaiwfat.ivu\Dockerfile"
            Sending build context to Docker daemon  1.845MB

            Step 1/4 : FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
             ---> bdca989bc8d3
            Step 2/4 : WORKDIR /app
             ---> Using cache
             ---> 5996961b8873
            Step 3/4 : COPY . /app
             ---> Using cache
             ---> 92eb73b8fb6d
            Step 4/4 : ENTRYPOINT ["dotnet", "backend.dll"]
             ---> Using cache
             ---> 5d5984cf4c72
            Successfully built 5d5984cf4c72
            Successfully tagged mtman/backend:1.0.0
            SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
        Done running 'docker build' exit code: 0
        Created Docker Image: 'mtman/backend:1.0.0'
    Done Building Docker Image...
    Pushing Docker Image...
        Running 'docker push'.
        > docker push mtman/backend:1.0.0
            The push refers to repository [docker.io/mtman/backend]
            7fb43298df8c: Preparing
            91249b65073e: Preparing
            2e849d361dc1: Preparing
            c370033eb984: Preparing
            2135da4d457a: Preparing
            b24b2d7a1887: Preparing
            d0f104dc0a1f: Preparing
            b24b2d7a1887: Waiting
            d0f104dc0a1f: Waiting
            2135da4d457a: Layer already exists
            91249b65073e: Layer already exists
            c370033eb984: Layer already exists
            2e849d361dc1: Layer already exists
            7fb43298df8c: Layer already exists
            b24b2d7a1887: Layer already exists
            d0f104dc0a1f: Layer already exists
            1.0.0: digest: sha256:d87572de6581eb6603f2370127907a1924c5d5d9edcaaba2ebe141b30b84ad4b size: 1792
        Done running 'docker push' exit code: 0
        Pushed docker image: 'mtman/backend:1.0.0'
    Done Pushing Docker Image...
    Validating Secrets...
        Already validated secret 'binding-production-redis-secret'.
    Done Validating Secrets...
    Generating Manifests...
    Done Generating Manifests...
Done Processing Service 'backend'...
Processing Service 'redis'...
    Applying container defaults...
        Service 'redis' does not have a project associated. Skipping.
    Done Applying container defaults...
    Compiling Services...
    Done Compiling Services...
    Publishing Project...
        Service 'redis' does not have a project associated. Skipping.
    Done Publishing Project...
    Building Docker Image...
        Service 'redis' does not have a project associated. Skipping.
    Done Building Docker Image...
    Pushing Docker Image...
        Service 'redis' does not have a project associated. Skipping.
    Done Pushing Docker Image...
    Validating Secrets...
    Done Validating Secrets...
    Generating Manifests...
        Service 'redis' does not have a container. Skipping.
    Done Generating Manifests...
Done Processing Service 'redis'...
Processing Service 'redis-cli'...
    Applying container defaults...
        Service 'redis-cli' does not have a project associated. Skipping.
    Done Applying container defaults...
    Compiling Services...
    Done Compiling Services...
    Publishing Project...
        Service 'redis-cli' does not have a project associated. Skipping.
    Done Publishing Project...
    Building Docker Image...
        Service 'redis-cli' does not have a project associated. Skipping.
    Done Building Docker Image...
    Pushing Docker Image...
        Service 'redis-cli' does not have a project associated. Skipping.
    Done Pushing Docker Image...
    Validating Secrets...
    Done Validating Secrets...
    Generating Manifests...
        Service 'redis-cli' does not have a container. Skipping.
    Done Generating Manifests...
Done Processing Service 'redis-cli'...
Deploying Application Manifests...

        Verifying kubectl installation...
        Verifying kubectl connection to cluster...
        Writing output to 'C:\Users\lingw\AppData\Local\Temp\tmpCFE9.tmp'.
        Running 'kubectl apply' in $current namespace
        > kubectl apply -f "C:\Users\lingw\AppData\Local\Temp\tmpCFE9.tmp"
            deployment.apps/frontend unchanged
            service/frontend unchanged
            deployment.apps/backend unchanged
            service/backend unchanged
        Done running 'kubectl apply' exit code: 0
        Deployed application 'tyepoc'.
Done Deploying Application Manifests...
Time Elapsed: 00:00:20:08

Win10 x64 19041.450

Issue Analytics

State:
Created 3 years ago
Comments:9 (1 by maintainers)

Top GitHub Comments

1reaction

jr-araujocommented, Nov 20, 2020

@johnden12 Yes, your tip definitely worked for me. Thank you for this tip!

@jkotalik Having all that said and showed in this thread, I think that this reinforces that tye should have a way to allow us to configure this stuffs in tye.yaml, for instance.

Does it make sense to implement this kind of configuration in tye yaml file?

1reaction

amitHansdacommented, Sep 9, 2020

@amitHansda As I understand it, the problem could be that Tye doesn’t set the docker credentials correctly on the deployment target. Do you use Docker Hub as well, or is it a different registry?

When I am using dockerhub registry tye deploy works perfectly. But when I am using Azure Container Registry, during tye deploy process it is able to push the image to registry, but not able to pull and deploy in next steps. In my linux machine with MicroK8s kubectl is even unable to create deploy. Even if I login to my ACR