Issues when using SSH connection method against IPv6-enabled agents

Version report

Jenkins and plugins versions report:

Jenkins: 2.263.3 OS: Linux - 4.15.0-1113-azure

Plugins:

ace-editor:1.1
ansicolor:0.7.5
ant:1.11
antisamy-markup-formatter:2.1
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
azure-ad:1.2.1
azure-commons:1.0.5
basic-branch-build-strategies:1.3.2
bitbucket-pullrequest-builder:1.5.0
block-queued-job:0.2.0
blueocean-autofavorite:1.2.4
blueocean-bitbucket-pipeline:1.24.4
blueocean-commons:1.24.4
blueocean-config:1.24.4
blueocean-core-js:1.24.4
blueocean-dashboard:1.24.4
blueocean-display-url:2.4.1
blueocean-events:1.24.4
blueocean-git-pipeline:1.24.4
blueocean-github-pipeline:1.24.4
blueocean-i18n:1.24.4
blueocean-jira:1.24.4
blueocean-jwt:1.24.4
blueocean-personalization:1.24.4
blueocean-pipeline-api-impl:1.24.4
blueocean-pipeline-editor:1.24.4
blueocean-pipeline-scm-api:1.24.4
blueocean-rest-impl:1.24.4
blueocean-rest:1.24.4
blueocean-web:1.24.4
blueocean:1.24.4
bootstrap4-api:4.6.0-1
bouncycastle-api:2.20
branch-api:2.6.2
build-timeout:1.20
caffeine-api:2.9.1-23.v51c4e2c879c8
cctray-xml:1.0
checks-api:1.4.1
cloud-stats:0.26
cloudbees-bitbucket-branch-source:2.9.7
cloudbees-disk-usage-simple:0.10
cloudbees-folder:6.15
command-launcher:1.5
config-file-provider:3.7.0
configuration-as-code:1.51
credentials-binding:1.24
credentials:2.3.14
display-url-api:2.3.4
docker-build-publish:1.3.2
docker-commons:1.17
docker-java-api:3.1.5.2
docker-plugin:1.2.2
docker-workflow:1.25
durable-task:1.35
echarts-api:4.9.0-3
email-ext:2.81
embeddable-build-status:2.0.3
extended-read-permission:3.2
external-monitor-job:1.7
favorite:2.3.2
font-awesome-api:5.15.2-1
git-client:3.6.0
git-server:1.9
git:4.5.2
github-api:1.122
github-branch-source:2.9.5
github-pullrequest:0.2.8
github:1.32.0
google-oauth-plugin:1.0.3
gradle:1.36
greenballs:1.15.1
handlebars:1.1.1
handy-uri-templates-2-api:2.1.8-1.0
hashicorp-vault-plugin:3.7.0
htmlpublisher:1.25
icon-shim:2.0.3
jackson2-api:2.12.1
javadoc:1.6
jclouds-jenkins:2.20
jdk-tool:1.4
jenkins-design-language:1.24.4
jira:3.1.3
jjwt-api:0.11.2-8.82737cbfa6f5
jquery-detached:1.2.1
jquery3-api:3.5.1-2
jquery:1.12.4-1
jsch:0.1.55.2
junit:1.48
kubernetes-cli:1.10.0
kubernetes-client-api:4.13.2-1
kubernetes-credentials:0.8.0
kubernetes:1.29.0
lockable-resources:2.10
mailer:1.32.1
mapdb-api:1.0.9.0
mask-passwords:2.13
matrix-auth:2.6.6
matrix-project:1.18
mercurial:2.12
metrics:4.0.2.7
momentjs:1.1.1
notification:1.14
oauth-credentials:0.4
okhttp-api:3.14.9
ownership:0.13.0
pam-auth:1.6
parameterized-scheduler:0.9.2
pipeline-build-step:2.13
pipeline-github-lib:1.0
pipeline-graph-analysis:1.10
pipeline-input-step:2.12
pipeline-milestone-step:1.3.2
pipeline-model-api:1.8.3
pipeline-model-definition:1.8.3
pipeline-model-extensions:1.8.3
pipeline-rest-api:2.19
pipeline-stage-step:2.5
pipeline-stage-tags-metadata:1.8.3
pipeline-stage-view:2.19
pipeline-utility-steps:2.6.1
plain-credentials:1.7
plugin-util-api:1.6.1
popper-api:1.16.1-1
prometheus:2.0.8
pubsub-light:1.13
resource-disposer:0.14
role-strategy:3.1
scm-api:2.6.4
script-security:1.76
slack:2.45
snakeyaml-api:1.27.0
sse-gateway:1.24
ssh-credentials:1.18.1
ssh-slaves:1.31.5
structs:1.21
subversion:2.14.0
timestamper:1.11.8
token-macro:2.13
trilead-api:1.0.13
variant:1.4
webhook-step:1.4
windows-slaves:1.7
workflow-aggregator:2.6
workflow-api:2.41
workflow-basic-steps:2.23
workflow-cps-global-lib:2.17
workflow-cps:2.87
workflow-durable-task-step:2.37
workflow-job:2.40
workflow-multibranch:2.22
workflow-scm-step:2.12
workflow-step-api:2.23
workflow-support:3.7
ws-cleanup:0.38

Docker version on agents: 20.10.7

`docker version` output

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:38 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:50 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

OS: Ubuntu 20.04 LTS on Jenkins master and every agent.

Reproduction steps

• Install jenkins with docker plugin on a virtual machine (VM)
• Configure docker cloud…
  1. to use docker on another IPv6-enabled VM via tcp (you don’t need to use IPv6 here, just keep it enabled system-wide)
  2. to spin ssh-agent-based image
  3. to use SSH connection method
• Load the agent with pipelines for a few days

Results

Expected result:

Jenkins can spin a new agent and connect to it using SSH at any time.

Actual result:

Jenkins can spin a new agent but unable to connect it using SSH by the reason explained below.

From docker ps output:

0f1a5876f016   [REDACTED]/jenkins-ci-dinfra:stable   "setup-sshd /usr/sbi…"   3 minutes ago   Up 3 minutes   0.0.0.0:49243->22/tcp, :::49242->22/tcp   musing_boyd
0a5530eb2201   [REDACTED]/jenkins-ci-dinfra:stable   "setup-sshd /usr/sbi…"   5 hours ago     Up 5 hours     0.0.0.0:49205->22/tcp, :::49204->22/tcp   vigorous_blackwell

You can see IPv4 port is different from IPv6 port (49243 vs 49242). Somehow Jenkins is using IPv6 port when trying to ssh into the agent.

I did docker inspect and get logs from Jenkins but for different case (not the same as docker ps output above). But situation is the same.

Logs from Jenkins master (hostnames are altered):

SSHLauncher{host='slavep3.node', port=49739, credentialsId='13457128-567e-4f7d-bd8c-1e85c619b69e', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=30, retryWaitTime=2, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[06/16/21 12:22:47] [SSH] Opening SSH connection to slavep3.node:49739.
Connection refused (Connection refused)
[long java trace here]

NetworkSettings.Ports from docker inspect output:

            "Ports": {
                "22/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "49740"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "49739"
                    }
                ]
            },

As you can see Jenkins was trying to connect to the port 49739 via IPv4 (we don’t have IPv6 connectivity at the moment). But docker-proxy was listening on port 49740 for IPv4 instead.

[mikjonsson]

I may be out on a limb here, as I’ve only been browsing the code on GitHub and haven’t debugged it (and may not even be looking at the correct part of the code for all I know), but in [DockerComputerSSHConnector.java getBindingForPort](https://github.com/jenkinsci/docker-plugin/blob/master/src/main/java/io/jenkins/docker/connector/DockerComputerSSHConnector.java#:~:text=private static InetSocketAddress-,getBindingForPort,-(DockerAPI api%2C InspectContainerResponse) there’s this:

        // Find where it's mapped to
        for (Ports.Binding b : sshBindings) {
            String hps = b.getHostPortSpec();
            port = Integer.valueOf(hps);
        }
        String host = getExternalIP(api, ir, networkSettings, sshBindings);
        return new InetSocketAddress(host, port);

Looks like in the case of multiple bindings it will always return the port for the last binding in sshBindings without validating that it is the correct port, which may cause an issue if the correct port is earlier in the array.

[Sulphurium-Brimstone]

This may be wrong but from what I can tell the the code above sets port number to last binding. However, getExternalIP returns the IP of the first binding if it is a swarm. If this is case that would explain the issue. It seems to me that the IP and port need to be synced to match bindings returned from docker.

        if (api.isSwarm()) {
            for (Ports.Binding b : sshBindings) {
                String ipAddress = b.getHostIp();
                if (ipAddress != null && !"0.0.0.0".equals(ipAddress)) {
                    return ipAddress;
                }
            }
        }