host.socket.get_listening_sockets() does not work on Ubuntu 18.04.1 LTS

I’m using this snippet of code to test listening ports:

def test_listening_socket(host):
    listening = host.socket.get_listening_sockets()
    for spec in (
"tcp://:::22",
"tcp://:::5432",
    ):  
        socket = host.socket(spec)
        assert socket.is_listening

but it fails on Ubuntu 18.04.1 LTS:

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"


==================================================================================================================================== FAILURES =====================================================================================================================================
_______________________________________________________________________________________________________________ test_listening_socket[paramiko://alpha] _______________________________________________________________________________________________________________

host = <testinfra.host.Host object at 0x7fc1b010e190>

    def test_listening_socket(host):
>       listening = host.socket.get_listening_sockets()

test_alpha.py:80: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python2.7/dist-packages/testinfra/modules/socket.py:171: in get_listening_sockets
    for sock in cls(None)._iter_sockets(True):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <socket None://None>, listening = True

    def _iter_sockets(self, listening):
        cmd = 'ss --numeric'
        if listening:
            cmd += ' --listening'
        else:
            cmd += ' --all'
        if self.protocol == 'tcp':
            cmd += ' --tcp'
        elif self.protocol == 'udp':
            cmd += ' --udp'
        elif self.protocol == 'unix':
            cmd += ' --unix'
    
        for line in self.run(cmd).stdout_bytes.splitlines()[1:]:
            # Ignore unix datagram sockets.
            if line.split(None, 1)[0] == b'u_dgr':
                continue
            splitted = line.decode().split()
    
            # If listing only TCP or UDP sockets, output has 5 columns:
            # (State, Recv-Q, Send-Q, Local Address:Port, Peer Address:Port)
            if self.protocol in ('tcp', 'udp'):
                protocol = self.protocol
                status, local, remote = (
                    splitted[0], splitted[3], splitted[4])
            # If listing all or just unix sockets, output has 6 columns:
            # Netid, State, Recv-Q, Send-Q, LocalAddress:Port, PeerAddress:Port
            else:
                protocol, status, local, remote = (
>                   splitted[0], splitted[1], splitted[4], splitted[5])
E               IndexError: list index out of range

/usr/local/lib/python2.7/dist-packages/testinfra/modules/socket.py:238: IndexError

[n0guest]

Yep, it’s definetly the commit mentioned above. Quickly builded 4.18 (from bionic-backports) and problem gone:

root@local-test-42-bionic-default:~# cat /root/iproute-width.patch 
diff --git a/misc/ss.c b/misc/ss.c
index c8970438..4d12fb5d 100644
--- a/misc/ss.c
+++ b/misc/ss.c
@@ -1260,7 +1260,7 @@ static void render(void)
 	while (token) {
 		/* Print left delimiter only if we already started a line */
 		if (line_started++)
-			printed = printf("%s", current_field->ldelim);
+			printed = printf("%s", f->ldelim);
 		else
 			printed = 0;
 
root@local-test-42-bionic-default:~# echo "$(/bin/ss --numeric --listening --tcp)"
State    Recv-Q    Send-Q        Local Address:Port        Peer Address:Port    
LISTEN   0         128           127.0.0.53%lo:53               0.0.0.0:*       
LISTEN   0         128                 0.0.0.0:22               0.0.0.0:*       
LISTEN   0         128                    [::]:22                  [::]:*       

root@local-test-42-bionic-default:~# dpkg -l iproute2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                    Version          Architecture     Description
+++-=======================-================-================-===================================================
ii  iproute2                4.18.0-1ubuntu2~ amd64            networking and traffic control tools

But it’s still a bit of “too hackish” way to solve “failed test” issue. Don’t you think? 8)

[n0guest]

Looks like this story hasn’t ended yet. We faced similar problem, but with LXD instead of Nginx. Here is how our tests crashed:

< ... SKIPPED ...>
    =================================== FAILURES ===================================
    ________ test_ports[ansible://local-test-42-bionic-default-tcp://8443] _________

    host = <testinfra.host.Host object at 0x7faec56facc0>, port = 'tcp://8443'

        @pytest.mark.parametrize("port", [
            (lxd_listen_tcp),
            (lxd_listen_unix),
        ])
        def test_ports(host, port):
            """Testing for listeners on port."""
    >       assert host.socket(port).is_listening

    tests/test_default.py:63:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    /usr/local/lib/python3.5/dist-packages/testinfra/modules/socket.py:112: in is_listening
        sockets = list(self._iter_sockets(True))
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    self = <socket tcp://8443>, listening = True

        def _iter_sockets(self, listening):
            cmd = '%s --numeric'
            if listening:
                cmd += ' --listening'
            else:
                cmd += ' --all'
            if self.protocol == 'tcp':
                cmd += ' --tcp'
            elif self.protocol == 'udp':
                cmd += ' --udp'
            elif self.protocol == 'unix':
                cmd += ' --unix'

            for line in self.run(cmd, self._command).stdout_bytes.splitlines()[1:]:
                # Ignore unix datagram sockets.
                if line.split(None, 1)[0] == b'u_dgr':
                    continue
                splitted = line.decode().split()

                # If listing only TCP or UDP sockets, output has 5 columns:
                # (State, Recv-Q, Send-Q, Local Address:Port, Peer Address:Port)
                if self.protocol in ('tcp', 'udp'):
                    protocol = self.protocol
                    status, local, remote = (
    >                   splitted[0], splitted[3], splitted[4])
    E               IndexError: list index out of range

    /usr/local/lib/python3.5/dist-packages/testinfra/modules/socket.py:239: IndexError
    ------------------------------ Captured log call -------------------------------
    INFO     testinfra:base.py:281 RUN CommandResult(command=b"lxc exec local-test-42-bionic-default --mode=non-interactive -- /bin/sh -c 'command -v ss'", exit_status=0, stdout=b'/bin/ss\n', stderr=None)
    INFO     testinfra:base.py:281 RUN CommandResult(command=b"lxc exec local-test-42-bionic-default --mode=non-interactive -- /bin/sh -c '/bin/ss --numeric --listening --tcp'", exit_status=0, stdout=b'State Recv-Q Send-Q                       Local Address:Port  Peer Address:Port \nLISTEN0      32                            10.120.149.1:53         0.0.0.0:*    \nLISTEN0      128                          127.0.0.53%lo:53         0.0.0.0:*    \nLISTEN0      128                                0.0.0.0:22         0.0.0.0:*    \nLISTEN0      32                [fd42:d8d2:8099:e96e::1]:53            [::]:*    \nLISTEN0      32      [fe80::d0d2:7eff:fefd:8273]%lxdbr0:53            [::]:*    \nLISTEN0      128                                   [::]:22            [::]:*    \nLISTEN0      128                                      *:8443             *:*    \n', stderr=None)
    ===================== 1 failed, 17 passed in 9.97 seconds ======================
writing pytestdebug information to /home/user/ansible/lxd/molecule/default/pytestdebug.log
wrote pytestdebug information to /home/user/ansible/lxd/molecule/default/pytestdebug.log

And here is an interesting part. How TCP socket will be checked in case of Ubuntu 16.04/xenial:

root@builder:~# lxc exec local-test-42-xenial-default --mode=non-interactive -- /bin/sh -c '/bin/ss --numeric --listening --tcp'
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port
LISTEN     0      5      10.27.204.1:53                       *:*
LISTEN     0      128          *:22                       *:*
LISTEN     0      5       fd42:419e:64e2:fe80::1:53                      :::*
LISTEN     0      5      fe80::fc16:eaff:fe6e:f9ed%lxdbr0:53                      :::*
LISTEN     0      128         :::22                      :::*
LISTEN     0      128         :::8443                    :::*

And the same for Ubuntu 18.04/bionic (which crashed):

root@builder:~# lxc exec local-test-42-bionic-default --mode=non-interactive -- /bin/sh -c '/bin/ss --numeric --listening --tcp'
State Recv-Q Send-Q                       Local Address:Port  Peer Address:Port
LISTEN0      32                            10.120.149.1:53         0.0.0.0:*
LISTEN0      128                          127.0.0.53%lo:53         0.0.0.0:*
LISTEN0      128                                0.0.0.0:22         0.0.0.0:*
LISTEN0      128                              127.0.0.1:8443       0.0.0.0:*
LISTEN0      32                [fd42:d8d2:8099:e96e::1]:53            [::]:*
LISTEN0      32      [fe80::d0d2:7eff:fefd:8273]%lxdbr0:53            [::]:*
LISTEN0      128                                   [::]:22            [::]:*

As you can see there is something strange with first two columns. They just merged to one by some reason. Digging deeper and trying to run same comand within bionic:

root@local-test-42-bionic-default:~# /bin/ss --numeric --listening --tcp
State     Recv-Q     Send-Q                               Local Address:Port         Peer Address:Port
LISTEN    0          32                                    10.120.149.1:53                0.0.0.0:*
LISTEN    0          128                                  127.0.0.53%lo:53                0.0.0.0:*
LISTEN    0          128                                        0.0.0.0:22                0.0.0.0:*
LISTEN    0          128                                      127.0.0.1:8443              0.0.0.0:*
LISTEN    0          32                        [fd42:d8d2:8099:e96e::1]:53                   [::]:*
LISTEN    0          32              [fe80::d0d2:7eff:fefd:8273]%lxdbr0:53                   [::]:*
LISTEN    0          128                                           [::]:22                   [::]:*

Looks ok, but here is subshell version (same as lxc exec … works?):

root@local-test-42-bionic-default:~# echo "$(/bin/ss --numeric --listening --tcp)"
State Recv-Q Send-Q                       Local Address:Port  Peer Address:Port
LISTEN0      32                            10.120.149.1:53         0.0.0.0:*
LISTEN0      128                          127.0.0.53%lo:53         0.0.0.0:*
LISTEN0      128                                0.0.0.0:22         0.0.0.0:*
LISTEN0      128                              127.0.0.1:8443       0.0.0.0:*
LISTEN0      32                [fd42:d8d2:8099:e96e::1]:53            [::]:*
LISTEN0      32      [fe80::d0d2:7eff:fefd:8273]%lxdbr0:53            [::]:*
LISTEN0      128                                   [::]:22            [::]:*

And columns again merged. Here is how to reproduced it without subshell (just make your terminal smaller):

root@local-test-42-bionic-default:~# stty size
30 85
root@local-test-42-bionic-default:~# /bin/ss --numeric --listening --tcp
State  Recv-Q  Send-Q                         Local Address:Port   Peer Address:Port
LISTEN 0       32                              10.120.149.1:53          0.0.0.0:*
LISTEN 0       128                            127.0.0.53%lo:53          0.0.0.0:*
LISTEN 0       128                                  0.0.0.0:22          0.0.0.0:*
LISTEN 0       128                                127.0.0.1:8443        0.0.0.0:*
LISTEN 0       32                  [fd42:d8d2:8099:e96e::1]:53             [::]:*
LISTEN 0       32        [fe80::d0d2:7eff:fefd:8273]%lxdbr0:53             [::]:*
LISTEN 0       128                                     [::]:22             [::]:*

root@local-test-42-bionic-default:~# stty size
30 84
root@local-test-42-bionic-default:~# /bin/ss --numeric --listening --tcp
State Recv-Q  Send-Q                         Local Address:Port   Peer Address:Port
LISTEN0       32                              10.120.149.1:53          0.0.0.0:*
LISTEN0       128                            127.0.0.53%lo:53          0.0.0.0:*
LISTEN0       128                                  0.0.0.0:22          0.0.0.0:*
LISTEN0       128                                127.0.0.1:8443        0.0.0.0:*
LISTEN0       32                  [fd42:d8d2:8099:e96e::1]:53             [::]:*
LISTEN0       32        [fe80::d0d2:7eff:fefd:8273]%lxdbr0:53             [::]:*
LISTEN0       128                                     [::]:22             [::]:*

Probably the main problem are the long addresses in column “Local Addres”? After some digging I found this commit:

While rendering columns, we use a local variable to keep track of the field currently being printed, without touching current_field, which is used for buffering.

Use the right pointer to access the left delimiter for the current column, instead of always printing the left delimiter for the last buffered field, which is usually an empty string.

This fixes an issue especially visible on narrow terminals, where some columns might be displayed without separation.

Reported-by: YoyPa [e-mail] Fixes: 691bd85 (“ss: Buffer raw fields first, then render them as a table”) Signed-off-by: Stefano Brivio [e-mail] Tested-by: YoyPa [e-mail] Signed-off-by: Stephen Hemminger [e-mail]

This looks suspiciously similar to our case here. I don’t sure in which versions exactly problem came. But I guess, that bionic wouldn’t be getting iproute >=5.1 in any time soon (or ever?).

@philpep, after all that, what are our options here? Only fallback to netstat if it’s available (or use it first)? 0_o