[Defect] Zuul-Core does not use updated IP address for a given DNS without app restart

Scenario:

1. There is a service registered in eureka with ‘service.company.aws’ DNS enpoint. That DNS resolves to IP: 192.168.0.5.
2. A request is made to the service endpoint via Zuul creating the pool with DNS/IP as serverAddr.
3. The service.company.aws’s IP changes to 192.168.0.17
4. A request is made to the service via Zuul.

Expected Result: The new DNS lookup should fetch the new IP (192.168.0.17) and set it in the connection pool config

Actual Result: Zuul is using the old IP (192.168.0.5) address stored in perServerConnectionPool resulting in a 502 Bad Gateway error.

https://github.com/Netflix/zuul/blob/master/zuul-core/src/main/java/com/netflix/zuul/netty/connectionpool/DefaultClientChannelManager.java#L399

The DNS lookup seems to be always happening and it is being assigned to the finalServerAddr var but the pool is not updated with it.

           try {
                serverAddr = new InetSocketAddress(rawHost, port);
            } catch (RuntimeException e2) {
                e1.addSuppressed(e2);
                throw e1;
            }
        }
        final InetSocketAddress finalServerAddr = serverAddr;

        selectedServer.set(chosenServer);
        // Now get the connection-pool for this server.
        IConnectionPool pool = perServerPools.computeIfAbsent(chosenServer, s -> {
            // Get the stats from LB for this server.
            LoadBalancerStats lbStats = loadBalancer.getLoadBalancerStats();
            ServerStats stats = lbStats.getSingleServerStat(chosenServer);

            final ClientChannelManager clientChannelMgr = this;
            PooledConnectionFactory pcf = createPooledConnectionFactory(chosenServer, instanceInfo, stats, clientChannelMgr, closeConnCounter, closeWrtBusyConnCounter);

            // Create a new pool for this server.
            return createConnectionPool(chosenServer, stats, instanceInfo, finalServerAddr, clientConnFactory, pcf, connPoolConfig,
                    clientConfig, createNewConnCounter, createConnSucceededCounter, createConnFailedCounter,
                    requestConnCounter, reuseConnCounter, connTakenFromPoolIsNotOpen, maxConnsPerHostExceededCounter,
                    connEstablishTimer, connsInPool, connsInUse);
        });

        return pool.acquire(eventLoop, passport, selectedHostAddr);

In such scenario where the IP for a given DNS changes how to avoid zuul using the old IP address?

proposed solution is to create a setServerAddr method in perServerConnectionPool so we can set the updated IP in the pool config before acquire.

            return createConnectionPool(chosenServer, stats, instanceInfo, finalServerAddr, clientConnFactory, pcf, connPoolConfig,
                    clientConfig, createNewConnCounter, createConnSucceededCounter, createConnFailedCounter,
                    requestConnCounter, reuseConnCounter, connTakenFromPoolIsNotOpen, maxConnsPerHostExceededCounter,
                    connEstablishTimer, connsInPool, connsInUse);
        });
        
        pool.setServerAddr(finalServerAddr)
        return pool.acquire(eventLoop, passport, selectedHostAddr);

[fang10]

@gb you can just override specific Zuul classes in your app as follows:

package com.netflix.zuul.netty.connectionpool;

YourserviceClientChannelManager implements ClientChannelManager {
// copy the implementation from ClientChannelManager accordingly and make the modification as necessary.
}

package com.netflix.zuul.netty.connectionpool;
/**
 * Similar as {@link PerServerConnectionPool}, but socket address is no longer cached across all connections in each
 * server pool. {@link SocketAddressProvider} is used to provide a DNS resolution at
 * runtime (https://github.com/Netflix/zuul/issues/780).
 */
YourservicePerServerConnectionPool implements IConnectionPool {
// copy the implementation from IConnectionPool accordingly and make the modification as necessary.

  protected void tryMakingNewConnection(....) {
       ......
       SocketAddress serverAddr = serverAddrProvider.get(server);
  }
}

The downside of this approach is that it creates complexity when you perform Zuul upgrade in the future because you need to be aware the changes of the two classes you just override

[mariakan1]

@carl-mastrangelo Any chance we can look at this again? We’re hitting this issue as well, but our situation is slightly different, we don’t use eureka. Our call just go to Application Load Balancer. @fang10 Can you show me how you hook up that customized ClientChannelManager into your application?