Adservice gets connection reset by peer

I am running the demo on GCP with Istio installed, and I’m using my own load generator that has 1500 users instead of the current load generator’s 10. After roughly 3 minutes of running, I stop getting regular responses and only get 503: Service Unavailable. I looked in the logs and it seems like it is due to an error in Adservice. The logs for Adservice repeatedly say

INFO: Transport failed
io.grpc.netty.shaded.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
GRPC: src/core/lib/security/credentials/oauth2/oauth2_credentials.cc:154 Call to http server ended with error 426 [].
Jun 14, 2021 2:22:43 PM io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport notifyTerminated

I believe the 426 is saying that something should be upgraded, but I’m not quite sure what. Any help would be appreciated.

[mathieu-benoit]

Maybe not related but JFYI, I found out what I was doing wrong about my issue mentioned earlier in this thread https://github.com/GoogleCloudPlatform/microservices-demo/issues/559#issuecomment-860810436. I actually set too low the values of the resources.requests/limits which landed to kill my adservice pod. When I set those values back to what it’s recommended in there https://github.com/GoogleCloudPlatform/microservices-demo/blob/master/release/kubernetes-manifests.yaml#L713, it fixed the issue I caused by myself. So JFYI @jessberg in case you have also your adservice taking too much memory/cpu and being killed with your load tests?

[askmeegs]

Closing this issue as possibly a non-Istio related bug, that instead may be related to insufficient resource requests + limits for the services. @jessberg please reopen if you adjust your resource requests and you’re still seeing this problem. Thanks!