adservice: opencensus trace spans failed to export (INVALID_ARGUMENT)

@rghetia randomly stumbled upon this in adservice logs while running on GKE. Looks like a You might want to forward to another repo for investigation.

Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Unrecognized region or location.: timeSeries[0-49]

Logs:

...
{"thread":"grpc-default-executor-2392","level":"INFO","loggerName":"hipstershop.AdService","message":"received ad request (context_words=[cycling])","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1552510190,"nanoOfSecond":990000000},"threadId":2631,"threadPriority":5,"logging.googleapis.com/trace":"0000000000000000cc45f5e21cc9de9e","logging.googleapis.com/spanId":"af92bde986bcf9f2","logging.googleapis.com/traceSampled":"true","time":"2019-03-13T20:49:50.990Z"}
Mar 13, 2019 8:49:51 PM io.opencensus.exporter.stats.stackdriver.StackdriverExporterWorker export
WARNING: ApiException thrown when exporting TimeSeries.
com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Unrecognized region or location.: timeSeries[0-49]
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1030)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:507)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:482)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:694)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
		at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
		at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
		at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1156)
		at io.opencensus.exporter.stats.stackdriver.StackdriverExporterWorker.export(StackdriverExporterWorker.java:210)
		at io.opencensus.exporter.stats.stackdriver.StackdriverExporterWorker.run(StackdriverExporterWorker.java:239)
		... 1 more
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Unrecognized region or location.: timeSeries[0-49]
	at io.grpc.Status.asRuntimeException(Status.java:530)
	... 23 more

[marcosfilho]

I raised this issue

https://github.com/census-instrumentation/opencensus-java/issues/1888

it’s been already fixed and released on version 0.22.0

https://github.com/census-instrumentation/opencensus-java/pull/1889

thanks

[rghetia]

@marcosfilho can you please open an issue on Java repo for opencensus?