[Stackdriver] One or more of the points specified had an older start time than the most recent point

Micrometer 1.1.3 Spring Boot 2.1.3.RELEASE

Hi,

We’re trying to setup Micrometer with Stackdriver with the Spring Boot integration and we have this error message below. Did we miss some configuration?

Thank you, Christophe

Points must be written in order. One or more of the points specified had an older start time than the most recent point.: timeSeries[12,13,18,19,49,50]

Stackdriver properties:

@Component
public class StackdriverProperties extends StepRegistryProperties {

    private final String projectId;

    public StackdriverProperties(@Value("${kolsquare.ingest.google.projectId}") final String projectId) {
        this.projectId = projectId;
    }

    public String getProjectId() {
        return projectId;
    }
}

Stackdriver configuration:

@Component
public class StackdriverPropertiesConfigAdapter extends StepRegistryPropertiesConfigAdapter<StackdriverProperties> implements StackdriverConfig {

    private final String prefix;

    public StackdriverPropertiesConfigAdapter(final StackdriverProperties properties,
                                              @Value("${kolsquare.ingest.stackdriver.prefix}") final String prefix) {
        super(properties);

        this.prefix = prefix;
    }

    @Override
    public String projectId() {
        return get(StackdriverProperties::getProjectId, StackdriverConfig.super::projectId);
    }

    @Override
    public String prefix() {
        return prefix;
    }
}

Meter registry declaration:

@Configuration
public class StackdriverConfig {

    @Bean
    public StackdriverMeterRegistry stackdriverMeterRegistry(final StackdriverConfig config) throws IOException {
        final MetricServiceSettings settings = MetricServiceSettings.newBuilder()
                                                                    .setCredentialsProvider(StackdriverGoogleCredentialsProvider.INSTANCE)
                                                                    .build();

        return StackdriverMeterRegistry.builder(config)
                                       .clock(Clock.SYSTEM)
                                       .metricServiceSettings(() -> settings)
                                       .build();
    }
}

Exception:

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/gc/live/data/size, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[54]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/gc/max/data/size, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[9]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/memory/committed, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[14,48,51,64,74,76]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/memory/used, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[8,15,22,32,39,53]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/threads/states, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[1,25,26,42,44,63]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/process/cpu/usage, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[43]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/process/uptime, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[40]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/system/cpu/usage, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[24]; Points must be written in order. One or more of the points specified had an older start time than the most recent point.: timeSeries[12,13,18,19,49,50]
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:507)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:482)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1156)
        at io.micrometer.stackdriver.StackdriverMeterRegistry.publish(StackdriverMeterRegistry.java:163)
        at io.micrometer.core.instrument.push.PushMeterRegistry.close(PushMeterRegistry.java:69)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at java.util.Collections$SetFromMap.forEach(Collections.java:5476)
        at io.micrometer.core.instrument.composite.CompositeMeterRegistry.close(CompositeMeterRegistry.java:232)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:337)
        at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:271)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1052)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1059)
        at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1035)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1011)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:931)
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/gc/live/data/size, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[54]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/gc/max/data/size, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[9]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/memory/committed, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[14,48,51,64,74,76]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/memory/used, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[8,15,22,32,39,53]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/jvm/threads/states, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[1,25,26,42,44,63]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/process/cpu/usage, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[43]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/process/uptime, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[40]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/system/cpu/usage, Timestamps: {Youngest Existing: '2019/03/27-09:04:51.023', New: '2019/03/27-09:04:51.023'}}: timeSeries[24]; Points must be written in order. One or more of the points specified had an older start time than the most recent point.: timeSeries[12,13,18,19,49,50]
    at io.grpc.Status.asRuntimeException(Status.java:526)
    ... 24 more

[hixichen]

same here with k8s deployment:

     - name: sidecar
       image: gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:0.8.2

prometheus-deployment-7c8ffbbfdd-664xd sidecar level=warn ts=2021-06-16T21:34:15.489Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: timeSeries[0-199]"

Update at 06/29/2021:

Initially, I thought this is caused by missing metrics tags, eg, pod name, which stackdriver can not figure aggregate metrics from multiple pods, but even after I add metrics tags, wan message still exist.

[rocketraman]

@lbaeumer I was having the same issue. I got it working with the information in this comment: https://github.com/micrometer-metrics/micrometer/issues/1907#issuecomment-601346553. The documentation is a bit misleading: it suggests adding the values to the “common tags” but the important values are actually the resourceType and resourceLabels.