"Invalid number format [epoch_millis#]" in grafana dashboards

Hi, I just integrated stagemonitor-0.29.0 into my servlet running on a wildfly-9. The data is correctly stored in elasticsearch-2.4.1 and all the dashboards show up in kibana-4.6.2 and grafana-3.1.1.

However, all the metrics in the grafana dashboards fail with the following error: Invalid number format [epoch_millis#] (note the hash at the end) In kibana everything works fine.

Did I miss something? Are the default dashboards broken or could this be a bug in grafana?

Log from elasticsearch

[2016-11-14 17:47:43,853][DEBUG][action.search            ] [Plazm] All shards failed for phase: [query]
RemoteTransportException[[Plazm][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: AggregationExecutionException[Invalid number format [epoch_millis#]];
Caused by: AggregationExecutionException[Invalid number format [epoch_millis#]]
	at org.elasticsearch.search.aggregations.support.format.ValueParser$Number$Pattern.parseLong(ValueParser.java:179)
	at org.elasticsearch.search.aggregations.bucket.histogram.ExtendedBounds.processAndValidate(ExtendedBounds.java:52)
	at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.doCreateInternal(HistogramAggregator.java:201)
	at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.doCreateInternal(HistogramAggregator.java:147)
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
	at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
	at org.elasticsearch.search.aggregations.AggregatorFactory.asMultiBucketAggregator(AggregatorFactory.java:119)
	at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.doCreateInternal(HistogramAggregator.java:193)
	at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.doCreateInternal(HistogramAggregator.java:147)
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
	at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
	at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:76)
	at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:69)
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48)
	at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregator.<init>(TermsAggregator.java:149)
	at org.elasticsearch.search.aggregations.bucket.terms.AbstractStringTermsAggregator.<init>(AbstractStringTermsAggregator.java:40)
	at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.<init>(GlobalOrdinalsStringTermsAggregator.java:79)
	at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$ExecutionMode$2.create(TermsAggregatorFactory.java:72)
	at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:246)
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
	at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
	at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:87)
	at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:85)
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Complete grafana error:

{
    "root_cause": [
        {
            "type": "aggregation_execution_exception",
            "reason": "Invalid number format [epoch_millis#]"
        }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
        {
            "shard": 0,
            "index": "stagemonitor-metrics-2016.11.14",
            "node": "PgQHT7nRQ5OKUYZuJK-JSA",
            "reason": {
                "type": "aggregation_execution_exception",
                "reason": "Invalid number format [epoch_millis#]"
            }
        }
    ]
}

[felixbarny]

Looks good. Delete the index via curl -XDELETE localhost:9200//stagemonitor-metrics-2016.11.14 and see the then created index has the correct mapping.

[felixbarny]

Great! Glad I could help. One scenario I can imagine which can lead to the situation is that you started Elasticsearch after your application. On application start, stagemonitor tries to put the index mapping to Elasticsearch. If ES is not available this will obviously fail.