Metrics not written to Stackdriver when using resource type other than global

Please read Tom’s post on down to my post in the previous reported bug which was support more resource types and while we can set another one, all writing of metrics then fails to GCP.

According to the current docs for Stackdriver Monitoring only these resource types support custom metrics:

• aws_ec2_instance
• dataflow_job
• gce_instance
• gke_container
• generic_node
• generic_task
• global
• k8s_cluster
• k8s_container
• k8s_node
• k8s_pod

Each of these resource types has different requirements for the labels provided. Currently only the projectId is added as a label which is only valid for the global resource type. The documentation even recommends not to use global

Use this resource when no other resource type is suitable. For most use cases, generic_node or generic_task are better choices than global.

Configuring the registry for any other resource type results in errors buried in Stackdriver logs that look like:

i.g.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (location namespace node_id).: timeSeries[0]

This could also be contributing a contributing factor for #1001 as the MetricServiceClient fails silently and just writes a log to Stackdriver.

Updating the StackdriverMeterRegistry where it creates the MonitoredResource to include the required labels for each resource type will allow for metrics to be associated with these other resource types.

https://github.com/micrometer-metrics/micrometer/blob/6d615c4af01dcc8bf9e9b4d64c5bce7aba23b444/implementations/micrometer-registry-stackdriver/src/main/java/io/micrometer/stackdriver/StackdriverMeterRegistry.java#L338-L341

@jkschneider - if you’re able to provide some guidance about pulling metadata from the Google Cloud compute metadata client I’d be happy to attempt a PR

_Originally posted by @tomwscott in https://github.com/micrometer-metrics/micrometer/issues/1003#issuecomment-556118415_

[deanorderly]

For completeness, in cloud run, this is what we do now

		String projectId = MetadataConfig.getProjectId();
		String service = System.getenv("K_SERVICE");
		String revisionName = System.getenv("K_REVISION");
		String config = System.getenv("K_CONFIGURATION");
		String zone = MetadataConfig.getZone();

		String instanceId = UUID.randomUUID().toString();
		Map<String, String> map = new HashMap<>();
		map.put("namespace", service);
		map.put("job", instanceId);
		map.put("task_id", revisionName);
		map.put("location", zone);

and that map is passed back in our config object like so…

private static class OurGoogleConfig implements StackdriverConfig {

	private String projectId;
	private Map<String, String> resourceLabels;

	public OurGoogleConfig(String projectId, Map<String, String> resourceLabels) {
		this.projectId = projectId;
		this.resourceLabels = resourceLabels;
	}

	@Override
	public String projectId() {
		return projectId;
	}
	@Override
	public String get(String key) {
		return null;
	}
	@Override
	public String resourceType() {
		return "generic_task";
	}

	@Override
	public Map<String, String> resourceLabels() {
		//they call this EVERY time, so save on memory by only passing the same
		//map every time instead of re-creating it...
		return resourceLabels;
	}
};

[shakuzen]

Glad you both got this working. @deanorderly could you open a separate issue (or pull request) about making what you need more configurable? Thanks.