apm agent finds server but failed to send data (security manager)

Describe the bug

Trying to use apm on elasticsearch jvm to get jvm metrics. APM agent finds APM server, but fails to send any data with java.net.SocketPermission access denied.

Steps to reproduce

elasticapm.properties:

service_name=elasticsearch
server_urls=http://apm:8200

dockercompose.yml:

  elasticsearch:
    image: elasticsearch:7.9.3
    mem_limit: 768m
    expose:
      - "9200"
    environment:
      ES_JAVA_OPTS: -Xmx768m
      HTTP_PORT: 9200
      discovery.type: single-node
      ELASTIC_APM_SERVER_URLS: http://apm:8200
    labels:
      co.elastic.logs/module: elasticsearch
      app: elasticsearch
    volumes:
      - es-data:/usr/share/elasticsearch/data
      - ./config/es/jvm.options:/usr/share/elasticsearch/config/jvm.options:ro
      - ./config/es/apm:/usr/share/elasticsearch/config/apm:ro
  apm:
    image: store/elastic/apm-server:7.9.3
    mem_limit: 64m
    depends_on:
      - elasticsearch
    expose:
      - "8200"
    labels:
      app: apm
    environment:
      apm-server.host: localhost:8200
      output.elasticsearch.hosts: http://elasticsearch:9200
    volumes:
      - apm-data:/usr/share/apm-server/data

Expected behavior

Data sent to apm server.

Debug logs

docker logs docker_elasticsearch_1 | grep apm

2021-01-20 18:17:39,569 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.20.0 as elasticsearch on Java 15 Runtime version: 15+36-1562 VM version: 15+36-1562 (Oracle Corporation) Linux 4.19.128-microsoft-standard
2021-01-20 18:17:39,570 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-11682935632529430251, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -javaagent:/usr/share/elasticsearch/config/apm/elastic-apm-agent.jar, -Des.cgroups.hierarchy.override=/, -Xmx768m, -XX:MaxDirectMemorySize=402653184, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]
2021-01-20 18:17:39,571 [main] WARN  co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2021-01-20 18:17:41,275 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2020-10-16T08:05:05Z",  "build_sha": "af7b123f289d79c34c9da11471803f457eafa5c0",  "version": "7.9.3"}
2021-01-20 18:17:41,301 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
{"type": "server", "timestamp": "2021-01-20T18:17:43,849Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "daa16e1cfd08", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-11682935632529430251, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -javaagent:/usr/share/elasticsearch/config/apm/elastic-apm-agent.jar, -Des.cgroups.hierarchy.override=/, -Xmx768m, -XX:MaxDirectMemorySize=402653184, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
2021-01-20 18:18:11,514 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: access denied ("java.net.SocketPermission" "apm:8200" "connect,resolve")

[SylvainJuge]

Hi @cdalexndr , are you using a security manager in your application ?

The following line in the log seems to indicate that the agent does not have permission to open a socket to reach apm-server.

java.security.AccessControlException: access denied ("java.net.SocketPermission" "apm:8200" "connect,resolve")

If that’s the case, you will have to modify the security policy of your setup to include configuration for the agent:

grant codeBase "file:/path/to/agent/elastic-agent.jar" {
  permission java.security.AllPermission;
};

I have to admit that it’s something probably missing from our documentation.

[eyalkoren]

Yeah, great idea, we can create this snippet with the actual agent location, which is very useful! 👍