Docker SNS is not able to publish message to another docker instance via docker link

I have a very basic docker-compose configuration, with 2 nodes. nodejs docker is linked via docker link as well as dependency

• fake-sns
• nodejs (as a subscriber)

The following are the api calls I’m using to test.

Create new topic

aws sns --endpoint-url http://socialcrm.dev:9911 create-topic --name mrt_rules_
{
    "TopicArn": "arn:aws:sns:us-east-1:1485239739350:mrt_rules"
}

Export ARN into env

ARN=arn:aws:sns:us-east-1:1485239739350:mrt_rules ; export ARN

Subscribe nodejs to topic

aws sns --endpoint-url http://socialcrm.dev:9911 subscribe --topic-arn $ARN --protocol http --notification-endpoint http:mrt_rules:8080/sns
{
    "SubscriptionArn": "89540363-acaf-47f6-945b-e86bc03b94d1"
}

Post new message

aws sns --endpoint-url http://socialcrm.dev:9911 publish --topic-arn $ARN --message "New message"
{
    "MessageId": "c6e10fb4-6891-4522-bf2e-e4906091d783"
}

And I get the following errors in Akka (with debug turned on)

06:50:18.125 [sns-akka.actor.default-dispatcher-20] DEBUG akka.io.TcpListener - New connection accepted
06:50:18.132 [sns-akka.actor.default-dispatcher-14] WARN  akka.actor.ActorSystemImpl - Illegal request header: Illegal 'authorization' header: Invalid input '/', expected tchar, OWS, listSep or 'EOI' (line 1, column 49): AWS4-HMAC-SHA256 Credential=AKIAJRU7EX22W6MFQFAA/20170124/ap-southeast-2/sns/aws4_request, SignedHeaders=host;x-amz-date, Signature=a41a998abb3a6c7c25ec4cfd8b9b16aa12888e3497c4a620440b472dc74db3e5
                                                ^
06:50:18.142 [sns-akka.actor.default-dispatcher-15] DEBUG me.snov.sns.actor.SubscribeActor - Sending message c6e10fb4-6891-4522-bf2e-e4906091d783 to http:mrt_rules:8080/sns
06:50:18.143 [sns-akka.actor.default-dispatcher-15] DEBUG me.snov.sns.actor.SubscribeActor - Sending message c6e10fb4-6891-4522-bf2e-e4906091d783 to http:172.18.0.4:8080/sns
06:50:18.147 [sns-akka.actor.default-dispatcher-17] DEBUG akka.actor.ActorSystemImpl - akka-http-sns: Response for
  Request : HttpRequest(HttpMethod(POST),http://socialcrm.dev:9911/,List(Host: socialcrm.dev:9911, Accept-Encoding: identity, X-Amz-Date: 20170124T065018Z, authorization: AWS4-HMAC-SHA256 Credential=AKIAJRU7EX22W6MFQFAA/20170124/ap-southeast-2/sns/aws4_request, SignedHeaders=host;x-amz-date, Signature=a41a998abb3a6c7c25ec4cfd8b9b16aa12888e3497c4a620440b472dc74db3e5, User-Agent: aws-cli/1.10.25 Python/2.7.10 Darwin/16.3.0 botocore/1.5.2, Timeout-Access: <function1>),HttpEntity.Strict(application/x-www-form-urlencoded; charset=UTF-8,Action=Publish&Message=New+message&Version=2010-03-31&TopicArn=arn%3Aaws%3Asns%3Aus-east-1%3A1485239739350%3Amrt_rules),HttpProtocol(HTTP/1.1))
  Response: Complete(HttpResponse(200 OK,List(),HttpEntity.Strict(text/xml; charset=UTF-8,<PublishResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/"><PublishResult><MessageId>c6e10fb4-6891-4522-bf2e-e4906091d783</MessageId></PublishResult><ResponseMetadata><RequestId>a21f6deb-2f4a-4ca5-bc27-95c70ca6dd17</RequestId></ResponseMetadata></PublishResponse>),HttpProtocol(HTTP/1.1)))
Jan 24, 2017 6:50:18 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing request: **Connection refused (Connection refused)**
Jan 24, 2017 6:50:18 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 24, 2017 6:50:18 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing request: **Connection refused (Connection refused)**
Jan 24, 2017 6:50:18 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 24, 2017 6:50:18 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing request: **Connection refused (Connection refused)**
Jan 24, 2017 6:50:18 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
06:50:18.250 [sns-akka.actor.default-dispatcher-20] ERROR akka.actor.OneForOneStrategy - Connection refused (Connection refused)
akka.camel.AkkaCamelException: Connection refused (Connection refused)
	at akka.camel.ProducerSupport$$anonfun$produce$1.applyOrElse(Producer.scala:73) ~[sns.jar:0.1.3]
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484) ~[sns.jar:0.1.3]
	at me.snov.sns.actor.ProducerActor.aroundReceive(ProducerActor.scala:10) ~[sns.jar:0.1.3]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [sns.jar:0.1.3]
	at akka.actor.ActorCell.invoke(ActorCell.scala:495) [sns.jar:0.1.3]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [sns.jar:0.1.3]
	at akka.dispatch.Mailbox.run(Mailbox.scala:224) [sns.jar:0.1.3]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [sns.jar:0.1.3]
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [sns.jar:0.1.3]
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [sns.jar:0.1.3]
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [sns.jar:0.1.3]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [sns.jar:0.1.3]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_111-internal]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_111-internal]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_111-internal]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_111-internal]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_111-internal]
	at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_111-internal]
	at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_111-internal]
	at java.net.Socket.<init>(Socket.java:434) ~[na:1.8.0_111-internal]
	at java.net.Socket.<init>(Socket.java:286) ~[na:1.8.0_111-internal]
	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) ~[sns.jar:0.1.3]
	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) ~[sns.jar:0.1.3]
	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) ~[sns.jar:0.1.3]
	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) ~[sns.jar:0.1.3]
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) ~[sns.jar:0.1.3]
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) ~[sns.jar:0.1.3]
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) ~[sns.jar:0.1.3]
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) ~[sns.jar:0.1.3]
	at org.apache.camel.component.http.HttpProducer.executeMethod(HttpProducer.java:253) ~[sns.jar:0.1.3]
	at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:148) ~[sns.jar:0.1.3]
	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[sns.jar:0.1.3]
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145) ~[sns.jar:0.1.3]
	at akka.camel.ProducerSupport$ProducerChild.produce(Producer.scala:137) ~[sns.jar:0.1.3]
	at akka.camel.ProducerSupport$ProducerChild$$anonfun$receive$1.applyOrElse(Producer.scala:111) ~[sns.jar:0.1.3]
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484) ~[sns.jar:0.1.3]
	at akka.camel.ProducerSupport$ProducerChild.aroundReceive(Producer.scala:108) ~[sns.jar:0.1.3]
	... 9 common frames omitted

I have manually subscribed the nodejs’s IP, and it published message fine, I have also subscribed google.com to it, and it published fine. It seems Akka is unable to resolve the DNS of the docker link for nodejs.

[sam-io]

@s12v I can reproduce this issue using the example docker-compose file that is include in your project. The problem appears to be that when the camel component that is used to send the message to sqs starts up it lists the queues to get their urls. If elasticmq has not been configured with a host name that matches the docker link then it will try to send the message to the wrong url.

I have submitted a pull request that shows the fix.

[A-Scott-Rowe]

@s12v I think my situation was actually a little different. I was using this in a docker container and also a fake sqs in another docker container. The two services worked great independently but when sns tried to connect to sqs to push the message it received a similar(but not the same) error as above.

I was using the same config as in your example but could not resolve it. I ended up finding an all-in-one docker image that works very well. https://github.com/p4tin/GoAws