IOException with AWSBatch

I’m having some problems running the AWSBatch examples from the aws opendata page - https://docs.opendata.aws/genomics-workflows/aws-batch/configure-aws-batch-start/ - as far as I can tell I have correctly set the permissions, however when I try running the examples I get the error below. It looks like the output files aren’t being written the S3 bucket.

I’m probably missing something obvious, but I can’t work out what I’m doing wrong. Would anyone have any suggestions for where might be good to look for errors?

Caused by: java.io.IOException: Could not read from s3://concr-genomics-results/cromwell-execution/wf_hello/b7e4cdce-ff14-4509-aec3-b226ed31043c/call-hello/hello-rc.txt: s3://s3.amazonaws.com/concr-genomics-results/cromwell-execution/wf_hello/b7e4cdce-ff14-4509-aec3-b226ed31043c/call-hello/hello-rc.txt
	at cromwell.engine.io.nio.NioFlow$$anonfun$withReader$2.applyOrElse(NioFlow.scala:146)
	at cromwell.engine.io.nio.NioFlow$$anonfun$withReader$2.applyOrElse(NioFlow.scala:145)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
	at scala.util.Failure.recoverWith(Try.scala:232)
	at cromwell.engine.io.nio.NioFlow.withReader(NioFlow.scala:145)
	at cromwell.engine.io.nio.NioFlow.limitFileContent(NioFlow.scala:154)
	at cromwell.engine.io.nio.NioFlow.$anonfun$readAsString$1(NioFlow.scala:98)
	at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:85)
	at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:336)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:357)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:303)
	at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.nio.file.NoSuchFileException: s3://s3.amazonaws.com/concr-genomics-results/cromwell-execution/wf_hello/b7e4cdce-ff14-4509-aec3-b226ed31043c/call-hello/hello-rc.txt
	at org.lerch.s3fs.S3FileSystemProvider.newInputStream(S3FileSystemProvider.java:350)
	at java.nio.file.Files.newInputStream(Files.java:152)
	at better.files.File.newInputStream(File.scala:337)
	at cromwell.core.path.BetterFileMethods.newInputStream(BetterFileMethods.scala:240)
	at cromwell.core.path.BetterFileMethods.newInputStream$(BetterFileMethods.scala:239)
	at cromwell.filesystems.s3.S3Path.newInputStream(S3PathBuilder.scala:156)
	at cromwell.core.path.EvenBetterPathMethods.mediaInputStream(EvenBetterPathMethods.scala:94)
	at cromwell.core.path.EvenBetterPathMethods.mediaInputStream$(EvenBetterPathMethods.scala:91)
	at cromwell.filesystems.s3.S3Path.mediaInputStream(S3PathBuilder.scala:156)
	at cromwell.engine.io.nio.NioFlow.$anonfun$withReader$1(NioFlow.scala:145)
	at cromwell.util.TryWithResource$.$anonfun$tryWithResource$1(TryWithResource.scala:14)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.util.TryWithResource$.tryWithResource(TryWithResource.scala:10)
	... 14 more

[matthewghgriffiths]

OK I think I got it working.

It turns out the custom AMI I created was incorrect.

When making a custom AMI using the cloud formation stacks as described here.

The AMI type needs to be specified as ‘cromwell’ and the Scratch mount point needs to be specified as \cromwell_mount

This information is stated elsewhere, but perhaps this page should be updated to include this information?

[multimeric]

Because the stack has changed a lot over time, I wouldn’t assume the bug discussed in this thread is the same as your bug. I’ll provide some more detail in the thread you created.