digdag-ui can't display task log if a task-log store in an S3.

• Digdag 0.9.19

digdag-ui can’t display task log if a task-log store in an S3.

digdag-ui try to fetch %20hoge%20disp_current_date%4059df5e31090013c0.42407%40hsato-mbp13.local.log.gz (Decode string is hoge disp_current_date@[e-mail].) This filename is wrong. It needs to replace (space) to +.

I’m not sure how to fix this yet, But the following ad-hoc patch fix this issue.

https://github.com/treasure-data/digdag/blob/2360e51f549c1bcfddece096081f49e1f0ef80a1/digdag-storage-s3/src/main/java/io/digdag/storage/s3/S3Storage.java#L198-L206

diff --git a/digdag-storage-s3/src/main/java/io/digdag/storage/s3/S3Storage.java b/digdag-storage-s3/src/main/java/io/digdag/storage/s3/S3Storage.java
index 16255a0c7..e0d0842e7 100644
--- a/digdag-storage-s3/src/main/java/io/digdag/storage/s3/S3Storage.java
+++ b/digdag-storage-s3/src/main/java/io/digdag/storage/s3/S3Storage.java
@@ -195,12 +195,19 @@ public class S3Storage
     }
 
     @Override
-    public Optional<DirectDownloadHandle> getDirectDownloadHandle(String key)
+    public Optional<DirectDownloadHandle> getDirectDownloadHandle(String key2)
     {
+        String key;
+        key = key2.replaceAll(" ","+");
         GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(bucket, key);
         req.setExpiration(Date.from(Instant.now().plusSeconds(10*60)));
 
         String url = client.generatePresignedUrl(req).toString();
 
         return Optional.of(DirectDownloadHandle.of(url));
     }

log-server.type: s3
log-server.s3.endpoint: http://xxx.xxx.xxx.xxx:9000
log-server.s3.bucket: log
log-server.s3.credentials.access-key-id: XXXXXXXXXXXXXXXXXXXX
log-server.s3.credentials.secret-access-key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

[hiroyuki-sato]

Hello, @David-Development

IIUC, digdag-ui request the log with + sign, not whitespace if task storage server use S3. However, It seems that digdag-ui request the log with whitespace if a task storage server uses minio.

We need to take a look more deeply. I’ll try to find someone for investigating this issue.

[David-Development]

@hiroyuki-sato thank you for the quick response!

The plan sounds good, however I still have a couple of question. The name of the tasks is shown with a leading + sign. e.g. +myworkflow+mytask.... The logs are stored on the s3 storage in the same format. Why is the UI requesting the logs with whitespaces instead of the + sign? Isn’t that true for the AWS Storage as well? (I don’t have an AWS account either - so I can’t verify this right now).

Or is it some kind of feature of AWS, that it converts whitespaces into + signs and that’s why it working there and not on minio? I guess there is some magic going on in the s3 aws storage backend as described here and here.