[feature-request] enable to using (next_/ last_)run_time in workflows definition

Hi team. Thank you for your developments.

I want to use some variables, current_run_time, next_run_time, last_run_time, are related to scheduled time, and they don’t mean session_time.

In follow workflow definition, we can’t refer to the schedule time.

timezone: UTC
schedule:
  daily> 06:00:00

+session_time:
  echo>: ${session_time}

echo>: ${session_time} outputs first time of the day, ex) 2018-09-05 00:00:00. And we can’t refer to the time when execute the task. I think also need to consideration about naming of these variables.

If you like my idea, I want develop the feature. So please help me to do the task. Best regards.

p.s The other day, in Twitter, I discussed with @hiroyuki-sato about same topic, please refer to following url. ( Sorry, in Japanese) https://twitter.com/hiroysato/status/1036568808385728512

[serihiro]

@onunu As a result of my research, I think it is very difficult to get some datetimes which are related with schedule.

In the Digdag data structure(Database Schema), there is no any relations between tasks and schedules. So when the task is executed, the tasks cannot know whether the task is scheduled or not. So task cannot get informations about schedulers when the task is executed. Digdag scheduler process just submits the task with following each schedule.

schedules table has columns of next_runtime, next_schedule_time, and last_session_time on the table. I think these informations are what you need.

digdag_server=# select * from schedules;
-[ RECORD 1 ]----------+------------------------------
id                     | 1
project_id             | 1
workflow_definition_id | 6
next_run_time          | 1537231320
next_schedule_time     | 1537228800
last_session_time      | 1537142400
created_at             | 2018-09-15 09:41:11.050962+09
updated_at             | 2018-09-18 04:57:46.571914+09
disabled_at            |

So, as a workaround, if you want to get these informations, you can execute http request to Digdag api /api/projects/:id/schedules directly and get them.

$ curl http://localhost:65432/api/projects/2/schedules
{
   "schedules":[
      {
         "id":"1",
         "project":{
            "id":"2",
            "name":"new-project"
         },
         "workflow":{
            "id":"2",
            "name":"new"
         },
         "nextRunTime":"2018-09-20T21:12:00Z",
         "nextScheduleTime":"2018-09-20T00:00:00+00:00",
         "disabledAt":null
      }
   ]
}

You can see the Digdag api endpoints list which are related with schedule here or here

And , if you want to share same value(e.g. a datetime when a task is finished) between different tasks, we are developing new operators param_set> and param_get> . These operators enable to save and read persistent data in DataStore(e.g. PostgreSQL, Redis). https://github.com/treasure-data/digdag/pull/845

[onunu]

I’ ll close this issue, because I understand the followings.

• Task and schedule have no relation.
  • It’s so difficult to get schedule information in tasks.
• We can get schedule information through requests to digdag API as a workaround.