Capture logs for individual jobs

It should be possible to easily inspect logs of a single job that has run.

Option 1

Update the logger formatting to include the task name & id so that backend logs can be easily filtered.

Option 2

• Dynamically create new logging handlers & filters when a job is run to collect logs only for the job.
• Save these in a new table associated with the job.
• Provide a way to access logs from job instances

job = app.job_manager.list_job(id=1)
for attempt in jobs.attempts:
    print(attempt.logs)

CREATE TABLE procrastinate_logs (
    id BIGSERIAL PRIMARY KEY,
    job_id integer NOT NULL REFERENCES procrastinate_jobs ON DELETE CASCADE,
    attempt_id integer NOT NULL,
    logs TEXT,
);

[ewjoachim]

Do you have a roadmap?

Not really at the point. I never found myself in an position that I’d need a roadmap for an opensource project. The roadmap is “eventually, let’s tackle all the issues” 😄

[ewjoachim]

Nice code !

I believe you don’t have to attach a new handler for every task, and remove it afterwards. Your approach of using a QueueHandler is spot on for separation of concerns. You could have a single queue handler. On the queue consumer side, you would store logs based on the job id, and when you receive a finish_task log, you know it’s over. (Additionally, if you’re afraid of missing a task end, you could always track the worker id and check whenever the same worker_id is associated to a different job id.

The only thing is where you would store the tasks, and I believe it’s perfectly appropriate to store them in the same database as procrastinate (even re-use the connection if you want), but I’m still unsure about Procrastinate doing it itself, this really seems like a specific usecase.

Let’s say that as long as we don’t store task results, I’m not comfortable storing logs 😃