No space left on device after upgrading to 2.289.2

Describe the bug Our integration test build step started failing after the runner upgraded to 2.289.2.

To Reproduce Sorry for the terrible bug description.

Steps to reproduce the behavior:

1. Go to https://github.com/Project-MONAI/monai-deploy-informatics-gateway/actions/runs/2093298340
2. The docker-compose log command would eventually time out and the job would get canceled.

System.IO.IOException: No space left on device : '/home/runner/runners/2.289.2/_diag/Worker_20220405-003310-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.289.2/_diag/Worker_20220405-003310-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Error(Exception exception)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.289.2/_diag/Worker_20220405-003310-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.Diagnostics.TraceSource.Flush()
   at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
   at GitHub.Runner.Common.TraceManager.Dispose()
   at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
   at GitHub.Runner.Common.HostContext.Dispose()
   at GitHub.Runner.Worker.Program.Main(String[] args)

Expected behavior Build step to pass https://github.com/Project-MONAI/monai-deploy-informatics-gateway/actions/runs/2080265670

Runner Version and Platform

Version of your runner? 2.289.2

OS of the machine running the runner? Linux

What’s not working?

See https://github.com/Project-MONAI/monai-deploy-informatics-gateway/runs/5826064123?check_suite_focus=true

Job Log Output

Can’t get to the output of the step as it spins forever.

Runner and Worker’s Diagnostic Logs

NA

[jerrychong25]

In my case, I’ve used Free Disk Space Ubuntu in GitHub Actions to solve my problem.

Free Disk Space Ubuntu GitHub Actions Sample Code:

name: sample-github-actions

jobs:
  sample_job:
    # The host should always be Linux
    runs-on: ubuntu-20.04
    name: Build on ubuntu-20.04 armv8
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2

      - name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
        uses: jlumbroso/free-disk-space@main
        with:
          # This might remove tools that are actually needed, if set to "true" but frees about 6 GB
          tool-cache: false

          # All of these default to true, but feel free to set to "false" if necessary for your workflow
          android: true
          dotnet: true
          haskell: true
          large-packages: true
          swap-storage: true

Hope this helps those face same issue.

[ruvceskistefan]

Thanks for your reply. Then I will close this issue for now, and if there are more users with a similar problem, we will reopen it. Thanks again @mocsharp!