Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error during fio job

See original GitHub issue

I’m not really sure what’s happening but here’s the most relevant output IMO:

2016-02-09 10:40:14,531 8dfe8095 MainThread fio(1/1) INFO     Running: ssh -A -p 22 ubuntu@52.34.238.33 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /var/folders/1n/619vv5jj2sv9cd8hgnr72md40000gn/T/perfkitbenchmarker/run_8dfe8095/perfkitbenchmarker_keyfile nohup python /tmp/pkb/execute_command.py --stdout /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stdout --stderr /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stderr --status /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.status --command 'sudo /tmp/pkb/fio/fio --output-format=json --directory=/scratch /tmp/pkb/fio.job' 1> /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.log 2>&1 &
2016-02-09 10:40:15,394 8dfe8095 MainThread fio(1/1) INFO     Running: ssh -A -p 22 ubuntu@52.34.238.33 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /var/folders/1n/619vv5jj2sv9cd8hgnr72md40000gn/T/perfkitbenchmarker/run_8dfe8095/perfkitbenchmarker_keyfile python /tmp/pkb/wait_for_command.py --stdout /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stdout --stderr /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stderr --status /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.status --delete
2016-02-09 10:53:11,852 8dfe8095 MainThread fio(1/1) INFO     Ran ssh -A -p 22 ubuntu@52.34.238.33 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /var/folders/1n/619vv5jj2sv9cd8hgnr72md40000gn/T/perfkitbenchmarker/run_8dfe8095/perfkitbenchmarker_keyfile python /tmp/pkb/wait_for_command.py --stdout /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stdout --stderr /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stderr --status /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.status --delete. Got return code (6).

STDERR: Warning: Permanently added '52.34.238.33' (RSA) to the list of known hosts.
fio: pid=0, err=2/file:filesetup.c:87, func=open, error=No such file or directory
fio: pid=0, err=2/file:filesetup.c:87, func=open, error=No such file or directory
fio: pid=0, err=2/file:filesetup.c:87, func=open, error=No such file or directory
fio: pid=0, err=2/file:filesetup.c:87, func=open, error=No such file or directory
fio: pid=0, err=2/file:filesetup.c:87, func=open, error=No such file or directory
fio: pid=0, err=2/file:filesetup.c:87, func=open, error=No such file or directory

2016-02-09 10:53:11,856 8dfe8095 MainThread fio(1/1) INFO     Running: ssh -A -p 22 ubuntu@52.34.238.33 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /var/folders/1n/619vv5jj2sv9cd8hgnr72md40000gn/T/perfkitbenchmarker/run_8dfe8095/perfkitbenchmarker_keyfile cat /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.log
2016-02-09 10:53:12,845 8dfe8095 MainThread fio(1/1) WARNING  Exception during RobustRemoteCommand. Wrapper script log:
INFO:root:Acquiring lock on /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.status
INFO:root:Started pid 10159: sudo /tmp/pkb/fio/fio --output-format=json --directory=/scratch /tmp/pkb/fio.job
INFO:root:Waiting on PID 10159
INFO:root:Return code: 6

2016-02-09 10:53:12,846 8dfe8095 MainThread fio(1/1) ERROR    Error during benchmark fio
Traceback (most recent call last):
  File "/Users/dhyon/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 376, in RunBenchmark
    detailed_timer)
  File "/Users/dhyon/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 260, in DoRunPhase
    samples = benchmark.Run(spec)
  File "/Users/dhyon/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/fio_benchmark.py", line 475, in Run
    RunIt()
  File "/Users/dhyon/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/fio_benchmark.py", line 455, in RunIt
    stdout, stderr = vm.RobustRemoteCommand(fio_command, should_log=True)
  File "/Users/dhyon/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 156, in RobustRemoteCommand
    return self.RemoteCommand(' '.join(wait_command), should_log=should_log)
  File "/Users/dhyon/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 313, in RemoteCommand
    suppress_warning, timeout)
  File "/Users/dhyon/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 374, in RemoteHostCommand
    raise errors.VirtualMachine.RemoteCommandError(error_text)
RemoteCommandError: Got non-zero return code (6) executing python /tmp/pkb/wait_for_command.py --stdout /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stdout --stderr /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stderr --status /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.status --delete
Full command: ssh -A -p 22 ubuntu@52.34.238.33 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /var/folders/1n/619vv5jj2sv9cd8hgnr72md40000gn/T/perfkitbenchmarker/run_8dfe8095/perfkitbenchmarker_keyfile python /tmp/pkb/wait_for_command.py --stdout /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stdout --stderr /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.stderr --status /tmp/pkb/cmd3632414d-742b-4448-b99e-9c814f65ecc0.status --delete

This is happening consistently while other tests such as unixbench are working fine.

Issue Analytics

State:
Created 8 years ago
Comments:7 (4 by maintainers)

Top GitHub Comments

1reaction

noahlcommented, Feb 10, 2016

Hi @jntkflow , can you share what cloud and OS image you’re using here? The full command you gave to PKB, and the full output, would also be great. I’d like to try to reproduce your setup.

I’ve seen errors like this before, and they’re often related to the --directory argument. That’s the first thing I would look into, but I can’t work on it myself until I can reproduce.

0reactions

noahlcommented, Feb 19, 2016

Hi @jntkflow , any update on how your benchmarking went? Since you said your custom job file contains parameters that don’t exist, I’m going to close this bug for now, but please reopen if you think there’s an issue in PKB!