[Feature Request] Direct SageMaker support?

What is the problem this feature will solve?

A lot of individuals and companies use SageMaker for model training and deployment, but often they are not experts in taking repositories like this one and understanding how to wrap them with SageMaker. So instead, they often default to examples they can find that are already integrated with SageMaker. However in the object detection space, these examples can often be much less capable than MMDetection.

What is the feature you are proposing to solve the problem?

Creating a tools/train_sagemaker.py and an example for training.

What alternatives have you considered?

Right now I have a train_sagemaker.py script that launches training by executingsubprocess.Popen with a command that uses torchrun to launch tools/train.py. For example:

    # Train script config
    launch_config = ["torchrun",
                     "--nnodes", str(world['number_of_machines']), "--node_rank", str(world['machine_rank']),
                     "--nproc_per_node", str(world['number_of_processes']), "--master_addr", world['master_addr'],
                     "--master_port", world['master_port']]

    train_config = [os.path.join(os.environ["MMDETECTION"], "tools/train.py"),
                    config_file,
                    "--launcher", "pytorch",
                    "--work-dir", '/opt/ml/checkpoints']

    if not args.validate:
        train_config.append("--no-validate")

    # Concat Pytorch Distributed Launch config and MMdetection config
    joint_cmd = " ".join(str(x) for x in launch_config+train_config)
    print("Following command will be executed: \n", joint_cmd)

    process = subprocess.Popen(joint_cmd,  stderr=subprocess.STDOUT, stdout=subprocess.PIPE, shell=True)

    while True:
        output = process.stdout.readline()

        if process.poll() is not None:
            break
        if output:
            print(output.decode("utf-8").strip())

    rc = process.poll()

    if process.returncode != 0:
        raise subprocess.CalledProcessError(returncode=process.returncode, cmd=joint_cmd)

But maybe there’s a better way to accomplish this and integrate it more directly?

[austinmw]

No problem, thanks for working on this, let me know if you need any help!

[ZwwWayne]

Hi @austinmw , Thanks for your sharing. The code is definitely helpful to us. We will deeply check the code and may have a design in the following month. It might take several weeks as we do not have AWS services for now and we already have plans in this and the next month.