[FEATURE] Integration with SageMaker

🚨🚨 Feature Request

• Related to an existing Issue
• A new implementation (Improvement, Extension)

Is your feature request related to a problem?

Include functionality that allows SageMaker to ingest data from Hub.

If your feature will improve HUB

SageMaker can easily fetch data from local filesystems and s3. By building an interface for SageMaker, we tie Hub to one of the most popular ML tools.

Description of the possible solution

We would have to duplicate a Hub dataset onto s3, which is then accessible to SageMaker.

For example, this is how SageMaker handles local files:

 input_mode (str): The input mode that the algorithm supports
                (default: 'File'). Valid modes: 'File' - Amazon SageMaker copies
                the training dataset from the S3 location to a local directory.
                'Pipe' - Amazon SageMaker streams data directly from S3 to the
                container via a Unix-named pipe. This argument can be overriden
                on a per-channel basis using

[KrishnaChaitanya1]

Hi @mynameisvinn . I have couple of doubts regarding this feature.

Firstly, where does the scope of Hub end. Like using boto3 to move the dataset from Hub to S3 and then to SageMaker or are you thinking something else?

Secondly, this is a noob question. Are we trying to use SageMaker in training model as well? I think this doesn’t make sense as it doesnt work for all the end users right?

Lastly, since we can build data pipelines in Hub, can we try if we can integrate this with Kinesis Firehose for streaming data? I haven’t worked on Kinesis much, but I think this will be a promising feature

[KrishnaChaitanya1]

Sure @mynameisvinn