How do I bootstrap packages available only within pip for running on Batch

Running on AWS I would usually define a step:

@batch(cpu=1, memory=5000)
@conda(libraries={'pandas': '1'})
@step
def hello(self):
    do stuff...

However, I am working with deeplearning libraries (MXnet/tensorflow/pytorch), and they are not particularly up to date on conda and it is best to install with pip.

How do I define pip dependancies for this step?

[dibgerge]

Thanks, this solved it! I didn’t see the topic. I was looking at the docs, and I used the command there: os.system('pip install my_package'), which was invoking the wrong python interpreter in the container. Didn’t occur to me to use sys.executable.

Are there plans to integrate this to have something like this instead:

@conda(libraries={...}, pip={...})

so that pip libraries can be cached along with the conda libraries in s3?

[savingoyal]

@dibgerge There is some discussion on that topic in #24. You can install your pip packages in your step by executing - subprocess.run([sys.executable, '-m', 'pip', 'install', '--quiet', ...]).