Intel MKL FATAL ERROR in 20.11 and 20.12

Description A clear and concise description of what the bug is.

• When trying to run a PyTorch script which was traced on CPU, the error: Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so is thrown. This is a CPU script that is working on 20.10, but fails on 20.11 and 20.12. After looking around a bit, I was able to get around that error by setting LD_PRELOAD="/opt/tritonserver/backends/pytorch/libmkl_avx2.so:/opt/tritonserver/backends/pytorch/libmkl_def.so", but now a different but similar error is thrown: Intel MKL FATAL ERROR: cannot load libmkl_vml_avx2.so or libmkl_vml_def.so..

When looking in the 20.12 container, I get nothing:

root@py-svc-triton-server-669bf765b-svvzm:/opt/tritonserver# find . -name '*libmkl_vml*'
root@py-svc-triton-server-669bf765b-svvzm:/opt/tritonserver# 

When looking in the 20.10 container, I can find one of the files from that error (which seems to be enough):

root@py-svc-triton-server-f744f4657-mqmk5:/opt/tritonserver# find . -name '*libmkl_vml*'
./lib/pytorch/libmkl_vml_def.so
root@py-svc-triton-server-f744f4657-mqmk5:/opt/tritonserver# 

Triton Information What version of Triton are you using?

• Currently using 20.10 and trying to upgrade to 20.12 Are you using the Triton container or did you build it yourself?
• Using the triton container

To Reproduce Steps to reproduce the behavior.

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

• Using a cpu traced PyTorch script using torch==1.5.

Expected behavior A clear and concise description of what you expected to happen.

• Expect a cpu trace PyTorch script which ran on 20.10 to continue working on 20.11 and 20.12.

[Debangshu132]

Same issue here, does not work in 20.11 or 20.12 but works fine in 20.10

[HarmitMinhas96]

@CoderHam Thank you for the fix! Sorry I was not able to get back with a reproducible example