❓ [Question] TRTorch and Pytorch Serve

[p1x31]

❓ Question

Is it possbile to use TRTorch with TorchServe?
If not what is the best way to deploy TRTorch programs?

What you have already tried

In the documentation, it is said I can continue to use programs via PyTorch API
I have converted all my models to TRTorch.

Environment

Build information about the TRTorch compiler can be found by turning on debug messages

• PyTorch Version (e.g., 1.0):
• CPU Architecture:
• OS (e.g., Linux):
• How you installed PyTorch (conda, pip, libtorch, source):
• Build command you used (if compiling from source):
• Are you using local sources or building from archives:
• Python version:
• CUDA version:
• GPU models and configuration:
• Any other relevant information:

Additional context

[narendasan]

I think you should be able to with some slight modifications to torchserve. Basically you just need to load the TRTorch runtime in your handler and also insure that the input size to your model is in the range specified at compile time. I tried out creating a mar file from a TRTorch compile TorchScript file from the torch-model-archiver tool and that seemed to work. Let us know if you try it and if you are successful or hit any issues, we'd love to hear about it

[p1x31]

Thanks for the heads up. I'll close this question for now and will let you know the results.