qp.backline.onnx_decoder¶
- onnx_decoder(model, *, provider='auto', device=0, threads=1)[source]¶
Create a coprocessor function that runs an ONNX model on each message, for use with
backline.Each message the controller sends is the model’s input tensor as raw bytes, and each reply is its output tensor. The tensors’ sizes become the placement’s message sizes (
in_bytesandout_bytes), so the controller needs noin_bytesorout_bytesof its own. The model runs on a GPU when the installed onnxruntime has a GPU provider, and on the CPU otherwise.Warning
Backlineis experimental and only usable through the Catalyst compiler.- Parameters:
model (str or os.PathLike) – Path to the
.onnxmodel, which must have one input and one output.provider (str) – The onnxruntime execution provider:
"auto","cpu","migraphx"for AMD GPUs, or"cuda"or"tensorrt"for NVIDIA GPUs. A GPU provider needs an onnxruntime build that contains it."auto"uses the first of"migraphx"and"cuda"that the installed onnxruntime has, or the CPU when it has neither. A GPU provider that is present but cannot be attached stops the coprocessor from starting, rather than falling back to the CPU.device (int) – the GPU a GPU provider runs on, from
0threads (int) – The number of onnxruntime intra-op threads, at least
1. One thread keeps onnxruntime’s thread pool from competing with the transport’s threads.
- Returns:
The function, ready to pass as
coprocessor_fn.- Return type:
- Raises:
FileNotFoundError – If
modeldoes not exist.ValueError – If
provideris unknown,deviceorthreadsis out of range, or the model does not have one input and one output of a supported tensor type.TypeError – If
deviceorthreadsis not an int.ImportError – If no onnxruntime package is installed, or it has no shared library.
Exception – An onnxruntime error, such as
InvalidProtobuf, if onnxruntime’s CPU provider cannot load the model, which it does to read the tensor sizes.
See also
Example
A model taking 120 bytes and returning 121 sets both message sizes:
>>> fn = qp.backline.onnx_decoder("predecoder.onnx") >>> coproc = qp.Coprocessor(hardware="gpu", coprocessor_fn=fn) >>> dev = qp.Backline( ... controller=qp.Controller(), coprocessors=[coproc], transport="memcpy" ... ) >>> dev.placement.in_bytes, dev.placement.out_bytes (120, 121)
Usage Details
Message sizes. Each payload is the input tensor as raw bytes in row-major order, and each reply is the output tensor. A dynamic dimension, such as a batch dimension, is taken as 1. A controller that sets
in_bytesorout_bytesto a size the model does not match is rejected when theBacklineis built.Choosing the device. With
provider="auto", the same program runs on an AMD GPU with theonnxruntime-migraphxpackage, on an NVIDIA GPU withonnxruntime-gpu, and on the CPU with plainonnxruntime."auto"never picks"tensorrt", so pass it explicitly to use it. The provider in use is printed when the coprocessor starts (or when a provider fails to attach)."cpu"and"migraphx"have been tested to work, whereas"cuda"and"tensorrt"are experimental and currently untested.In-process only. The model and onnxruntime paths are resolved on the compiling machine, so the coprocessor runs in the compiling process: it takes neither
executor_optionsnorexecutor, and so cannot setremote. Catalyst rejects a coprocessor running this function that has either.