qp.backline.onnx_decoder

onnx_decoder(model, *, provider='auto', device=0, threads=1)[source]

Create a coprocessor function that runs an ONNX model on each message, for use with backline.

Each message the controller sends is the model’s input tensor as raw bytes, and each reply is its output tensor. The tensors’ sizes become the placement’s message sizes (in_bytes and out_bytes), so the controller needs no in_bytes or out_bytes of its own. The model runs on a GPU when the installed onnxruntime has a GPU provider, and on the CPU otherwise.

Warning

Backline is experimental and only usable through the Catalyst compiler.

Parameters:
  • model (str or os.PathLike) – Path to the .onnx model, which must have one input and one output.

  • provider (str) – The onnxruntime execution provider: "auto", "cpu", "migraphx" for AMD GPUs, or "cuda" or "tensorrt" for NVIDIA GPUs. A GPU provider needs an onnxruntime build that contains it. "auto" uses the first of "migraphx" and "cuda" that the installed onnxruntime has, or the CPU when it has neither. A GPU provider that is present but cannot be attached stops the coprocessor from starting, rather than falling back to the CPU.

  • device (int) – the GPU a GPU provider runs on, from 0

  • threads (int) – The number of onnxruntime intra-op threads, at least 1. One thread keeps onnxruntime’s thread pool from competing with the transport’s threads.

Returns:

The function, ready to pass as coprocessor_fn.

Return type:

CoprocessorFunction

Raises:
  • FileNotFoundError – If model does not exist.

  • ValueError – If provider is unknown, device or threads is out of range, or the model does not have one input and one output of a supported tensor type.

  • TypeError – If device or threads is not an int.

  • ImportError – If no onnxruntime package is installed, or it has no shared library.

  • Exception – An onnxruntime error, such as InvalidProtobuf, if onnxruntime’s CPU provider cannot load the model, which it does to read the tensor sizes.

Example

A model taking 120 bytes and returning 121 sets both message sizes:

>>> fn = qp.backline.onnx_decoder("predecoder.onnx")
>>> coproc = qp.Coprocessor(hardware="gpu", coprocessor_fn=fn)
>>> dev = qp.Backline(
...     controller=qp.Controller(), coprocessors=[coproc], transport="memcpy"
... )
>>> dev.placement.in_bytes, dev.placement.out_bytes
(120, 121)

Message sizes. Each payload is the input tensor as raw bytes in row-major order, and each reply is the output tensor. A dynamic dimension, such as a batch dimension, is taken as 1. A controller that sets in_bytes or out_bytes to a size the model does not match is rejected when the Backline is built.

Choosing the device. With provider="auto", the same program runs on an AMD GPU with the onnxruntime-migraphx package, on an NVIDIA GPU with onnxruntime-gpu, and on the CPU with plain onnxruntime. "auto" never picks "tensorrt", so pass it explicitly to use it. The provider in use is printed when the coprocessor starts (or when a provider fails to attach). "cpu" and "migraphx" have been tested to work, whereas "cuda" and "tensorrt" are experimental and currently untested.

In-process only. The model and onnxruntime paths are resolved on the compiling machine, so the coprocessor runs in the compiling process: it takes neither executor_options nor executor, and so cannot set remote. Catalyst rejects a coprocessor running this function that has either.