사전 공개: Qt AI 추론 API 살펴보기 (Proof-of-Concept)

Usually more than one AI model is needed to implement a use case. Also, there is a need to connect them together to create a pipeline of AI models, here a "speech->text->text->speech" pipeline for a speech based AI chat application.

There are typically two ways to implement that, either by directly accessing every AI model from the application and creating the AI pipeline inside the application, or by creating an external, custom made and use case specific AI pipeline that the application uses.

In the first case, on the left side of Image 1, the application accesses each AI model via some API, typically a REST API or sometimes via gRPC or WebSocket protocol. The API is provided by a Python server running either locally in the same device or in the cloud. The Python server then accesses a Python API of some AI Framework to do the AI model loading and the actual deployment of the AI model, called as "Inference". The application also needs to integrate the different AI models, with different APIs, into an AI pipeline. This results in many thousands of lines of code that is both use case and AI Framework dependent and that could be avoided.

In the second case, on the right side of the Image 1, the external AI pipeline moves that code away from the application to an external, use-case specific AI pipeline. It means that the AI pipeline needs to be implemented for each use case separately. The application is now accessing fully use-case specific private APIs of the AI pipeline. Any change in the use case requires changes on both the application and the AI pipeline. Again there are thousands of lines of code that are use-case and AI Framework specific and could be avoided.

In both cases, for any video, audio or speech data there is a need to use some platform dependent Multimedia Framework to record the media and to play it, meaning that the application and/or the Python servers need to be implemented differently for each target platform.

How can Qt do it better?

For media recording and playback the Qt Framework already provides a cross-platform Qt Multimedia module and its Qt Multimedia API. Since the Qt Framework supports also Python that API can be used both in the application side and in the Python servers. This means that both are platform independent i.e. can be used in many target platforms.

To provide the cross-platform and cross-AI-framework API for Inference, there is now the new Qt AI Inference API (PoC). It is provided by a new "Qt Data Processing" module. For each AI Framework there is its own backend adaptation plugin that connects to the Python server to do the Inference.

The Qt Framework already provides its own mechanism for inter-process communication, called as Qt Remote Objects module (QtRO). This enables direct communication between the backends and the Python servers without adding code for message encoding, decoding, sending and receiving.

In this way thousands of lines of code is saved, the complexity of the application is reduced, and both the client and the Python servers are usable in any supported platforms and AI Frameworks without modifications. More importantly, the AI pipeline is automatically created in the background by simply defining it in QML in the application. This enables seamless use of AI elements for AI models and pipelines together with the UI elements to let the AI elements control both the application logic and UI behaviour. This enables use of AI in more comprehensive ways than before.

The new Data Processing module provides the new Qt AI Inference API and different AI backend plugins to provide adaptation between it and different Inference services and AI frameworks. An Inference service provides a higher level API that hides the supported AI frameworks.

Each embedded board vendor provides one or two AI frameworks for Inference operation. E.g., one can be a proprietary AI framework for generative AI optimized for the available HW acceration on the board and the other for more simple AI models.

The AI frameworks are using either CPU, GPU or some AI processor for executing the inference operation using parallel processing techniques.

A multimedia framework is needed for image, video and audio/speech processing and generation for decoding and encoding the data and for converting the data to be suitable for the inference operation. It is needed also for accessing audio or video sources, e.g. microphones and cameras. Likewise it is needed for playing audio or video. The existing Qt Multimedia module provides a cross-platform API for these services hiding the platform specific multimedia framework.