This post includes some additional background information to support and
clarify our recent QtMobility Multimedia Blog post.
General architecture
The above block diagram gives a high-level view of multimedia in Qt.
Services
The Multimedia services classes that are in Qt 4.6 in the QtMultimedia module
form Qt's access to the lowest level of Multimedia functionality. The Audio
classes provide a cross platform means to access audio devices, to send data to
or receive data from them, the audio data would usually be PCM, but could be in
other formats as well. The Video classes have a similar purpose, those in 4.6
exist to provide an abstraction for video outputs.
The Audio classes can be used in a self contained way when you are manipulating
or creating your audio data directly in your application. Some examples of this
would be, a sound recorder, an effect generator, VOIP client, event sounds,
games or a situation requiring intermittent or lower-latency playback.
The service classes are also meant for building other media functionality; to
simplify and provide cross platform support for the building of playback and
recording functionality.
High Level APIs
Phonon
Phonon is a high level playback API. Phonon provides facilities for playback of
media content. Phonon is also currently used in Qt for media playback. We will
continue to maintain Phonon for the duration of the 4.x series, but new
functionality will be added in the new multimedia module.
So why didn't we try and improve Phonon? There are a lot of interconnected
reasons that would prevent Phonon from moving forward. We felt that that there
would either need to be too much compromise or major source-compatibility
breaking changes (let alone BC breakage), in order to achieve our goal of a
full featured high level framework.
Briefly, some of the problems found with Phonon were as follows -
- The MediaSource class can not adequately represent the variety of data
sources available. For example; Media sources are often not single items of
interest, but a choice of available content (not substreams of content).
Additionally it is often required to bundle additional data with the playback
content, previews, posters etc, there needs to be a way of uniformly
representing these items and tagging them appropriately
- Some playlist like functionality is bound into the media object and limits
the ability of a media subsystem to participate in playlist management. This
is vitally important for certain classes of playback service.
- Metadata support is bound to the media object, this limits the scope for
collaboration with other media service providers that may have useful metadata.
- The attempt to express a high-level media graph imposes a model of operation
on the underlying frameworks that can not necessarily be supported, the
obvious means of routing around this failure can not be made to work
reliability.
-- A Qt Multimedia API needs to exist in a cross platform (hardware and
software) environment, this means the attempt at graph manipulation imposes an
added customization burden on existing Phonon backends.
- The backend is responsible for creating objects, not allowing applications to
become involved in the media conversation, this places limits on presentation
options.
- Dynamic backend changes
- Services only being available by plugins loaded by Phonon, again limiting the
media conversation.
- Non-local, out of process, playback services are difficult to implement.
QtMobility Multimedia APIs
The Mobility Multimedia APIs are high-level APIs. There is a place for
high-level APIs in a media universe, for although there are a class of
applications that require the ability to manipulate media streams at varying
levels, there is also a class of application that can happily work without
detailed knowledge of the media subsystem or the elements used in the subsystem
in order to do useful work.
The Mobility Media APIs are made to support playback, recording, playlist
management, metadata, radio, camera, and as time permits transcoding, media
editing, TV and potentially high-level stream management.
There is already support for playback, recording, radio, metadata and playlist
management and some experimental camera support.
Like all of the Mobility projects, the QtMobility Multimedia API is targeted to
merge with Qt at some future date, which has not been decided yet.
The QtMobility Multimedia API competes with Phonon on playback services.
There is little sense in having two competing media frameworks in Qt.
Future Direction
There are lots of places we would like to go, Multimedia is an exciting area to
work in. As mentioned above at the higher level we would like to see
transcoding, media editing, and TV support, we would also like to shift down
slightly and open up access to streams, without confusing the "top-level" API.
We would also like to build upon the services layer, provide a more complete
graph orientated audio only framework, as well as low level video capture
service, but only time will tell what gets implemented and when.
Further information
I hope this brings some clarity to your view of multimedia in Qt. If you have
any further questions you can always comment here, catch us on IRC (#qt-labs;
just ping multimedia, someone should pick it up) or send an email to
qt-interest at trolltech.com.
