Graphics are an essential element of UI applications. Indeed, what differentiates UI applications from other types of software is the presence of a visual interface enhancing user interaction. Visuals make software more intuitive and immediate, easier to use and to understand. Colors, icons, and animations provide visual cues or status updates in a non-invasive way. Charts, graphs, or maps help display complex information and data. Advanced 3D graphics represent real-world objects and life-like situations on the UI. And in addition to their utility, there is an undisputed aesthetic value in the creation of visually appealing software—good UX design and pleasant visuals attract more users and reinforce brands.
Real-time graphics
In UI applications, the images on the screen dynamically change based on user interaction or live data flow. Real-time animations give feedback to the user in a series of subtle ways that contribute to making the user experience engaging and natural. A button expanding while hovering over it, popping up when pressed may pass almost as unnoticed but is an important cue of the device's responsivity to the user’s action. Real-time graphics enable interactive visualizations of data, both in the form of 2D charts and graphs and in that of complex 3D objects.
These types of visuals are substantially different from the computer graphics seen in films and video games (Computer Generated Imagery – CGI) for the creation of characters, environments, and effects, where the level of interactivity is null or very limited and visual elements are rendered once and for all. On embedded devices, UI applications rely on real-time rendering to provide a dynamic and interactive user experience.
Data binding
At the core of the possibility for the UI application to dynamically and interactively represent the world, there is the ability to link and sync visual properties of the UI to real objects and data. This connection is pervasive in UI applications and defines their ultimate meaning: the fuel level displayed on the car’s dashboard represents the real car’s fuel level, the heart rate chart on a smartwatch shows that of the user, and so forth. Advanced data-binding mechanisms are a core UI framework’s tool enabling such connection between UI visuals and real-world data by simple drag-and-drop and automatic sync.
Given the level of interactivity and the life-like situation that UI apps aim to model, the UI framework should provide rich 2D and 3D tooling for the creation of various types of visual content, ranging from basic layout design to complex 3D scenes in real time.
The interplay of 2D and 3D graphics
While the UI layout is typically created by means of 2D graphics tools enabling the easy setup of ground elements, like frames, buttons, icons, charts, or text, there is a growing trend across various industries to increase photorealism by including 3D elements within the UI. Flat 2D elements are typically used to represent controls and other types of abstract or symbolic information, while 3D objects represent real-world entities and even complex situations.
The two types of visuals play complementary roles and provide together the right balance between performance and realism. Where 3D graphics add an extra dimension of expressivity, 2D elements are simpler to create and use and have a lower footprint. With technological advances, the interplay of 2D and 3D graphics has reached such a level of quality and fluidity that complex situations can be represented in the 3D space and seamlessly accessed by means of simple 2D controls. For these two types of graphics to play well together, the UI framework should ensure perfect synchronization between the 2D and 3D elements. By enabling the same features and usability on both 2D and 3D content, the UI framework ensures broader functionality and fluid performance in its applications.
Immersive 3D graphics
With 3D graphics, complex life-like situations can be quickly and easily grasped at a glance. As a well-known example, advanced driving assistance systems (ADAS) in automotive displays increasingly include the real-time rendering of the car in its actual driving environment. To provide drivers with a clear, blind-spot-free view of the surrounding of the car, a 360° camera feed from the physical car creates the 3D scene where the rendered car lives. Real-time reflections, shadows, and other effects contribute to making the resulting 3D scene on the car’s dashboard undistinguishable from the physical car’s situation.
Similar applications that take advantage of real-time 3D graphics for the creation of avatars or digital twins, often within the space of virtual or augmented reality, are being developed in every industry. In the medical sector, 3D graphics provide interactive visualizations of organs and tissues that can be used to aid in diagnosis, surgical planning, and medical research. Here, diagnostic imagining essentially relies on hyper-realistic 3D visuals to identify the causes of an illness or to confirm a diagnosis.
More in general and across industries, highly detailed and articulated 3D representations of the world increase safety, improve efficiency, reduce costs, and enhance the overall quality of the product or service.
Clearly, the development of such visually rich UI applications requires powerful design tools for 2D and 3D graphics and rendering engines capable of displaying with high fidelity and in real time. This is why, in terms of graphics capabilities, the UI framework includes many features typical of the most advanced graphics engines.
HDR lighting
High Dynamic Range lighting allows the creation of photorealistic, life-like 3D environments that are highly detailed, vivid, and accurate. By capturing and displaying a wider range of brightness and colors than traditional pictures, HDR images provide a basis for more accurate lighting and shading calculations in the 3D scene, resulting in more realistic and dynamic lighting effects.
Physically based rendering
By including a physically based rendering (PBR) pipeline, the UI framework ensures a more accurate and realistic rendering of materials and light and simplifies the creation of high-quality graphics. By adhering to such a standard, as specified for instance in the glTF format, UI designers readily gain compatibility with general-purpose material models—like wood, leather, rubber, metal, etc. PBR allows for more consistent and predictable results across different lighting conditions and materials resulting in a reduced need for technical artists to create custom shaders for different surface types.
As the PBR standard is increasingly popular in the 3D design community, designers also readily get access to thousands of 3D models available online that can be used and customized according to the UI application’s needs.
Postprocessing effects
Postprocessing effects apply before each frame is rendered by the actual graphics hardware to improve its visual quality or tweak the overall look and feel of the scene with little setup time. As the quality standard is largely set by the CGI offering—which is not real-time though—the UI framework’s graphics subsystem needs to include advanced algorithms to eliminate any glitches that can reduce photorealism. Anti-aliasing is a good example: it removes from the rendering the jagged edges that may appear when a coarse model geometry is used.
Theming
Control over the theme and overall appearance of the UI is fundamental for ensuring a consistent look and feel of the UI across applications and platforms. Theming can include changes in colors, fonts, and icons, but also the overall restructuring of the UI by adding or removing widgets for better adaptation to different form factors. Such operations are essential to ensure, for instance, consistent branding and essential functionality across a varied product portfolio.
Embedding game-engine graphics
A state-of-the-art UI framework typically supports different types of graphics APIs for the creation of advanced 3D visuals, such as OpenGL, OpenGL ES, Vulkan, Direct3D, and Metal. In addition, it can easily leverage content produced by game engines by either embedding such content within a UI window or by acting as middleware for the integration of the game engine’s advanced visual and physics features with standard UI services like input management, networking, media, and even certified safe-rendering solutions that may be otherwise lacking.
Integrating the game engine’s graphics within the UI framework may be beneficial at various levels when graphics quality, performance, safety, and cross-platform capabilities are at stake. Thanks to its optimized resource management system, the UI framework enables porting game-like experiences within resource-constrained devices and their certification for safety-critical applications.
Instead of rendering visual content constantly at the highest frame rate—which is the standard for standalone gaming on PCs—the UI engine’s event-driven architecture minimizes the CPU, GPU, and memory needed for each process. In other words, where the game engine demands unconstrained resource usage for high-quality rendering and smooth gameplay of a single application, the UI engine’s on-demand behavior ensures that resources are used sparingly and dynamically allocated across multiple processes based on real-time needs. While keeping the number of UI elements being loaded, drawn, and held in memory minimal, the platform-agnostic nature of the UI engine also enables to scale the same user experience across hardware types, with all the benefits that have been discussed. Among these, not least, delivering compelling visuals also on low-end hardware as a cost-effective solution for high-quality software.
After discussing the UI framework foundations, performance, and graphics, the next blog will illustrate how the various UI framework components play together to standardize the delivery and maintenance lifecycle across diverse product lines while efficiently achieving the required degree of differentiation needed to deliver true signature UIs across global markets.
