ATHENA Christian Doppler (CD) Laboratory

Low-Latency Online Per-Title Encoding

US Patent

[PDF]

Vignesh Menon (Alpen-Adria-Universität Klagenfurt, Austria), Hadi Amirpour (Alpen-Adria-Universität Klagenfurt, Austria), and Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Austria)

Abstract: The technology described herein relates to online per-title encoding. A method for online per-title encoding includes receiving a video input, generating segments of the video input, extracting a spatial feature and a temporal feature, predicting bitrate-resolution pairs based on the spatial feature and the temporal feature, using a discrete cosine transform (DCT)-based energy function, and per-title encoding segments of the video input for the predicted bitrate-resolution pairs. A system for online per-title encoding may include memory for storing a set of bitrates, a set of resolutions, and a machine learning module configured to predict bitrate resolution pairs based on low-complexity spatial and temporal features.

IEEE International Conference on Image Processing (IEEE ICIP)

Grand Challenge on

27-30 October 2024, Abu Dhabi, UAE

https://cd-athena.github.io/GCVC

Organizers:

• Ioannis Katsavounidis (Meta, USA)
• Hadi Amirpour (AAU, Austria)
• Ali Ak (Nantes Univ., France)
• Anil Kokaram (TCD, Ireland)
• Christian Timmerer (AAU, Austria)

Abstract: Video compression standards rely heavily on eliminating spatial and temporal redundancy within and across video frames. Intra-frame encoding targets redundancy within blocks of a single video frame, whereas inter-frame coding focuses on removing redundancy between the current frame and its reference frames. The level of spatial and temporal redundancy, or complexity, is a crucial factor in video compression. Generally, videos with higher complexity require a greater bitrate to maintain a specific quality level. Understanding the complexity of a video beforehand can significantly enhance the optimization of video coding and streaming workflows. While Spatial Information (SI) and Temporal Information (TI) are traditionally used to represent video complexity, they often exhibit low correlation with actual video coding performance. In this challenge, the goal is to find innovative methods that can quickly and accurately predict the spatial and temporal complexity of a video, with a high correlation to actual performance. These methods should be efficient enough to be applicable in live video streaming scenarios, ensuring real-time adaptability and optimization.

Picture Coding Symposium (PCS) 

12-14 June 2024, Taichung, Taiwan

[PDF]

Jingwen Zhu (University of Nantes, France), Hadi Amirpour (AAU, Austria), Raimund Schatz (AIT, Austria), Patrick Le Callet (University of Nantes, France), and Christian Timmerer (AAU, Austria)

Abstract: Just Noticeable Difference (JND) establishes the threshold between two images or videos wherein differences in quality remain imperceptible to an individual. This threshold, collectively known as the Satisfied User Ratio (SUR), holds significant importance in image and video compression applications, ensuring that differences in quality are imperceptible to the majority (p%) of users, known as p%SUR. While substantial efforts have been dedicated to predicting the p%SUR for various encoding parameters (e.g., QP) and quality metrics (e.g., VMAF), referred to as proxies, systematic consideration of the prediction uncertainties associated with these proxies has hitherto remained unexplored. In this paper, we analyze the uncertainty of p%SUR through Confidence Interval (CI) estimation and assess the consistency of various Video Quality Metrics (VQMs) as proxies for SUR. The analysis reveals challenges in directly using p%SUR as ground truth for training models and highlights the need for uncertainty estimation for SUR with different proxies.

IEEE International Conference on Image Processing (IEEE ICIP)

Grand Challenge on

27-30 October 2024, Abi Dhabi, UAE

https://www.icip24-video360sr.ae/home

Abstract: Omnidirectional visual content, commonly referred to as 360-degree images and videos, has garnered significant interest in both academia and industry, establishing itself as the primary media modality for VR/XR applications. 360-degree videos offer numerous features and advantages, allowing users to view scenes in all directions, providing an immersive quality of experience with up to 3 degrees of freedom (3DoF). When integrated on embedded devices with remote control, 360-degree videos offer additional degrees of freedom, enabling movement within the space (6DoF). However, 360-degree videos come with specific requirements, such as high-resolution content with up to 16K video resolution to ensure a high-quality representation of the scene. Moreover, limited bandwidth in wireless communication, especially under mobility conditions, imposes strict constraints on the available throughput to prevent packet loss and maintain low end-to-end latency. Adaptive resolution and efficient compression of 360-degree video content can address these challenges by adapting to the available throughput while maintaining high video quality at the decoder. Nevertheless, the downscaling and coding of the original content before transmission introduces visible distortions and loss of information details that cannot be recovered at the decoder side. In this context, machine learning techniques have demonstrated outstanding performance in alleviating coding artifacts and recovering lost details, particularly for 2D video. Compared to 2D video, 360-degree video presents a lower angular resolution issue, requiring augmentation of both the resolution and the quality of the video. This challenge presents an opportunity for the scientific research and industrial community to propose solutions for quality enhancement and super-resolution for 360-degree videos.

In the wake of the passing of Prof. Mohammad Ghanbari, we extend our deepest condolences to his family during this challenging time. Prof. Ghanbari was a distinguished member of our Christian Doppler Laboratory ATHENA since its inception, and we consider ourselves privileged to have had the opportunity to collaborate with him. His contributions, reflected in over 30 joint publications in video coding and streaming, have been accepted at renowned publication venues such as IEEE TCSVT, ACM TOMM, IEEE TIP, IEEE TNSM, IEEE ICIP, IEEE ICASSP, IEEE ICME, ACM MMSys, PCS, IEEE MMSP, among others. Prof. Ghanbari played a pivotal role in the success of our research endeavors, and his profound knowledge, insightful input, and invaluable guidance were consistently valued.

The entire Institute for Information Technology, especially those at the Christian Doppler Laboratory ATHENA, feels deeply saddened by the loss of Prof. Ghanbari. As we come to terms with this period of mourning, reflection, and farewell, we extend our warmest wishes and heartfelt sympathies to his family and the wider research community.

The 15th ACM Multimedia Systems Conference

15-18 April, 2024 | Bari, Italy

Conference website

[PDF]

Emanuele Artioli (AAU, Austria), Farzad Tashtarian (AAU, Austria), and Christian Timmerer (AAU, Austria)

Abstract:

As the popularity of video streaming entertainment continues to grow, understanding how users engage with the content and react to its changes becomes a critical success factor for every stakeholder. User engagement, i.e., the percentage of video the user watches before quitting, is central to customer loyalty, content personalization, ad relevance, and A/B testing. This paper presents DIGITWISE, a digital twin-based approach for modeling adaptive video streaming engagement. Traditional adaptive bitrate (ABR) algorithms assume that all users react similarly to video streaming artifacts and network issues, neglecting individual user sensitivities. DIGITWISE leverages the concept of a digital twin, a digital replica of a physical entity, to model user engagement based on past viewing sessions. The digital twin receives input about streaming events and utilizes supervised machine learning to predict user engagement for a given session. The system model consists of a data processing pipeline, machine learning models acting as digital twins, and a unified model to predict engagement. DIGITWISE employs the XGBoost model in both digital twins and unified models. The proposed architecture demonstrates the importance of personal user sensitivities, reducing user engagement prediction error by up to 5.8% compared to non-user-aware models. Furthermore, DIGITWISE can optimize content provisioning and delivery by identifying the features that maximize engagement, providing an average engagement increase of up to 8.6 %.

Keywords: digital twin, user engagement, xgboost