ATHENA Christian Doppler (CD) Laboratory

The Christian Doppler (CD) Laboratory ATHENA was established in October 2019 to tackle current and future research and deployment challenges of HTTP Adaptive Streaming (HAS) and emerging streaming methods. The goal of CD laboratories is to conduct application-oriented basic research, promote collaboration between universities and companies, and facilitate technology transfer. They are funded through a public-private partnership between companies and the Christian Doppler Research Association, which is funded by the Ministry for Digital and Economic Affairs and the National Foundation for Research, Technology, and Development (Nationalstiftung für Forschung, Technologie und Entwicklung (FTE)). ATHENA is supported by Bitmovin as a company partner.

The CD laboratories have a duration of seven years and undergo rigorous scientific review after two and five years. This spring, the CD lab ATHENA completed its 5-year evaluation, and we have just received official notification from the CDG that we have successfully passed the review. Consequently, it is time to briefly outline the main achievements during this second phase (i.e., years 2 to 5) of the CD lab ATHENA.

Before exploring the achievements, it’s important to highlight the ongoing relevance of research in video streaming, given its dominance in today’s Internet usage. The January 2024 Sandvine Internet Phenomena report revealed that video streaming accounts for 68% of fixed/wired Internet traffic and 64% for mobile Internet traffic. Specifically, Video on Demand (VoD) represents 54% of fixed/wired and 57% of mobile traffic, while live streaming contributes to 14% of fixed/wired and 7% of mobile traffic. The major services in this domain include YouTube and Netflix, each commanding more than 10% of the overall Internet traffic, with TikTok, Amazon Prime, and Disney+ also playing significant roles.

ATHENA is structured into four work packages, each with distinct objectives as detailed below:

1. Content provisioning: Primarily involves video encoding for HAS, quality-aware encoding, learning-based encoding, and multi-codec HAS.
2. Content delivery: Addresses HAS issues by utilizing edge computing, exchanging information between CDN/SDN and clients, providing network assistance for clients, and evaluating corresponding utilities.
3. Content consumption: Focuses on bitrate adaptation schemes, playback improvements, context and user awareness, and studies on Quality of Experience (QoE).
4. End-to-end aspects: Offers a comprehensive view of application and transport layer enhancements, Quality of Experience (QoE) models, low-latency HAS, and learning-based HAS.

During the 2nd phase of ATHENA’s work, we achieved significant results, including publications in respected academic journals and conferences. Specifically, our publications were featured in key multimedia, signal processing, computer networks & wireless communication, and computing systems venues, as categorized by Google Scholar under engineering and computer science. Some of the notable publications include IEEE Communications Surveys & Tutorials (impact factor: 35.6), IEEE Transactions on Image Processing (10.6), IEEE Internet of Things Journal (10.6), IEEE Transactions on Circuits and Systems for Video Technology (8.4), and IEEE Transactions on Multimedia (7.3).

Furthermore, we focused on technology transfer by submitting 16 invention disclosures, resulting in 13 patent applications (including provisionals). Collaborating with our company partner, we obtained 6 granted patents. Additionally, we’re pleased to report on the progress of our spin-off projects, as well as the funding secured for two FFG-funded projects named APOLLO and GAIA, and an EU Horizon Europe-funded innovation action called SPIRIT.

The ATHENA team was also active in organizing scientific events such as workshops, special sessions, and special issues at IEEE ICME, ACM MM, ACM MMSys, ACM CoNEXT, IEEE ICIP, PCS, and IEEE Network. We also contributed to reproducibility in research through open source tools (e.g., Video Complexity Analyzer and LLL-CAdViSE) and datasets (e.g., Video Complexity Dataset and Multi-Codec Ultra High Definition 8K MPEG-DASH Dataset) among others.

We also note our contributions to the applications of AI in video coding & streaming, for example in video coding and video streaming as follows:

• Fast Multi-Rate Encoding with Machine Learning (using Convolutional Neural Networks (CNNs))
• LiDeR: Lightweight video Super Resolution for mobile devices (using Deep Neural Networks (DNNs))
• Blind Visual Quality Assessment Using Vision Transformers
• Video Complexity Analysis (VCA) and optimizations for per-title encoding (using Linear Regression, Random Forest, and XGBoost models)
• DeepStream: Video streaming enhancements using compressed deep neural networks (using Deep Neural Networks (DNNs))
• ECAS-ML: Edge-assisted adaptive bitrate switching (using Long Short-Term Memory (LSTM))
• Quality Optimization of Live Streaming Services over HTTP with Reinforcement Learning (RL)

A major outcome of the second phase is the successful defense of the inaugural cohort of PhD students:

• Dr. Alireza Erfanian: “Optimizing QoE and Latency of Video Streaming using Edge Computing and In-Network Intelligence”, May 25, 2023
• Dr. Ekrem Çetinkaya: “Video Coding Enhancements for HTTP Adaptive Streaming using Machine Learning”, June 7, 2023
• Dr. Minh Nguyen: “Policy-driven Dynamic HTTP Adaptive Streaming Player Environment”, June 30, 2023
• Dr. Jesús Aguilar Armijo: “Multi-access Edge Computing for Adaptive Video Streaming”, July 10, 2023
• Dr. Reza Farahani: “Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, SDN, and MEC”, August 22, 2023
• Dr. Vignesh V Menon: “Content-adaptive Video Coding for HTTP Adaptive Streaming”, January 15, 2024
• Dr. Babak Taraghi, “End-to-end Quality of Experience Evaluation for HTTP Adaptive Streaming”, July 10, 2024

Two postdoctoral scholars have reached a significant milestone on their path toward habilitation

• Dr. Hadi Amirpour, “Video Coding for Efficient HTTP Adaptive Streaming”, February 8, 2024
• Dr. Farzad Tashtarian, “How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions”, February 27, 2023 & “End-to-End Adaptive Video Streaming Optimization”, June 26, 2024

During the second phase, each work package produced excellent publications in their domain, briefly highlighted in the following. Content provisioning (WP-1) focuses mainly on video coding for HAS (43 papers) and immersive media coding for streaming (4 papers). The former can be further subdivided into the following topic areas:

• Video complexity: spatial and temporal feature extraction (4 papers)
• Compression efficiency improvement of individual representations (1 paper)
• Encoding parameter prediction for HAS (9 papers)
• Efficient bitrate ladder construction (4 papers)
• Fast multi-rate encoding (3 papers)
• Data security and data hiding (7 papers)
• Energy-efficient video encoding for HAS (4 papers)
• Advancing video quality evaluation (7 papers)
• Datasets (4 papers)

Content delivery (WP-2) dealt with SDN/CDN assistance for HAS, edge computing support for HAS, and network-embedded media streaming support, resulting in 21 papers. Content consumption (WP-3) worked on QoE enhancement mechanisms at client-side and QoE- and energy-aware content consumption (11 papers). Finally, end-to-end Aspects (WP-4) produced 15 papers in the area of end-to-end QoE improvement in multimedia video streaming. We reported 94 papers published/accepted for the ATHENA 5-year evaluation.

In this context, it is also important to highlight the collaboration within ATHENA, which has resulted in joint publications across various work packages (WPs) and with other ITEC members. For example, collaborations with Prof. Schöffmann (FWF-funded project OVID), FFG-funded projects APOLLO/GAIA, and EU-funded project SPIRIT. In addition, we would like to acknowledge our international collaborators, such as Prof. Hongjie He from Southwest Jiaotong University, Prof. Patrick Le Callet from the University of Nantes, Prof. Wassim Hamidouche from the Technology Innovation Institute (UAE), Dr. Sergey Gorinsky from IMDEA, Dr. Abdelhak Bentaleb from Concordia University, Dr. Raimund Schatz from AIT, and Prof. Pablo Cesar from CWI. We are also pleased to report the successful technology transfers to Bitmovin, particularly CAdViSE (WP-4) and WISH ABR (WP-3). Regular “Fun with ATHENA” meetups and Break-out Groups are utilized for in-depth discussions about innovations and potential technology transfers.

Over the next two years, the ATHENA project will prioritize the development of deep neural network/AI-based image and video coding within the context of HAS. This includes energy- and cost-aware video coding for HAS, immersive video coding such as volumetric video and holography, as well as Quality of Experience (QoE) and energy-aware content consumption for HAS (including energy-efficient, AI-based live video streaming) and generative AI for HAS.

Thanks to all current and former ATHENA team members: Samira Afzal, Hadi Amirpour, Jesús Aguilar Armijo, Emanuele Artioli, Christian Bauer, Alexis Boniface, Ekrem Çetinkaya, Reza Ebrahimi, Alireza Erfanian, Reza Farahani, Mohammad Ghanbari (late), Milad Ghanbari, Mohammad Ghasempour, Selina Zoë Haack, Hermann Hellwagner, Manuel Hoi, Andreas Kogler, Gregor Lammer, Armin Lachini, David Langmeier, Sandro Linder, Daniele Lorenzi, Vignesh V Menon, Minh Nguyen, Engin Orhan, Lingfeng Qu, Jameson Steiner, Nina Stiller, Babak Taraghi, Farzad Tashtarian, Yuan Yuan, and Yiying Wei. Finally, thanks to ITEC support staff Martina Steinbacher, Nina Stiller, Margit Letter, Marion Taschwer, and Rudolf Messner.

We also would like to thank the Christian Doppler Research Association for continuous support, organizing the review, and the reviewer for constructive feedback!

Klagenfurt, July 10, 2024

Congratulations to Dr. Babak Taraghi for successfully defending his dissertation on “End-to-end Quality of Experience Evaluation for HTTP Adaptive Streaming” at Universität Klagenfurt in the context of the Christian Doppler Laboratory ATHENA.

Abstract

HTTP Adaptive Streaming (HAS) has risen to prominent acclaim as the prevailing approach for distributing video content across the Internet. The emergence of popular online streaming platforms, which mainly leverage HAS, has led to a surge in the number of users actively generating and consuming high-quality content. Nonetheless, this remarkable surge presents an intricate puzzle for scholars and service providers, who must contend with varying network conditions and limited network resources to meet user expectations for quality.

In response to these challenges, this dissertation explores the end-to-end evaluation of Quality of Experience (QoE) in the context of HAS. This dissertation investigates evaluation methodologies and frameworks designed to measure QoE and end-to-end latency, particularly in live HAS deployments. We identified the gaps and challenges in current QoE evaluation methodologies through extensive literature reviews and analysis of existing approaches. This thesis proposes novel contributions to address these gaps, encompassing the development of evaluation frameworks, enhancing the understanding of QoE, in-depth studies on QoE impacting factors, and curating a comprehensive dataset.

This dissertation’s first category of contributions is the development of two evaluation frameworks, CAdViSE and LLL-CAdViSE. These frameworks provide researchers and developers with powerful tools to assess the performance and QoE of HAS systems. By harnessing the potential of cloud-based architectures and cuttingedge testing functionalities, these frameworks empower the undertaking of expansive evaluations, incorporating various streaming protocols, codecs, and various network
scenarios. As a result, they contribute significantly to the refinement of streaming systems. Notably, both frameworks are available to the public as open-source projects, marking a noteworthy stride in advancing the field.

As a second category of contributions, we present two extensive studies
investigating the metrics and factors influencing QoE. We investigated the impact of the performance of heuristic-based algorithms on QoE by employing subjective assessment methods and analyzing the influence of algorithmic decisions on user perception. We did an in-depth analysis of stall events and quality switches by conducting subjective assessments and Analysis of Variance (ANOVA) to unveil their influence on QoE. We found that the longer stall events led to greater dissatisfaction.
Further investigation focused on stall event duration and rebuffering’s impact on QoE. Our evaluations revealed that stall events under 4ms went unnoticed by users. Shorter stall durations were generally more tolerable, and improved buffering strategies helped mitigate stall effects on QoE.

In the third contribution category, this thesis fulfills the requirement for contemporary datasets that mirror the latest progress in video technology. A thoroughgoing collection named the ”Multi-codec Ultra High Definition 8K MPEG DASH Dataset” has been meticulously curated. It encompasses a wide array of video content, encoded with cutting-edge codecs like VVC and boasting resolutions up to 8K. This comprehensive dataset forms the bedrock for evaluations across diverse streaming scenarios.

This dissertation advances the field of QoE evaluation for HAS through the development of evaluation frameworks, insightful studies, in-depth analysis, and the presentation of a comprehensive dataset. It provides a ground for researchers and developers to assess and enhance the streaming experience, leading to improved algorithms, optimized systems, and enhanced user satisfaction in HAS.

Slides are available here.

COBIRAS: Offering a Continuous Bit Rate Slide to Maximize DASH Streaming Bandwidth Utilization

ACM Transactions on Multimedia Computing Communications and Applications (ACM TOMM)

[PDF]

Michael Seufert (University of Augsburg, Germany), Marius Spangenberger (University of Würzburg, Germany), Fabian Poignée (University of Würzburg, Germany), Florian Wamser (Lucerne University of Applied Sciences and Arts, Switzerland), Werner Robitza (AVEQ GmbH, Austria), Christian Timmerer (Christian Doppler-Labor ATHENA, Alpen-Adria-Universität, Austria), Tobias Hoßfeld (University of Würzburg, Germany)

Comparison of classical DASH system to proposed DASH system with COBIRAS, JITE at the DASH server, and MinOff at the DASH client.}

Abstract: Reaching close-to-optimal bandwidth utilization in Dynamic Adaptive Streaming over HTTP (DASH) systems can, in theory, be achieved with a small discrete set of bit rate representations. This includes typical bit rate ladders used in state-of-the-art DASH systems. In practice, however, we demonstrate that bandwidth utilization, and consequently the Quality of Experience (QoE), can be improved by offering a continuous set of bit rate representations, i.e., a continuous bit rate slide (COBIRAS). Moreover, we find that the buffer fill behavior of different standard adaptive bit rate (ABR) algorithms is sub-optimal in terms of bandwidth utilization. To overcome this issue, we leverage COBIRAS’ flexibility to request segments with any arbitrary bit rate and propose a novel ABR algorithm MinOff, which helps maximizing bandwidth utilization by minimizing download off-phases during streaming. To avoid extensive storage requirements with COBIRAS and to demonstrate the feasibility of our approach, we design and implement a proof-of-concept DASH system for video streaming that relies on just-in-time encoding (JITE), which reduces storage consumption on the DASH server. Finally, we conduct a performance evaluation on our testbed and compare a state-of-the-art DASH system with few bit rate representations and our JITE DASH system, which can offer a continuous bit rate slide, in terms of bandwidth utilization and video QoE for different ABR algorithms.

Additional Key Words and Phrases: Dynamic Adaptive Streaming over HTTP, DASH, HTTP Adaptive Streaming, HAS, Encoding, Bit Rate Representations, Adaptive Bit Rate, ABR, Bandwidth Utilization, Quality of Experience, QoE

The paper titled “Beyond Curves and Thresholds – Introducing Uncertainty Estimation to Satisfied User Ratios for Compressed Video,” co-authored by Jingwen Zhu, Hadi Amirpour, Raimund Shatz, Patrick Le Callet, and Christian Timmerer, received the Best Paper Award at the 37th Picture Coding Symposium.

This special issue aims to collect extended versions of the accepted papers at ACM Multimedia Systems 2024 and co-located workshops (i.e., NOSSDAV, MMVE, and GMSys). Similarly, as for 2023, it is planned that all accepted MMSys full research papers and workshop papers are eligible for submission, which must have at least 25% new material compared to the accepted paper at MMSys or co-located workshops, respectively.

The ACM Multimedia Systems Conference and associated workshops seek to bring together experts from academia and industry to share their latest research findings in the field of multimedia systems. While research about specific aspects of multimedia systems is regularly published in various venues covering networking, operating systems, real-time systems, databases, mobile computing, distributed systems, computer vision, and middleware communities, MMSys aims to cut across these domains in the context of multimedia data types. This provides a unique opportunity to investigate the intersections and the interplay of the various approaches and solutions developed across these domains.
Topics Submissions are solicited on all aspects of multimedia systems, including but not limited to:

• Content generation, adaptation, and summarization
• Adaptive streaming of multimedia content
• AI (e.g., machine/deep learning) for all aspects of multimedia systems
• Network and system support for multimedia
• Video games and cloud gaming
• Virtual and augmented reality content and systems
• Multiview, 360 degrees, 3D, and volumetric videos
• Internet of Things (IoTs) and multimedia
• Mobile multimedia and 5G/6G
• Wearable multimedia
• Cloud and edge computing for multimedia systems
• Digital twins
• Cyber-physical systems
• Multi-sensory experiences
• Autonomous multimedia systems
• Quality of Experience (QoE)
• Multimedia systems for robotics and unmanned vehicles
• Multimedia systems for health
• Audio, image and video coding for humans and machines
• Analytics for multimedia systems
• Sustainable (green) multimedia systems

Important Dates

• Open for submissions: July 15, 2024
• Submission deadline: September 15, 2024
• First-round review decisions: November 15, 202
• Deadline for revision submissions: January 15, 2025
• Notification of final decisions: March 15, 2025
• Tentative publication: April 2025

Submission Information

Prospective authors are invited to submit their manuscripts electronically adhering to the ACM TOMM journal guidelines (see https://tomm.acm.org/authors.cfm). The manuscript will not be entertained if guidelines are not followed. The manuscript should be within the scope of ACM TOMM. Please submit your papers through the online system (https://mc.manuscriptcentral.com/tomm) and be sure to select the special issue. Manuscripts should not be published or currently submitted for publication elsewhere.

Guest Editors

• Christian Timmerer, University of Klagenfurt, Austria, christian.timmerer@aau.at
• Maria Martini, Kingston University, M.Martini@kingston.ac.uk
• Ali C. Begen, Ozyegin University, Türkiye, ali.begen@ozyegin.edu.tr
• Lucca De Cicco, Politecnico di Bari, Italy, luca.decicco@poliba.it

For questions and further information, please contact guest editors using acm-tomm-si-msys2024@itec.aau.at.

HTTP Adaptive Streaming – Quo Vadis?

Jun 27, 2024, 04:00 PM Dubai

[Slides]

Abstract: Video traffic on the Internet is constantly growing; networked multimedia applications consume a predominant share of the available Internet bandwidth. A major technical breakthrough and enabler in multimedia systems research and of industrial networked multimedia services certainly was the HTTP Adaptive Streaming (HAS) technique. This resulted in the standardization of MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) which, together with HTTP Live Streaming (HLS), is widely used for multimedia delivery in today’s networks. Existing challenges in multimedia systems research deal with the trade-off between (i) the ever-increasing content complexity, (ii) various requirements with respect to time (most importantly, latency), and (iii) quality of experience (QoE). Optimizing towards one aspect usually negatively impacts at least one of the other two aspects if not both. This situation sets the stage for our research work in the ATHENA Christian Doppler (CD) Laboratory (Adaptive Streaming over HTTP and Emerging Networked Multimedia Services; https://athena.itec.aau.at/), jointly funded by public sources and industry. In this talk, we will present selected novel approaches and research results of the first year of the ATHENA CD Lab’s operation. We will highlight HAS-related research on (i) multimedia content provisioning (machine learning for video encoding); (ii) multimedia content delivery (support of edge processing and virtualized network functions for video networking); (iii) multimedia content consumption and end-to-end aspects (player-triggered segment retransmissions to improve video playout quality); and (iv) novel QoE investigations (adaptive point cloud streaming). We will also put the work into the context of international multimedia systems research.

Biography: Christian Timmerer is a full professor of computer science at Alpen-Adria-Universität Klagenfurt (AAU), Institute of Information Technology (ITEC) and he is the director of the Christian Doppler (CD) Laboratory ATHENA (https://athena.itec.aau.at/). His research interests include multimedia systems, immersive multimedia communication, streaming, adaptation, and quality of experience where he co-authored more than 20 patent applications and more than 300 articles. He was the general chair of WIAMIS 2008, QoMEX 2013, MMSys 2016, and PV 2018 and has participated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, COST IC1003 QUALINET, ICoSOLE, and SPIRIT. He also participated in ISO/MPEG work for several years, notably in the area of MPEG-21, MPEG-M, MPEG-V, and MPEG-DASH where he also served as standard editor. In 2012 he cofounded Bitmovin (http://www.bitmovin.com/) to provide professional services around MPEG-DASH where he holds the position of the Chief Innovation Officer (CIO) –- Head of Research and Standardization. Further information at http://timmerer.com.

Per-Title Encoding Using Spatial and Temporal Resolution Downscaling

US Patent

[PDF]

Hadi Amirpour (Alpen-Adria-Universität Klagenfurt, Austria) and Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Austria)

Abstract: Techniques relating to per-title encoding using spatial and temporal resolution downscaling is disclosed. A method for per-title encoding includes receiving a video input comprised of video segments, spatially downscaling the video input, temporally downscaling the video input, encoding the video input to generate an encoded video, then temporally and spatially upscaling the encoded video. Spatially downscaling may include reducing a resolution of the video input, and temporally downscaling may include reducing a framerate of the video input. Objective metrics for the upscaled encoded video show improved quality over conventional methods.