Energy-Quality-aware Variable Framerate Pareto-Front for Adaptive Video Streaming

Energy-Quality-aware Variable Framerate Pareto-Front for Adaptive Video Streaming

IEEE Visual Communications and Image Processing (IEEE VCIP 2024)

Tokyo, Japan, December 8-11, 2024

[PDF]

Prajit T Rajendran (Universite Paris-Saclay), Samira Afzal (Alpen-Adria-Universität Klagenfurt), Vignesh V Menon (Fraunhofer HHI), Christian Timmerer (Alpen-Adria-Universität Klagenfurt)

Abstract: Optimizing framerate for a given bitrate-spatial resolution pair in adaptive video streaming is essential to maintain perceptual quality while considering decoding complexity. Low framerates at low bitrates reduce compression artifacts and decrease decoding energy. We propose a novel method, Decoding-complexity aware Framerate Prediction (DECODRA), which employs a Variable Framerate Pareto-front approach to predict an optimized framerate that minimizes decoding energy under quality degradation constraints. DECODRA dynamically adjusts the framerate based on current bitrate and spatial resolution, balancing trade-offs between framerate, perceptual quality, and decoding complexity. Extensive experimentation with the Inter-4K dataset demonstrates DECODRA’s effectiveness, yielding an average PSNR and VMAF increase of 0.87 dB and 5.14 points, respectively, for the same bitrate compared to the default 60 fps encoding. Additionally, DECODRA achieves an average reduction in decoding energy consumption of 13.27 %, enhancing the viewing experience, extending mobile device battery life, and reducing the energy footprint of streaming services.

Index Terms—Adaptive video streaming; Decoding complexity; Framerate optimization; Perceptual quality; Decoding energy consumption

Posted in ATHENA | Comments Off on Energy-Quality-aware Variable Framerate Pareto-Front for Adaptive Video Streaming

ACM Mile-High Video Conference 2025: Call for Contributions

MHV 2025: ACM Mile-High Video Conference 2025
Call for Contributions
February 18-20, 2025, The Cable Center, Denver, Colorado
https://www.mile-high.video/
Posted in ATHENA, News | Comments Off on ACM Mile-High Video Conference 2025: Call for Contributions

University assistant predoctoral (all genders welcome) (in German: Universitätsassistent:in)

The University of Klagenfurt, with approximately 1,500 employees and over 12,000 students, is located in the Alps-Adriatic region and consistently achieves excellent placements in rankings. The motto “per aspera ad astra” underscores our firm commitment to the pursuit of excellence in all research, teaching, and university management activities. The principles of equality, diversity, health, sustainability, and compatibility of work and family life serve as the foundation for our work at the university.

The University of Klagenfurt is pleased to announce the following open position at the Department of Information Technology at the Faculty of Technical Sciences with an expected starting date of November 4, 2024:

University assistant predoctoral (all genders welcome) (in German: Universitätsassistent:in)

within the Ada Lovelace Programme (project title: Streaming of Holographic Content and its Impact on the Quality of Experience).

  • Level of employment: 100 % (40 hours/week)
  • Minimum salary: € 50,103.20 per annum (gross); Classification according to collective agreement: B1
  • Contract duration: 4 years
  • Application deadline: by September 11, 2024
  • Reference code: 348/24

Tasks and responsibilities:

  • Autonomous scientific work, including the publication of research articles in the fields of coding and streaming of holographic content, Quality of Experience (QoE), and behavioural sciences
  • Conducting independent scientific research with the aim of submitting a dissertation and acquiring a doctoral degree in technical sciences
  • Teaching exercises and lab courses (e.g., in the computer science Bachelor’s or/and Master’s programme)
  • Participating in research projects of the department, especially within the Ada Lovelace Programme (Streaming of Holographic Content and its Impact on the Quality of Experience)
  • Mentoring students
  • Assisting in public relations activities, science to public communication, and extra-curricular events of the department and the faculty

Prerequisites for the appointment:

  • Completed Diploma or Master’s degree from a recognized university in the field of computer science, information and communications engineering, electrical engineering, or related fields. The completion of this degree must be fulfilled no later than two weeks before the starting date; hence, the last possible deadline for meeting this requirement is October 20, 2024
  • Strong background in one or more of the following fields: multimedia systems (i.e., video/holographic content coding/streaming, Quality of Experience) and empirical research methods (i.e., statistical methods, interdisciplinary research with behavioural sciences)
  • Fluent in written and spoken English
  • Programming experience in multimedia systems

Additional desired qualifications:

  • Experience with scientific publications or presentations
  • Experience in interdisciplinary research projects, ideally in the behavioural sciences, as the project involves empirical research
  • Excellent ability to work with teams
  • Scientific curiosity and enthusiasm for research in multimedia systems and empirical research

The doctoral student will be co-supervised by Christian Timmerer, Heather Foran, and Hadi Amirpour.

Our offer:

This position serves the purposes of the vocational and scientific education of graduates of Master’s or Diploma degree programmes and sets the goal of completing a Doctoral degree / a Ph.D. in Technical Sciences. Therefore, applications by persons who have already completed a subject-specific doctoral degree or a subject-relevant Ph.D. program cannot be considered.

The employment contract is concluded for the position of university assistant (predoctoral) and stipulates a starting salary of € 3,578.80 gross per month (14 times a year; previous experience deemed relevant to the job can be recognized in accordance with the collective agreement).

The University of Klagenfurt also offers:

  • Personal and professional advanced training courses, management, and career coaching
  • Numerous attractive additional benefits, see also https://jobs.aau.at/en/the-university-as-employer/
  • Diversity- and family-friendly university culture
  • The opportunity to live and work in the attractive Alps-Adriatic region with a wide range of leisure activities in the spheres of culture, nature, and sports

The application:

If you are interested in this position, please apply in English by providing the following documents:

  • Letter of application/cover letter including motivation statement for the given position
  • Curriculum vitae (with clear information about the degrees, including date/place/grade, the experience acquired, the thesis title, the list of publications (if any), and any other relevant information)
  • Copy of the degree certificates and transcripts of the courses
  • Any certificates that can prove the fulfilment of the required and additional qualifications listed above (e.g., the submission of the final thesis if required by the degree programme, copy of publications, programming skills certificates, language skills certificates, etc.)
  • Final thesis or other study-related written work (like seminar reports) or excerpts thereof
  • If an applicant has not received the Diploma or Master’s degree by the application deadline, the applicant should provide a declaration, written either by a supervisor or by the candidate themselves, on the feasibility of finishing the Diploma or Master’s degree by October 30, 2024 at the latest.

To apply, please select the position with the reference code 348/24 in the category “Scientific Staff” using the link “Apply for this position” in the job portal at jobs.aau.at/en/.

Candidates must furnish proof that they meet the required qualifications by October 20, 2024 at the latest.

For further information on this specific vacancy, please contact Univ.-Prof. DI Dr. Christian Timmerer (christian.timmerer@aau.at). General information about the university as an employer can be found at https://jobs.aau.at/en/the-university-as-employer/. At the University of Klagenfurt, recruitment and staff matters are accompanied not only by the authority responsible for the recruitment procedure but also by the Equal Opportunities Working Group and, if necessary, by the Representative for Disabled Persons.

The University of Klagenfurt aims to increase the proportion of women and, therefore, invites explicitly qualified women to apply for the position. Where the qualification is equivalent, women will be given preferential consideration.

People with disabilities or chronic diseases, who fulfill the requirements, are particularly encouraged to apply.

Travel and accommodation costs incurred during the application process will not be refunded. Translations into other languages shall serve informational purposes only. Solely the version advertised in the University Bulletin (Mitteilungsblatt) shall be legally binding.

Posted in ATHENA, News | Comments Off on University assistant predoctoral (all genders welcome) (in German: Universitätsassistent:in)

Adaptivity in Video Streaming through the Transition Lens

Adaptivity in Video Streaming through the Transition Lens

In: Schulte, S., Koldehofe, B. (eds) From Multimedia Communications to the Future Internet. Lecture Notes in Computer Science, vol 15200. Springer, Cham.

[PDF]

Amr Rizk (Leibniz Universität Hannover, Germany), Hermann Hellwagner (AAU, Austria), Christian Timmerer (AAU, Austria), and Michael Zink (University of Massachusetts Amherst, MA, USA)

Abstract: Adaptivity is a cornerstone concept in video streaming. Equipped with the concept of Transitions, we review in this paper adaptivity mechanisms known from classical video streaming scenarios. We specifically highlight how these mechanisms emerge in a specific context, such that their performance finally depends on the deployment conditions. Using multiple examples we highlight the strength of the concept of adaptivity at runtime for video streaming.

Posted in ATHENA, News | Comments Off on Adaptivity in Video Streaming through the Transition Lens

Interns at ATHENA (Summer 2024)

 

Between July and August 2024, the ATHENA Christian Doppler Laboratory hosted four interns working on the following topics:

  • Halime Lezi: Image and Video Compression Pipeline
  • Luka Kaiser: VidStream
  • Julius van Dillen: Enhancing Video Quality with Super-Resolution

At the conclusion of their internships, they presented their work and results, receiving official certificates from the university. This collaboration was mutually beneficial for both the researchers at ATHENA and the interns. Their learning process was enhanced by the dedicated guidance they received, which included personalized mentorship, hands-on training, and continuous support. This comprehensive supervision ensured that they not only developed practical skills but also gained a deeper understanding of the research methodologies and technologies used in the video streaming field. We extend our gratitude to all three interns for their genuine interest, productive efforts, and valuable feedback on the laboratory.

Halime Lezi: I had an awesome time during my four-week internship at ATHENA. My project was about image and video compression, and I learned a lot about how it works. I also got to use Python, which was both fun and challenging. The work environment at ATHENA was really supportive and interesting. My supervisor, Emanuele Artioli, was super helpful and always ready to answer my questions. He made sure I understood both the practical and theoretical parts of my work, which was really cool. It was also great to work with people from different countries. The team was friendly, and we got along well. The work-life balance was good, with a nice mix of work and relaxation. Overall, my time at ATHENA was very educational and enjoyable. The skills and knowledge I gained during this internship will be really useful for my future studies and career. I’m thankful for the opportunity and the support I received. I would highly recommend this internship to anyone looking for a rewarding and fulfilling experience. It’s a great place to learn, grow, and meet new people.

Luka Kaiser: I had an amazing four weeks at ATHENA. It was really nice meeting new colleagues and even making new friends. The project I worked on was exactly what interests me, and if I had any questions, my supervisor, Christian, was always there to help me out. So, thank you very much for that. Overall, this experience has been incredibly valuable, and I learned a lot and gained practical skills that I will definitely use in the future. My time here was both productive and enjoyable. I am grateful for the opportunity and would love to stay connected with everyone I’ve met. Thank you once again for everything.

Julius van Dillen: My four-week internship at ATHENA was incredible. I focused on improving image and video quality through Super-Resolution techniques. I had the opportunity to work with a variety of tools and technologies, including FFmpeg, Visual Studio Code, Python, several Super-Resolution architectures, and various video quality metrics. I was truly surprised by how much Super-Resolution enhances video and image quality. I really enjoyed working with these technologies and found the entire process fascinating. I am grateful to have had Daniele as my supervisor; his guidance and support made the experience both easier and more enjoyable. During my internship, I gained valuable insights into the research process and the fundamentals of Python programming.

Posted in ATHENA | Comments Off on Interns at ATHENA (Summer 2024)

Energy-Efficient Spatial and Temporal Resolution Selection for Per-Title Encoding

Energy-Efficient Spatial and Temporal Resolution Selection for Per-Title Encoding

IEEE Access

[PDF]

Mohammad Ghasempour (AAU, Austria), Hadi Amirpour (AAU, Austria), and Christian Timmerer (AAU, Austria)

Abstract: Video streaming has become an integral part of our digital lives, driving the need for efficient video delivery. With the growing demand for seamless video delivery, adaptive video streaming has emerged as a solution to support users with varying device capabilities and network conditions. Traditional adaptive streaming relies on a predetermined set of bitrate-resolution pairs, known as bitrate ladders, for encoding. However, this “one-size-fits-all” approach is suboptimal when dealing with diverse video content. Consequently, per-title encoding approaches dynamically select the bitrate ladder for each content. However, in an era when carbon dioxide emissions have become a paramount concern, it is crucial to consider energy consumption. Therefore, this paper addresses the pressing issue of increasing energy consumption in video streaming by introducing a novel approach, ESTR, which goes beyond traditional quality-centric resolution selection approaches. Instead, the ESTR considers both video quality and decoding energy consumption to construct an optimal bitrate ladder tailored to the unique characteristics of each video content. To accomplish this, ESTR encodes each video content using a range of spatial and temporal resolutions, each paired with specific bitrates. It then establishes a maximum acceptable quality drop threshold (τ), carefully selecting resolutions that not only preserve video quality above this threshold but also minimize decoding energy consumption. Our experimental results, at a fixed τ of 2 VMAF steps, demonstrate a 32.87% to 41.86% reduction in decoding energy demand for HEVC-encoded videos across various software decoder implementations and operating systems, with a maximum bitrate increase of 2.52%. Furthermore, on a hardware-accelerated client device, a 46.37% energy saving was achieved during video playback at the expense of a 2.52% bitrate increase. Remarkably, these gains in energy efficiency are achieved while maintaining consistent video quality.

Posted in ATHENA | Comments Off on Energy-Efficient Spatial and Temporal Resolution Selection for Per-Title Encoding

Successful 5-year Evaluation of Christian Doppler Laboratory ATHENA

The Christian Doppler (CD) Laboratory ATHENA was established in October 2019 to tackle current and future research and deployment challenges of HTTP Adaptive Streaming (HAS) and emerging streaming methods. The goal of CD laboratories is to conduct application-oriented basic research, promote collaboration between universities and companies, and facilitate technology transfer. They are funded through a public-private partnership between companies and the Christian Doppler Research Association, which is funded by the Ministry for Digital and Economic Affairs and the National Foundation for Research, Technology, and Development (Nationalstiftung für Forschung, Technologie und Entwicklung (FTE)). ATHENA is supported by Bitmovin as a company partner.

The CD laboratories have a duration of seven years and undergo rigorous scientific review after two and five years. This spring, the CD lab ATHENA completed its 5-year evaluation, and we have just received official notification from the CDG that we have successfully passed the review. Consequently, it is time to briefly outline the main achievements during this second phase (i.e., years 2 to 5) of the CD lab ATHENA.

Before exploring the achievements, it’s important to highlight the ongoing relevance of research in video streaming, given its dominance in today’s Internet usage. The January 2024 Sandvine Internet Phenomena report revealed that video streaming accounts for 68% of fixed/wired Internet traffic and 64% for mobile Internet traffic. Specifically, Video on Demand (VoD) represents 54% of fixed/wired and 57% of mobile traffic, while live streaming contributes to 14% of fixed/wired and 7% of mobile traffic. The major services in this domain include YouTube and Netflix, each commanding more than 10% of the overall Internet traffic, with TikTok, Amazon Prime, and Disney+ also playing significant roles.

ATHENA is structured into four work packages, each with distinct objectives as detailed below:

  1. Content provisioning: Primarily involves video encoding for HAS, quality-aware encoding, learning-based encoding, and multi-codec HAS.
  2. Content delivery: Addresses HAS issues by utilizing edge computing, exchanging information between CDN/SDN and clients, providing network assistance for clients, and evaluating corresponding utilities.
  3. Content consumption: Focuses on bitrate adaptation schemes, playback improvements, context and user awareness, and studies on Quality of Experience (QoE).
  4. End-to-end aspects: Offers a comprehensive view of application and transport layer enhancements, Quality of Experience (QoE) models, low-latency HAS, and learning-based HAS.

During the 2nd phase of ATHENA’s work, we achieved significant results, including publications in respected academic journals and conferences. Specifically, our publications were featured in key multimedia, signal processing, computer networks & wireless communication, and computing systems venues, as categorized by Google Scholar under engineering and computer science. Some of the notable publications include IEEE Communications Surveys & Tutorials (impact factor: 35.6), IEEE Transactions on Image Processing (10.6), IEEE Internet of Things Journal (10.6), IEEE Transactions on Circuits and Systems for Video Technology (8.4), and IEEE Transactions on Multimedia (7.3).

Furthermore, we focused on technology transfer by submitting 16 invention disclosures, resulting in 13 patent applications (including provisionals). Collaborating with our company partner, we obtained 6 granted patents. Additionally, we’re pleased to report on the progress of our spin-off projects, as well as the funding secured for two FFG-funded projects named APOLLO and GAIA, and an EU Horizon Europe-funded innovation action called SPIRIT.

The ATHENA team was also active in organizing scientific events such as workshops, special sessions, and special issues at IEEE ICME, ACM MM, ACM MMSys, ACM CoNEXT, IEEE ICIP, PCS, and IEEE Network. We also contributed to reproducibility in research through open source tools (e.g., Video Complexity Analyzer and LLL-CAdViSE) and datasets (e.g., Video Complexity Dataset and Multi-Codec Ultra High Definition 8K MPEG-DASH Dataset) among others.

We also note our contributions to the applications of AI in video coding & streaming, for example in video coding and video streaming as follows:

A major outcome of the second phase is the successful defense of the inaugural cohort of PhD students:

Two postdoctoral scholars have reached a significant milestone on their path toward habilitation

During the second phase, each work package produced excellent publications in their domain, briefly highlighted in the following. Content provisioning (WP-1) focuses mainly on video coding for HAS (43 papers) and immersive media coding for streaming (4 papers). The former can be further subdivided into the following topic areas:

  • Video complexity: spatial and temporal feature extraction (4 papers)
  • Compression efficiency improvement of individual representations (1 paper)
  • Encoding parameter prediction for HAS (9 papers)
  • Efficient bitrate ladder construction (4 papers)
  • Fast multi-rate encoding (3 papers)
  • Data security and data hiding (7 papers)
  • Energy-efficient video encoding for HAS (4 papers)
  • Advancing video quality evaluation (7 papers)
  • Datasets (4 papers)

Content delivery (WP-2) dealt with SDN/CDN assistance for HAS, edge computing support for HAS, and network-embedded media streaming support, resulting in 21 papers. Content consumption (WP-3) worked on QoE enhancement mechanisms at client-side and QoE- and energy-aware content consumption (11 papers). Finally, end-to-end Aspects (WP-4) produced 15 papers in the area of end-to-end QoE improvement in multimedia video streaming. We reported 94 papers published/accepted for the ATHENA 5-year evaluation.

In this context, it is also important to highlight the collaboration within ATHENA, which has resulted in joint publications across various work packages (WPs) and with other ITEC members. For example, collaborations with Prof. Schöffmann (FWF-funded project OVID), FFG-funded projects APOLLO/GAIA, and EU-funded project SPIRIT. In addition, we would like to acknowledge our international collaborators, such as Prof. Hongjie He from Southwest Jiaotong University, Prof. Patrick Le Callet from the University of Nantes, Prof. Wassim Hamidouche from the Technology Innovation Institute (UAE), Dr. Sergey Gorinsky from IMDEA, Dr. Abdelhak Bentaleb from Concordia University, Dr. Raimund Schatz from AIT, and Prof. Pablo Cesar from CWI. We are also pleased to report the successful technology transfers to Bitmovin, particularly CAdViSE (WP-4) and WISH ABR (WP-3). Regular “Fun with ATHENA” meetups and Break-out Groups are utilized for in-depth discussions about innovations and potential technology transfers.

Over the next two years, the ATHENA project will prioritize the development of deep neural network/AI-based image and video coding within the context of HAS. This includes energy- and cost-aware video coding for HAS, immersive video coding such as volumetric video and holography, as well as Quality of Experience (QoE) and energy-aware content consumption for HAS (including energy-efficient, AI-based live video streaming) and generative AI for HAS.

Thanks to all current and former ATHENA team members: Samira Afzal, Hadi Amirpour, Jesús Aguilar Armijo, Emanuele Artioli, Christian Bauer, Alexis Boniface, Ekrem Çetinkaya, Reza Ebrahimi, Alireza Erfanian, Reza Farahani, Mohammad Ghanbari (late), Milad Ghanbari, Mohammad Ghasempour, Selina Zoë Haack, Hermann Hellwagner, Manuel Hoi, Andreas Kogler, Gregor Lammer, Armin Lachini, David Langmeier, Sandro Linder, Daniele Lorenzi, Vignesh V Menon, Minh Nguyen, Engin Orhan, Lingfeng Qu, Jameson Steiner, Nina Stiller, Babak Taraghi, Farzad Tashtarian, Yuan Yuan, and Yiying Wei. Finally, thanks to ITEC support staff Martina Steinbacher, Nina Stiller, Margit Letter, Marion Taschwer, and Rudolf Messner.

We also would like to thank the Christian Doppler Research Association for continuous support, organizing the review, and the reviewer for constructive feedback!

Posted in ATHENA | Comments Off on Successful 5-year Evaluation of Christian Doppler Laboratory ATHENA