IEEE Communication Magazine: From Capturing to Rendering: Volumetric Media Delivery With Six Degrees of Freedom

From Capturing to Rendering: Volumetric Media Delivery With Six Degrees of Freedom

Teaser: “Help me, Obi-Wan Kenobi. You’re my only hope,” said the hologram of Princess Leia in Star Wars: Episode IV – A New Hope (1977). This was the first time in cinematic history that the concept of holographic-type communication was illustrated. Almost five decades later, technological advancements are quickly moving this type of communication from science fiction to reality.

IEEE Communication Magazine

[PDF]

Jeroen van der Hooft (Ghent University), Maria Torres Vega (Ghent University), Tim Wauters (Ghent University), Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Bitmovin), Ali C. Begen (Ozyegin University, Networked Media), Filip De Turck (Ghent University), and Raimund Schatz (AIT Austrian Institute of Technology)

Abstract: Technological improvements are rapidly advancing holographic-type content distribution. Significant research efforts have been made to meet the low-latency and high-bandwidth requirements set forward by interactive applications such as remote surgery and virtual reality. Recent research made six degrees of freedom (6DoF) for immersive media possible, where users may both move their heads and change their position within a scene. In this article, we present the status and challenges of 6DoF applications based on volumetric media, focusing on the key aspects required to deliver such services. Furthermore, we present results from a subjective study to highlight relevant directions for future research.

Posted in News | Comments Off on IEEE Communication Magazine: From Capturing to Rendering: Volumetric Media Delivery With Six Degrees of Freedom

MTAP paper: Automated Bank Cheque Verification Using Image Processing and Deep Learning Methods

MTAP paper: Automated Bank Cheque Verification Using Image Processing and Deep Learning Methods

Multimedia tools and applications (Springer Journal)

[PDF]

Prateek Agrawal (University of Klagenfurt, Austria), Deepak Chaudhary (Lovely Professional University, India), Vishu Madaan (Lovely professional University, India), Anatoliy Zabrovskiy (University of Klagenfurt, Austria), Radu Prodan (University of Klagenfurt, Austria), Dragi Kimovski (University of Klagenfurt, Austria), Christian Timmerer (University of Klagenfurt, Austria)

Abstract: Automated bank cheque verification using image processing is an attempt to complement the present cheque truncation system, as well as to provide an alternate methodology for the processing of bank cheques with minimal human intervention. When it comes to the clearance of the bank cheques and monetary transactions, this should not only be reliable and robust but also save time which is one of the major factor for the countries having large population. In order to perform the task of cheque verification, we developed a tool which acquires the cheque leaflet key components, essential for the task of cheque clearance using image processing and deep learning methods. These components include the bank branch code, cheque number, legal as well as courtesy amount, account number, and signature patterns. our innovation aims at benefiting the banking system by re-innovating the other competent cheque-based monetary transaction system which requires automated system intervention. For this research, we used institute of development and research in banking technology (IDRBT) cheque dataset and deep learning based convolutional neural networks (CNN) which gave us an accuracy of 99.14% for handwritten numeric character recognition. It resulted in improved accuracy and precise assessment of the handwritten components of bank cheque. For machine printed script, we used MATLAB in-built OCR method and the accuracy achieved is satisfactory (97.7%) also for verification of Signature we have used Scale Invariant Feature Transform (SIFT) for extraction of features and Support Vector Machine (SVM) as classifier, the accuracy achieved for signature verification is 98.10%.

Keywords: Cheque Truncation system, Image Segmentation, Bank Cheque Clearance, Image Feature Extraction, Convolution Neural Network, Support Vector Machine, Scale Invariant Feature Transform.

Acknowledgment: This work has been partly supported by the European Union Horizon 2020 Research and Innovation Programme under the ARTICONF Project with grant agreement number 644179 and in part by the Austrian Research Promotion Agency (FFG) under the APOLLO project.

Posted in APOLLO, News | Comments Off on MTAP paper: Automated Bank Cheque Verification Using Image Processing and Deep Learning Methods

VCIP’20: FaME-ML: Fast Multirate Encoding for HTTP Adaptive Streaming Using Machine Learning

FaME-ML: Fast Multirate Encoding for HTTP Adaptive Streaming Using Machine Learning

IEEE International Conference on Visual Communications and Image Processing (VCIP)

1-4 December 2020, Macau

http://www.vcip2020.org/

[PDF][Slides][Video]

Ekrem Çetinkaya (Alpen-Adria-Universität Klagenfurt), Hadi Amirpour (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Bitmovin), and Mohammad Ghanbari (School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK)

Abstract: HTTP Adaptive Streaming(HAS) is the most common approach for delivering video content over the Internet. The requirement to encode the same content at different quality levels (i.e., representations) in HAS is a challenging problem for content providers. Fast multirate encoding approaches try to accelerate this process by reusing information from previously encoded representations. In this paper, we propose to use convolutional neural networks (CNNs) to speed up the encoding of multiple representations with a specific focus on parallel encoding. In parallel encoding, the overall time-complexity is limited to the maximum time-complexity of one of the representations that are encoded in parallel. Therefore, instead of reducing the time-complexity for all representations, the highest time-complexities are reduced. Experimental results show that FaME-ML achieves significant time-complexity savings in parallel encoding scenarios(41%in average) with a slight increase in bitrate and quality degradation compared to the HEVC reference software.

Keywords: Video Coding, Convolutional Neural Networks, HEVC, HTTP Adaptive Streaming (HAS)

Posted in News | Comments Off on VCIP’20: FaME-ML: Fast Multirate Encoding for HTTP Adaptive Streaming Using Machine Learning

Interns at the ATHENA Christian Doppler Laboratory

In July 2020, the ATHENA Christian Doppler Laboratory hosted three interns working on the following topics:

  • Indira Pal: Quality of Experience for HTTP Adaptive Streaming based use cases and scenarios.
  • Ilja Pronegg: Machine learning methods for HTTP Adaptive Streaming.
  • Miriam Gütl: State-of-the-art video streaming technologies.

At the end of the internship, the interns presented their work and the results in the form of a presentation and report. We believe that the joint work was useful both for the laboratory and for the interns themselves. We would like to thank the interns for their productive work, useful results, and excellent feedback about our laboratory.

Indira Pal: “I liked the internship very much and felt like a part of the team very soon. This was most likely due to the awesome atmosphere at the Christian Doppler laboratory. In my brief time at ATHENA, I was able to learn a lot about video compression and the challenges associated with this fairly widespread topic. Hadi was an excellent supervisor because he was always at the office and let me work independently while still providing assistance when I needed it. I enjoyed getting to know a Computer Scientist’s/Engineer’s work and now have a much clearer view of what a PhD student’s or Postdoc’s work looks like. I am very glad that I applied for this internship as I believe that my work at ATHENA was truly meaningful and I gained a lot from this experience.”

Miriam Gutl: “I really enjoyed every day of my internship at ATHENA and I am actually quite sad that the four weeks passed by so quickly. Already after the first day I felt so intergrated in the whole team which was so great for the whole atmosphere. In general the whole team was always so nice to us which was a real isperation. Also I would say that I have never learned so much in such a short periode of time till now. So I am really very thankfull that I was taken for this job it really did learned me a lot and in between it was also extremly fun. When I started looking for a internship I wanted something like that and I have actually found it at ATHENA so thank you.”

Ilja Pronegg:The internship was awesome! Everyone was nice and the topics I worked on were very interesting. I would make a second internship here!”

We wish the interns every success in their journey through life and we hope seeing them soon back at the University of Klagenfurt and ATHENA.

Posted in News | Comments Off on Interns at the ATHENA Christian Doppler Laboratory

ACM Multimedia’20: Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Neural Networks

ACM MM’20: Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Neural Networks

ACM International Conference on Multimedia 2020, Seattle, United States.
https://2020.acmmm.org

[PDF][Slides][Video]

Negin Ghamsarian (Alpen-Adria-Universität Klagenfurt), Hadi Amirpour (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Bitmovin), Mario Taschwer (Alpen-Adria-Universität Klagenfurt), and Klaus Schöffmann (Alpen-Adria-Universität Klagenfurt)

Abstract: Recorded cataract surgery videos play a prominent role in training and investigating the surgery, and enhancing the surgical outcomes. Due to storage limitations in hospitals, however, the recorded cataract surgeries are deleted after a short time and this precious source of information cannot be fully utilized. Lowering the quality to reduce the required storage space is not advisable since the degraded visual quality results in the loss of relevant information that limits the usage of these videos. To address this problem, we propose a relevance-based compression technique consisting of two modules: (i) relevance detection, which uses neural networks for semantic segmentation and classification of the videos to detect relevant spatio-temporal information, and (ii) content-adaptive compression, which restricts the amount of distortion applied to the relevant content while allocating less bitrate to irrelevant content. The proposed relevance-based compression framework is implemented considering five scenarios based on the definition of relevant information from the target audience’s perspective. Experimental results demonstrate the capability of the proposed approach in relevance detection. We further show that the proposed approach can achieve high compression efficiency by abstracting substantial redundant information while retaining the high quality of the relevant content.

Keywords: Video Coding, Convolutional Neural Networks, HEVC, ROI Detection, Medical Multimedia.

Posted in News | Comments Off on ACM Multimedia’20: Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Neural Networks

QUALINET White Paper on Definitions of Immersive Media Experience (IMEx)

QUALINET announces its recent White Paper on Definitions of Immersive Media Experience (IMEx).

It is online available here https://arxiv.org/abs/2007.07032 for free.

With the coming of age of virtual/augmented reality and interactive media, numerous definitions, frameworks, and models of immersion have emerged across different fields ranging from computer graphics to literary works. Immersion is oftentimes used interchangeably with presence as both concepts are closely related. However, there are noticeable interdisciplinary differences regarding definitions, scope, and constituents that are required to be addressed so that a coherent understanding of the concepts can be achieved. Such consensus is vital for paving the directionality of the future of immersive media experiences (IMEx) and all related matters.

The aim of this white paper is to provide a survey of definitions of immersion and presence which leads to a definition of immersive media experience (IMEx). The Quality of Experience (QoE) for immersive media is described by establishing a relationship between the concepts of QoE and IMEx followed by application areas of immersive media experience. Influencing factors on immersive media experience are elaborated as well as the assessment of immersive media experience. Finally, standardization activities related to IMEx are highlighted and the white paper is concluded with an outlook related to future developments.

This White Paper is a contribution by QUALINET, the European Network on Quality of Experience in Multimedia Systems and Services (http://www.qualinet.eu/) to the discussions related to Immersive Media Experience (IMEx). It is motivated by the need for definitions around this term to foster a deeper understanding of ideas and concepts originating from multidisciplinary groups but with a joint interest in multimedia experiences. Thus, this white paper has been created mainly with such multimedia experiences in mind but may be also used beyond.

The QUALINET community aims at extending the notion of network-centric Quality of Service (QoS) in multimedia systems, by relying on the concept of Quality of Experience (QoE). The main scientific objective is the development of methodologies for subjective and objective quality metrics taking into account current and new trends in multimedia communication systems as witnessed by the appearance of new types of content and interactions. QUALINET (2010-2014 as COST Action IC1003) meets once a year collocated with QoMEX (http://qomex.org/) to coordinate its activities around 4 Working Groups (WGs): (i) research, (ii) standardization, (iii) training, and (iv) innovation.

List of Authors and Contributors
Andrew Perkis (andrew.perkis@ntnu.no, editor), Christian Timmerer (christian.timmerer@itec.uni-klu.ac.at, editor), Sabina Baraković, Jasmina Baraković Husić, Søren Bech, Sebastian Bosse, Jean Botev, Kjell Brunnström, Luis Cruz, Katrien De Moor, Andrea de Polo Saibanti, Wouter Durnez, Sebastian Egger-Lampl, Ulrich Engelke, Tiago H. Falk, Asim Hameed, Andrew Hines, Tanja Kojic, Dragan Kukolj, Eirini Liotou, Dragorad Milovanovic, Sebastian Möller, Niall Murray, Babak Naderi, Manuela Pereira, Stuart Perry, Antonio Pinheiro, Andres Pinilla, Alexander Raake, Sarvesh Rajesh Agrawal, Ulrich Reiter, Rafael Rodrigues, Raimund Schatz, Peter Schelkens, Steven Schmidt, Saeed Shafiee Sabet, Ashutosh Singla, Lea Skorin-Kapov, Mirko Suznjevic, Stefan Uhrig, Sara Vlahović, Jan-Niklas Voigt- Antons, Saman Zadtootaghaj.

How to reference this white paper
Perkis, A., Timmerer, C., et al., “QUALINET White Paper
on Definitions of Immersive Media Experience (IMEx)”, European Network on Quality of Experience in Multimedia Systems and Services, 14th QUALINET meeting (online), May 25, 2020.

Alternatively, you may export the citation from arXiv: https://arxiv.org/abs/2007.07032.

Posted in News | Comments Off on QUALINET White Paper on Definitions of Immersive Media Experience (IMEx)

BigMM’20: ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Sequences using Artificial Neural Network

ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Sequences Using Artificial Neural Network

The Sixth IEEE International Conference on Multimedia Big Data (BigMM 2020)

September 24-26, 2020 New Delhi. http://bigmm2020.org/

[PDF][Slides][Video]

Anatoliy Zabrovskiy (Alpen-Adria-Universität Klagenfurt), Prateek Agrawal (Alpen-Adria-Universität Klagenfurt, Lovely Professional  University), Roland Mathá (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Bitmovin) and Radu Prodan (Alpen-Adria-Universität Klagenfurt).

Abstract: HTTP Adaptive Streaming of video content is becoming an integral part of the Internet and accounts for the majority of today’s traffic. Although Internet bandwidth is constantly increasing, video compression technology plays an important role and the major  challenge is to select and set up multiple video codecs, each with hundreds of transcoding parameters. Additionally, the transcoding speed depends directly on the selected transcoding parameters and the infrastructure used. Predicting transcoding time for multiple transcoding parameters with different codecs and processing units is a challenging task, as it depends on many factors. This paper provides a novel and considerably fast method for transcoding time prediction using video content classification and neural network prediction. Our artificial neural network (ANN) model predicts the transcoding times of video segments for state-of-the-art video codecs based on transcoding parameters and content complexity. We evaluated our method for two video codecs/implementations (AVC/x264 and HEVC/x265) as part of large-scale HTTP Adaptive Streaming services. The ANN model of our method is able to predict the transcoding time by minimizing the mean absolute error (MAE) to 1.37 and 2.67 for x264 and x265 codecs, respectively. For x264, this is an improvement of 22% compared to the state of the art.

Keywords: Transcoding time prediction, adaptive streaming, video transcoding, neural networks, video encoding, video complexity class, MPEG-DASH

Acknowledgment: This work has been supported in part by the Austrian Research Promotion Agency (FFG) under the APOLLO project and by the European Union Horizon 2020 Research and Innovation Programme under the ASPIDE Project with grant agreement number 801091.

Posted in APOLLO, News | Comments Off on BigMM’20: ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Sequences using Artificial Neural Network

Hadi Amirpour awarded the Best Engagement Award at ACM MMSys 2020

Hadi Amirpour has been awarded the Best Engagement Award at ACM MMSys 2020.

This year’s ACM MMSys was held as a fully virtual/online event and Slido was used for asking questions about keynotes and presentations including offline discussions with presenters. The interaction report provides some interesting key insights including the word cloud below which provides an overview of this year’s discussion items.

Although ACM MMSys 2020 is over, everyone is welcome joining the MMSys Slack workspace where the discussion will continue until ACM MMSys 2021 (available soon!) and hopefully beyond.

Posted in News | Comments Off on Hadi Amirpour awarded the Best Engagement Award at ACM MMSys 2020

EPIQ’20: Scalable High Efficiency Video Coding based HTTP Adaptive Streaming over QUIC Using Retransmission

Scalable High Efficiency Video Coding based HTTP Adaptive Streaming over QUIC Using Retransmission

ACM SIGCOMM 2020 Workshop on Evolution, Performance, and Interoperability of QUIC (EPIQ 2020)
August 10–14, 2020
https://conferences.sigcomm.org/sigcomm/2020/workshop-epiq.html

[PDF][Slides][Video]

Minh Nguyen (Alpen-Adria-Universität Klagenfurt), Hadi Amirpour (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt / Bitmovin Inc.), Hermann Hellwagner (Alpen-Adria-Universität Klagenfurt)

Abstract: HTTP/2 has been explored widely for video streaming, but still suffers from Head-of-Line blocking, and three-way hand-shake delay due to TCP. Meanwhile, QUIC running on top of UDP can tackle these issues. In addition, although many adaptive bitrate (ABR) algorithms have been proposed for scalable and non-scalable video streaming, the literature lacks an algorithm designed for both types of video streaming approaches. In this paper, we investigate the impact of quick and HTTP/2 on the performance of adaptive bitrate(ABR) algorithms in terms of different metrics. Moreover, we propose an efficient approach for utilizing scalable video coding formats for adaptive video streaming that combines a traditional video streaming approach (based on non-scalable video coding formats) and a retransmission technique. The experimental results show that QUIC benefits significantly from our proposed method in the context of packet loss and retransmission. Compared to HTTP/2, it improves the average video quality and also provides a smoother adaptation behavior. Finally, we demonstrate that our proposed method originally designed for non-scalable video codecs also works efficiently for scalable videos such as Scalable High EfficiencyVideo Coding(SHVC).

Keywords: QUIC, H2BR, HTTP adaptive streaming, Retransmission, SHVC

Posted in News | Comments Off on EPIQ’20: Scalable High Efficiency Video Coding based HTTP Adaptive Streaming over QUIC Using Retransmission

PV’20: H2BR: An HTTP/2-based Retransmission Technique to Improve the QoE of Adaptive Video Streaming

H2BR: An HTTP/2-based Retransmission Technique to Improve the QoE of Adaptive Video Streaming

Packet Video Workshop 2020 (PV)
June 10-11, 2020, Istanbul, Turkey (co-located with ACM MMSys’20)

https://2020.packet.video/

[PDF][Slides][Video]

Minh Nguyen (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt / Bitmovin Inc.), Hermann Hellwagner (Alpen-Adria-Universität Klagenfurt)

Abstract: HTTP-based Adaptive Streaming (HAS) plays a key role in over-the-top video streaming. It contributes towards reducing the rebuffering duration of video playout by adapting the video quality to the current network conditions. However, it incurs variations of video quality in a streaming session because of the throughput fluctuation, which impacts the user’s Quality of Experience (QoE). Besides, many adaptive bitrate (ABR) algorithms choose the lowest-quality segments at the beginning of the streaming session to ramp up the playout buffer as soon as possible. Although this strategy decreases the startup time, the users can be annoyed as they have to watch a low-quality video initially. In this paper, we propose an efficient retransmission technique, namely H2BR, to replace low-quality segments being stored in the playout buffer with higher-quality versions by using features of HTTP/2 including (i) stream priority, (ii) server push, and (iii) stream termination. The experimental results show that H2BR helps users avoid watching low video quality during video playback and improves the user’s QoE. H2BR can decrease by up to more than 70% the time when the users suffer the lowest-quality video as well as benefits the QoE by up to 13%.

Keywords: HTTP adaptive streaming, DASH, ABR algorithms, QoE, HTTP/2

Posted in News | Comments Off on PV’20: H2BR: An HTTP/2-based Retransmission Technique to Improve the QoE of Adaptive Video Streaming