Real-time Gesture Animation Generation from Speech for Virtual Human Interaction

Manuel Rebol, Christian Gütl, Krzysztof Pietroszek

Publikation: Beitrag in Buch/Bericht/KonferenzbandBeitrag in einem KonferenzbandBegutachtung

Abstract

We propose a real-time system for synthesizing gestures directly from speech. Our data-driven approach is based on Generative Adversarial Neural Networks to model the speech-gesture relationship. We utilize the large amount of speaker video data available online to train our 3D gesture model. Our model generates speaker-specific gestures by taking consecutive audio input chunks of two seconds in length. We animate the predicted gestures on a virtual avatar. We achieve a delay below three seconds between the time of audio input and gesture animation.

Originalspracheenglisch
TitelExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021
Herausgeber (Verlag)Association of Computing Machinery
Seiten327-344
ISBN (elektronisch)9781450380959
DOIs
PublikationsstatusVeröffentlicht - 8 Mai 2021
Veranstaltung2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths: CHI 2021 - Virtual, Online, Japan
Dauer: 8 Mai 202113 Mai 2021

Publikationsreihe

NameConference on Human Factors in Computing Systems - Proceedings

Konferenz

Konferenz2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths
Land/GebietJapan
OrtVirtual, Online
Zeitraum8/05/2113/05/21

ASJC Scopus subject areas

  • Human-computer interaction
  • Computergrafik und computergestütztes Design
  • Software

Fields of Expertise

  • Information, Communication & Computing

Fingerprint

Untersuchen Sie die Forschungsthemen von „Real-time Gesture Animation Generation from Speech for Virtual Human Interaction“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren