COVID-19 Update: We are currently shipping orders daily. However, due to transit disruptions in some geographies, deliveries may be delayed. To provide all customers with timely access to content, we are offering 50% off Science and Technology Print & eBook bundle options. Terms & conditions.
Multimodal Signal Processing - 1st Edition - ISBN: 9780123748256, 9780080888699

Multimodal Signal Processing

1st Edition

Theory and Applications for Human-Computer Interaction

0.0 star rating Write a review
Editors: Jean-Philippe Thiran Ferran Marqués Hervé Bourlard
Hardcover ISBN: 9780123748256
eBook ISBN: 9780080888699
Imprint: Academic Press
Published Date: 17th November 2009
Page Count: 352
Sales tax will be calculated at check-out Price includes VAT/GST
Price includes VAT/GST

Institutional Subscription

Secure Checkout

Personal information is secured with SSL technology.

Free Shipping

Free global shipping
No minimum order.


Multimodal signal processing is an important research and development field that processes signals and combines information from a variety of modalities – speech, vision, language, text – which significantly enhance the understanding, modelling, and performance of human-computer interaction devices or systems enhancing human-human communication. The overarching theme of this book is the application of signal processing and statistical machine learning techniques to problems arising in this multi-disciplinary field. It describes the capabilities and limitations of current technologies, and discusses the technical challenges that must be overcome to develop efficient and user-friendly multimodal interactive systems.

With contributions from the leading experts in the field, the present book should serve as a reference in multimodal signal processing for signal processing researchers, graduate students, R&D engineers, and computer engineers who are interested in this emerging field.

Key Features

  • Presents state-of-art methods for multimodal signal processing, analysis, and modeling
  • Contains numerous examples of systems with different modalities combined
  • Describes advanced applications in multimodal Human-Computer Interaction (HCI) as well as in computer-based analysis and modelling of multimodal human-human communication scenes.


Signal, acoustic, speech, image and video processing university (applied) researchers, R&D engineers, computer engineers

Table of Contents

1. Introduction
Jean-Philippe Thiran, Ferran Marqués, and Hervé Bourlard

Part I -- Signal Processing, Modelling and Related Mathematical Tools

2. Statistical Machine Learning for HCI
Samy Bengio

2.1. Introduction
2.2. Introduction to Statistical Learning
2.3. Support Vector Machines for Binary Classification
2.4. Hidden Markov Models for Speech Recognition
2.5. Conclusion

3. Speech Processing
Thierry Dutoit and Stéphane Dupont

3.1. Introduction
3.2. Speech Recognition
3.3. Speaker Recognition
3.4. Text-to-Speech Synthesis
3.5. Conclusions

4. Natural Language and Dialogue Processing
Olivier Pietquin

4.1. Introduction
4.2. Natural Language Understanding
4.3. Natural Language Generation
4.4. Dialogue Processing
4.5. Conclusion

5. Image and Video Processing Tools for HCI
Montse Pardàs, Verónica Vilaplana and Cristian Canton-Ferrer

5.1. Introduction
5.2. Face Analysis
5.3. Hand-Gesture Analysis
5.4. Head Orientation Analysis and FoA Estimation
5.5. Body Gesture Analysis
5.6. Conclusions

6. Processing of Handwriting and Sketching Dynamics
Claus Vielhauer

6.1. Introduction
6.2. History of Handwriting Modality and the Acquisition of Online Handwriting Signals
6.3. Basics in Acquisition, Examples for Sensors
6.4. Analysis of Online Handwriting and Sketching Signals
6.5. Overview of Recognition Goals in HCI
6.6. Sketch Recognition for User Interface Design
6.7. Similarity Search in Digital Ink
6.8. Summary and Perspectives for Handwriting and Sketching in HCI

Part II -- Multimodal Signal Processing and Modelling

7. Basic Concepts of Multimodal Analysis
Mihai Gurban and Jean-Philippe Thiran

7.1. Defining Multimodality
7.2. Advantages of Multimodal Analysis
7.3. Conclusion

8. Multimodal Information Fusion
Norman Poh and Josef Kittler

8.1. Introduction
8.2. Levels of Fusion
8.3. Adaptive versus Non-Adaptive Fusion
8.4. Other Design Issues
8.5. Conclusions

9. Modality Integration Methods
Mihai Gurban and Jean-Philippe Thiran

9.1. Introduction
9.2. Multimodal Fusion for AVSR
9.3. Multimodal Speaker Localisation
9.4. Conclusion

10. A Multimodal Recognition Framework for Joint Modality Compensation and Fusion
Konstantinos Moustakas, Savvas Argyropoulos and Dimitrios Tzovaras

10.1. Introduction
10.2. Joint Modality Recognition and Applications
10.3. A New Joint Modality Recognition Scheme
10.4. Joint Modality Audio-Visual Speech Recognition
10.5. Joint Modality Recognition in Biometrics
10.6. Conclusions

11. Managing Multimodal Data, Metadata and Annotations: Challenges and Solutions
Andrei Popescu-Belis

11.1. Introduction
11.2. Setting the Stage: Concepts and Projects
11.3. Capturing and Recording Multimodal Data
11.4. Reference Metadata and Annotations
11.5. Data Storage and Access
11.6. Conclusions and Perspectives

Part III -- Multimodal Human–Computer and Human-to-Human Interaction

12. Multimodal Input
Natalie Ruiz, Fang Chen, and Sharon Oviatt

12.1. Introduction
12.2. Advantages of Multimodal Input Interfaces
12.3. Multimodality, Cognition and Performance
12.4. Understanding Multimodal Input Behaviour
12.5. Adaptive Multimodal Interfaces
12.6. Conclusions and Future Directions

13. Multimodal HCI Output: Facial Motion, Gestures and Synthesised Speech Synchronisation
Igor S. Pandžic

13.1. Introduction
13.2. Basic AV Speech Synthesis
13.3. The Animation System
13.4. Coarticulation
13.5. Extended AV Speech Synthesis
13.6. Embodied Conversational Agents
13.7. T TS Timing Issues
13.8. Conclusion

14. Interactive Representations of Multimodal Databases
Stéphane Marchand-Maillet, Donn Morrison, Enikö Szekely, and Eric Bruno

14.1. Introduction
14.2. Multimodal Data Representation
14.3. Multimodal Data Access
14.4. Gaining Semantic from User Interaction
14.5. Conclusion and Discussion

15. Modelling Interest in Face-to-Face Conversations from Multimodal Nonverbal Behaviour
Daniel Gatica-Perez

15.1. Introduction
15.2. Perspectives on Interest Modelling
15.3. Computing Interest from Audio Cues
15.4. Computing Interest from Multimodal Cues
15.5. Other Concepts Related to Interest
15.6. Concluding Remarks


No. of pages:
© Academic Press 2010
17th November 2009
Academic Press
Hardcover ISBN:
eBook ISBN:

About the Editors

Jean-Philippe Thiran

Affiliations and Expertise

EPFL, Lausanne, Switzerland

Ferran Marqués

Affiliations and Expertise

Technical University of Catalonia, Spain

Hervé Bourlard

Affiliations and Expertise

Director, IDIAP Research Institute, EPFL, Lausanne, Switzerland

Ratings and Reviews