Strategies for combining audio and visual modalities of speech
There are 2 strategies, based on the 2 theories regarding fusion of audio & visual speech information in the human brain:
- Early integration strategy:
E.1. Combine the acoustic & visual parameters
set into a larger parameters set
E.2. Find the word whose template is
best matched to the audio-visual parameters set
- Late integration strategy:
L.1. Compare the audio against an acoustic
template for each word
L.2. Compare the video against a visual
template for each word
L.3. Combine the audio & visual recognition scores
-
-
Department of Informatics
Aristotle University of Thessaloniki