Export Publication

Freitas, J., Teixeira, A. & Dias, M. S. (2013). Multimodal silent speech interface based on video, depth, surface electromyography and ultrasonic Doppler: Data collection and first recognition results. In Bilmes, J., Fosler-Lussier, E., Hasegawa-Johnson, M., and Livescu, K. (Ed.), Workshop on Speech Production in Automatic Speech Recognition (SPASR-2013). (pp. 44-49). Lyon: International Speech and Communication Association.

J. Freitas et al.,  "Multimodal silent speech interface based on video, depth, surface electromyography and ultrasonic Doppler: Data collection and first recognition results", in Workshop on Speech Production in Automatic Speech Recognition (SPASR-2013), Bilmes, J., Fosler-Lussier, E., Hasegawa-Johnson, M., and Livescu, K., Ed., Lyon, International Speech and Communication Association, 2013, pp. 44-49

@inproceedings{freitas2013_1716143003331,
	author = "Freitas, J. and Teixeira, A. and Dias, M. S.",
	title = "Multimodal silent speech interface based on video, depth, surface electromyography and ultrasonic Doppler: Data collection and first recognition results",
	booktitle = "Workshop on Speech Production in Automatic Speech Recognition (SPASR-2013)",
	year = "2013",
	editor = "Bilmes, J., Fosler-Lussier, E., Hasegawa-Johnson, M., and Livescu, K.",
	volume = "",
	number = "",
	series = "",
	pages = "44-49",
	publisher = "International Speech and Communication Association",
	address = "Lyon",
	organization = "Institut des Sciences de l'Homme in Lyon",
	url = "https://home.ttic.edu/~klivescu/SPASR2013/"
}

TY  - CPAPER
TI  - Multimodal silent speech interface based on video, depth, surface electromyography and ultrasonic Doppler: Data collection and first recognition results
T2  - Workshop on Speech Production in Automatic Speech Recognition (SPASR-2013)
AU  - Freitas, J.
AU  - Teixeira, A.
AU  - Dias, M. S.
PY  - 2013
SP  - 44-49
CY  - Lyon
UR  - https://home.ttic.edu/~klivescu/SPASR2013/
AB  - Silent Speech Interfaces use data from the speech production 
process, such as visual information of face movements. 
However, using a single modality limits the amount of available 
information. In this study we start to explore the use of multiple 
data input modalities in order to acquire a more complete 
representation of the speech production model. We have 
selected 4 non-invasive modalities – Visual data from Video 
and Depth, Surface Electromyography and Ultrasonic Doppler 
- and created a system that explores the synchronous 
combination of all 4, or of a subset of them, into a multimodal 
Silent Speech Interface (SSI). This paper describes the system 
design, data collection and first word recognition results. As the 
first acquired corpora are necessarily small for this SSI, we use 
for classification an example based recognition approach based 
on Dynamic Time Warping followed by a weighted k-Nearest 
Neighbor classifier. The first classification results using 
different vocabularies, with digits, a small set of commands 
related to Ambient Assisted Living and minimal nasal pairs, 
show that word recognition benefits can be obtained from a 
multimodal approach.
ER  -