Effective visualization of articulatory processes using visual speech synthesis technologies for corrective feedback generation in computer-aided pronunciation training for Chinese learners of English

Principal Supervisor

Professor Helen Meng, Department of Systems Engineering and Engineering Management
Professor Virginia Yip, Department of Linguistics and Modern Languages
Duration

3 years
Approved Budget

HK $200,000
 
  • Project Objectives
  • Key deliverables expected
  • Evaluation of deliverables/outcomes
  • Dissemination of deliverables/outcomes

Project Objectives

This project proposes to incorporate visual synthesis technologies that generate articulatory motions in synchrony with synthetic speech. This will be used to extend an existing computer-aided pronunciation training (CAPT) system, by generating audiovisual corrective feedback to facilitate pronunciation training. We will attempt to synthesize speech segments corresponding to detected mispronunciations should be synthesized with hyper-articulation. The degree of hyper-articulation should balance clearer visualization against excessive distortion. We will leverage our existing CAPT platform (referred as the Enunciate system), which is primarily based on audio and has been made accessible across CUHK campus.

Key deliverables expected and timelines for completion

  • Acoustic-phonetic analysis of Chinese-accented English speech (Jan to Dec 2013)
  • Development of Text-to-Visual Speech Synthesis Technology (Jan to Dec 2014)
  • Integration with the Enunciate System and testing with uses (Jan to Dec 2015)

    • Arrangements for evaluation of project deliverables/outcomes

    Our system will be made accessible across CUHK campus and system logs are recorded to inform further development. We will also liaise with the English Language Teaching Unit and the Independent Learning Centre for in-class trials.

    Means for disseminating project deliverables/outcomes

    We will also list the system in repositories of electronic learning resources at CUHK. We will also submit papers to relevant journal publications.