Description

This project aims to develop a multimodal chatbot WordPress plugin to assist individuals with aphasia—a language and communication disorder often caused by stroke or brain injury. The chatbot integrates voice, video (gesture), text, and mouse input to help users interact seamlessly with the Communication Connect platform developed by La Trobe University's CDAC team. Designed for inclusivity, the chatbot will allow aphasia patients to navigate websites more independently, improving their online experience and accessibility.

Motivation and Research Objective

People with aphasia face significant challenges in reading, typing, or understanding complex web interfaces. Traditional chatbots do not accommodate multimodal communication needs such as gesture or speech input. With over 250,000 Australians living with aphasia, there is a growing need for assistive technologies that are adaptable and intuitive.

This project is driven by the need to:

• Enable intent recognition from multiple input types (text, voice, video, clicks).

• Fuse multimodal signals to improve the accuracy of chatbot understanding.

• Develop a robust, accessible chatbot plugin that enhances web navigation for users with communication disabilities.

The final outcome will be a reusable multimodal chatbot plugin for WordPress, promoting digital accessibility and empowering users with aphasia through AI for good.

Results and Impact

Our multimodal chatbot demonstrated strong potential to assist individuals with aphasia in navigating websites independently. By integrating advanced gesture recognition, speech transcription, and intent classification, the chatbot supports multiple input types—voice, video, text, and mouse clicks—delivering a highly accessible user experience.

Through extensive testing, the system proved to be more intuitive and inclusive than traditional chatbots, especially for users with communication disabilities. Its modular WordPress plugin design ensures broad compatibility across websites, extending its reach to a wide range of users and healthcare platforms.

This project showcases how AI can be used for social good, significantly improving digital accessibility and quality of life for people with language impairments.

Team members

R.L.A. Sanduni Ayesha

Department of Computer Science and Engineering,
University of Moratuwa, Sri Lanka.

L.B.H.M. Vijayarathna

Department of Computer Science and Engineering,
University of Moratuwa, Sri Lanka.

N.P.A. Vithanage

Department of Computer Science and Engineering,
University of Moratuwa, Sri Lanka.