Dr. Diyana creates Hear Me app to assist students with special needs

KUANTAN, 5 July 2021 - For more than a year, the country has been facing the Covid-19 pandemic. Since then, a new norm learning environment has been established to implement home-based teaching and learning (PdPR) for students, including those with special needs.

There are currently 92,755 students with special needs in Malaysia, with almost 4,000 students under the hearing-related category.

Hearing problems are one of the factors that contribute to learning difficulties in school that can cause these students to fall behind.

Concerned with the problem, researchers from Universiti Malaysia Pahang (UMP) designed an app named Hear Me, a teaching tool for teachers and students with hearing disabilities to learn Manually Coded Malay (BMKT).

According to a lecturer from the Faculty of Industrial Management (FPI), Dr. Diyana Kamarudin, 37, the Hear Me app is equipped with artificial intelligence (AI) features and various other features contained in this mobile app.

“The AI feature on the Hear Me app can be used through the process of recognising an object through a picture taken, and then it will process it into the correct hand signal code.

“It is also equipped with various interactive activities that can help the learning process with short answer activities or fill in the blanks and match with the correct answer.

“Besides that, Hear Me also provides short stories and songs with links to some contents to users,” she said.

She added that it is interesting and simple for every user because it has been programmed by categories.

“The programmed categories are Letters, Colours, Activities, Short Stories and Songs, Numbers and Words for Manually Coded Malay (fruits and vegetables, about yourself, family, animals, days, my home, food and drinks, clothing, school and career) and AI functions.

“The app was developed in 2019 and successfully completed in early 2021.

“Hear Me will continue to improve various aspects along with the development of more advanced technology,” she said. 

She said that Hear Me is a mobile app realised based on brainstorming sessions, storyboard preparation process, and opinion and data collection process with full cooperation from the Ministry of Education Malaysia (MOE). 

“I am grateful because it is not my effort alone. My mother is also active in the field of Special Education.

“She inspired me a lot to continue to help the community with special needs,” she said.

During the development of Hear Me, they held discussions and visits to schools and interacted with teachers to discuss problems faced by teachers and students with special needs.

Based on her observations, many students with hearing problems will follow the process of copying a dictionary to improve the vocabulary learning process. 

In addition, the results of the meetings and communication with the deaf community and associations have found their first language is sign language, whether Manually Coded Malay (BMKT) or Malaysian Sign Language (BIM).
However, she added, they should learn Malay and English.

“The method of copying a dictionary to master additional languages is not an easy process.

“It was also confirmed by the teachers that sometimes students have difficulty remembering the vocabulary copied from the dictionary.

“Therefore, based on the language proficiency problem, I began to realise there is a need to help students, teachers and stakeholders in special education in facilitating teaching and learning (PdP),” she said.

This research also collaborated with Shenyang Ligong University China, City University Malaysia and MOE.

Talking about herself, the KL-born Dr. Diyana also shared her future plans, where the Hear Me app will be added with the functions of converting one- and two-dimensional pictures to three-dimensional (3D) pictures and producing illustrations in the form of cartoons to increase children’s engagement to use the mobile app easily. In addition, more games and new AI features will be added.

This Hear me app has acquired intellectual property rights and is still collecting data for project empowerment.
For those who are interested, this app can be downloaded through Google Playstore to help teachers and students with special needs throughout Malaysia.

The Hear Me app recently won a gold medal and a special award for the National STEM Association Award and Euro Business-Haller Award categories in the Malaysia Technology Expo (MTE) Covid-19 Special Edition 2021.

By: Nor Salwana Mohd Idris, Corporate Communications Unit, The Office of The Vice-Chancellor
Translation by: Dr. Rozaimi Abu Samah, Engineering College/Faculty of Chemical and Process Engineering Technology