Increasing the efficiency and lessening the burden of effort on language preservation researchers
Language is a medium for intangible aspects of culture and heritage, including traditional knowledge, oral histories, and personal identity. And although apps like Duolingo and the rise in popularity of international media from South Korea and other countries suggest otherwise, 41% of all languages today are considered endangered, a trend largely correlated to increasing globalization, urbanization, and rapidly aging populations (The Language Conservancy, 2023).
Accurately translating one story from Apache to English took roughly 40hrs for Dr. Veronica Tiller’s research team
Researcher from the Linguistic Society of America transcribing interview notes with pen and paper
The Lulamogi tradition of eating termites is described through specific verbs that are at risk of being forgotten
Language preservation research has three main limitations: funding, time, and accessibility. Researchers often have to travel to far and remote locations with extensive equipment while working on a strict timeline and within the limits of their funding. With that in mind, I guided my work with the following 4 values:
Portable
Compact, light, and already used daily by most researchers
Easy to use
Utilizes simple functionalities and familiar UI/UX patterns
Cost & time efficient
Uses existing technology and UI/UX patterns
High quality
Assuring quality of transcription and data with human oversight
FEATURE
Realtime AI parsing
The main and defining functionality of this app. The algorithm parses out what the interviewer and interviewee say visually on the screen both for quality control and assurance that the session is being properly recorded.
RESEARCH INSIGHT
The interview process is the most crucial step to language preservation, requiring painstaking preparation, the hours dedicated to interviews themselves, and afterwards, meticulous human review and analysis of the raw data. By utilizing AI parsing, we can distribute the burden of effort from the researchers to the ML model itself, allowing them to spend that time on other important areas of their work.
At the same time, on the back end, each word is being broken down and parsed into a .CSV file to be uploaded into a database post-interview. This file will include attributes such as date recorded, part of speech, pronunciation, frequency, etc., and will visually identify low confidence information for human review.
Example of ChatGPT parsing the San Zi Jing, a Chinese literary classic, from Chinese into .CSV format that would be imported into a database
FEATURE
Two ways to interact with parsed data
If a word or phrase is parsed incorrectly, users have two ways to update the data that are helpful in different contexts and for different interviewing styles. In both cases, the mistranslated/parsed information will be updated ________.
METHOD 1
Tap to mark for review
Users can physically mark lines for review by tapping them as the appear on the screen, or by scrolling up through the live transcript. Marking these lines adds them to a queue that the user can then reference at a later point in time, such as at the end of the interview.
This functionality was designed to preserve the flow of interviews and prevent interruptions in scenarios such as when a live performance is happening.
METHOD 2
Touch free voice commands
Alternatively, users can prompt updates to datapoints by using certain phrases such as "what do you mean by..." or "can you clarify..." to ask for clarification.
This feature is best used in scenarios when a back and forth is happening or a lull in the conversation is conducive to more discussion.
FEATURE
Dual microphone mode
This feature utilizes the microphones present on both ends of modern smartphones to better parse and differentiate between the interviewer and interviewee’s lines and can be toggled on or off.
Dual microphones are best known and used for their noise cancelling functionalities, where one mic picks up all sound and the other filters out unwanted sound by playing equal frequencies with opposite amplitudes. Because there isn't a third microphone to cancel out unwanted sound, this feature is best used in traditional quiet interviewing settings.
INSPIRATION
Mozart's genius table music allowed musicians to play across from each other using the same sheet of music as they read from opposite ends. In the same vein, the dual microphone feature allows the user to conduct interviews in a candid, natural manner, as if they were engaging in normal conversation.
This work opens up a whole world of possibilities ranging from drawing features to record and document handwriting styles and stroke order/direction, expanding to a desktop version of the app, and even providing user submission functionality to build public databases and crowdsource research material. In addition there are so many extended applications beyond just language preservation that this work can be applied to, a few of which are highlighted below.
I hope to continue refining this core experience while also exploring some of the ideas outlined above--and if you're interested in making this project come to life, please contact me!
Journalism
Globalization of media (Instant dubbing/subtitles)
Medical translation for minority language speakers
Thank you for reading!