By the time Matthew Fernandez and Akash Krishnan were old enough to drive, they had invented an algorithm that can listen to human speech and detect dozens of emotions.
Now, the pair are using their algorithm — the centerpiece of a company called Simple Emotion — to revolutionize call centers around the globe.
Often, the worst part of a customer service experience is speaking to an agent or robot that cannot empathize with a caller about the situation that led him or her to complain. This tension can cause hair-pulling frustration for a customer.
As part of a high school science research project, Fernandez and Krishnan set out to build something that could recognize emotions in people's voices and provide real-time feedback to the representative taking the call. The information could be used to provide the best service experience, as well as for quality assurance and training purposes.
The Simple Emotion algorithm works by monitoring acoustic features in speech — such as voice frequency, volume, and changes in tone over time — and comparing them to a library of sounds and tones. It identifies an emotion by finding the closest match in the catalog.
Fernandez and Krishnan culled audio files expressing specific emotions from a database and fed those examples to the algorithm to train it about how different emotions sound. Today, it understands between 30 and 40 emotions, according to Fernandez.
Companies can hire Simple Emotion to do speech analysis for them, or license its API and integrate the algorithm into their own customer service tools.
So, how did two 16-year-old kids decide to craft something like this?
"We watched the movie 'I, Robot,'" Fernandez, who's now a soon-to-be graduate of Stanford University, tells Tech Insider on the phone.
Though he says the film is "only okay," there's a scene in which the robot comes into a room where two people are fighting and offers to help. "I detected elevated stress patterns in your voice," the robot says to Will Smith.
"We were inspired enough to Google it," Fernandez says. "[Emotion detection] is this small, niche thing, so we decided to build our own algorithm."
Fernandez and Krishnan started to stay up late reading research papers and blow off their other homework. When they finally submitted their science project, it included 30 pages of code and 60 pages of technical writing to explain how the whole thing worked.
They later took the grand prize at the Siemens Competition in Math, Science & Technology, the nation's most prestigious research competition for high school students.
Once they enrolled in college (Fernandez at Stanford and Krishnan at MIT), the pair continued working together across the country to hone their algorithm, spending hours talking on Skype. Fernandez guesses he spends as many hours working on Simple Emotion as he does on his school work.
Though theirs is not the first system that attempts to gauge customers' emotions, Fernandez and Krishnan argue that Simple Emotion's algorithm is the most accurate because it scrapes data directly from the source: voice.
Other systems convert the speech to text first, and identify emotions by searching for certain keywords in the script. (A customer that swore a lot is probably angry, for example.) Companies can also improve their customer service response by paying a human to listen to a call once it's been recorded and make an evaluation based on quality assurance standards, but doing so is expensive. Or a representative can send a customer survey by email after hanging up, but Fernandez estimates 90% to 95% of recipients will ignore it.
"How your customer feels — that's this Holy Grail nugget of information that a call center representative wants," Fernandez says, suggesting that his system is roughly 50% more accurate than comparable systems.
In the future, companies using Simple Emotion could program the robot that greets you at the beginning of a call (asking you to say your account number aloud, for example) to route you to the agent best suited to handle your needs. The algorithm might even prioritize customers in the queue who sound irate.
Fernandez and Krishnan plan to continue growing their company after they finish college. Simple Emotion has raised about half a million dollars in seed funding from the Portland Seed Fund and ZenShin Capital, and currently has a pilot project underway at the Mizuho Financial Group, Japan's second largest bank by assets. Fernandez declined to give additional details about the pilot program.
Although a computer that knows how you're feeling and can respond accordingly sounds manipulative, Fernandez says that can be a positive.
"It's a little bit creepy," Fernandez says. "But hey, if you get better service that's only a win for everybody."