People

Spotlights

Christopher Raphael

Professor of Informatics

Music for the 21st Century

Technology has made the world an almost magical place.

Events from around the world can be streamed in real-time and watched on your phone as you wait in line at the grocery store. You can take a photo of your child and send it to his grandparents on the other side of the country in the time it takes for you to finish reading this sentence. Speech recognition software can produce the text of a major speech in minutes, or it can help you send a text without ever typing a letter. Computer vision is improving on an almost daily basis. We’re a long way from computers matching humans when it comes to recognizing images, but the gap continues to narrow.

When it comes to music, however, we’re pretty far from the cutting edge. We’re stuck in the last century.

Yes, more music is available in digital form than ever before. The access to new music across every imaginable genre is nearly unlimited as long as you have the desire, the money—sometimes not even that—and access to the Internet. But that doesn’t mean music is handled in a modern way.

“Ironically, music has not entered the 21st century in the way it has been digitized or represented symbolically,” says Christopher Raphael, a professor of informatics at the School of Informatics and Computing. “You can get audio of everything, which is a digital representation of music, but what drives the information revolution with text is that the text is represented symbolically. There isn’t anything analogous to that for music.”

Raphael is working on fixing that problem.

Raphael is striving to perfect music Optical Character Recognition in an effort to teach computers how to read sheet music, opening the door for the true digitalization and transposition of music by creating symbolic representations of large quantities of music.

Raphael knows those representations could come from a lot of different places, including the labor of humans, but Raphael believes it can be done more efficiently by computers.

“To computer science, it’s a pretty unappealing way of using people,” Raphael says. “People are going to be involved in the process because they have to be. They have to bless the results at the very least, but you would like to use them much more efficiently than that. You want the computer to do the big portion of the recognition.”

Exactly how to go about that, however, wasn’t easy for Raphael to grasp. The idea for music OCR had been bouncing around in his head for more than a decade, but he couldn’t quite come up with an approach or paradigm that made sense for him. Part of the issue is the nature of music.

“Music representations are just fundamentally two-dimensional,” Raphael says. “You can’t really get your algorithm to scan from left to right and consider all the possibilities as it does so.”

The goal of music OCR is to input sheet music into a computer and have it separate the information needed for each individual instrument, play the music, transpose the music, and re-notate it. All of that, however, leads to various issues with the music.

Music, after all, has always been written by humans with little thought to machines needing to read it. Humans can adapt to slang terms in music notations. Computers struggle if the music wanders from what the machine expects.

Researchers are slowly making breakthroughs to help computers better recognize and understand sheet music, and Raphael dreams of a day when musicians will have easy and uncomplicated access to sheet music both modern and old.

“In the future, we will have digital music stands just like e-readers,” Raphael says. “This will be the preferred way of doing music in the future because it’s so much more flexible if you can instantly download parts wirelessly. You never lose them, you can change them in lots of ways, or automatically analyze them. You can use them to comment, criticize or correct a person who is practicing. It’s hugely more flexible as a representation than it is as an image that you can just read.”

The good news is although the problem is challenging, Raphael and his students at SoIC are making strides, even if Raphael doesn’t have a single-minded goal.

“I would really like to have this single theme that unites all of my ideas, but I don’t have that,” Raphael says. “I think we’re making good progress. I have three students who are working on this. We have a great time doing it.”

Find out more about Christopher Raphael.