Your Kitchen Table Is Now A Musical Instrument

Pocket-sized instruments have always appealed to musicians. Pulling out a harmonica, xaphoon, or ocarina and playing on the go seems like the pinnacle of convenience for the musically minded. But imagine the ability to transform any surface you come across into a musical instrument, using only your smartphone and a small device named after a tree.

Last year, four RPI students formed startup ToneTree and began work on such a device, which they've dubbed Birch. Birch consists of a very-high-frame-rate infrared camera that allows users to draw a sonic interface on a table or other flat surface. It's technologically similar to an infrared QWERTY keyboard, albeit with a much faster camera for real-time sound production.

ToneTree co-founder and CEO Brian Cook, who's pursuing an MFA in Integrated Electronic Arts, has been using his interest in music and technology to come up with innovative instruments since his undergrad days. At the University of Hartford he wrote a piece for the commemoration of a new building and invented a wired mallet device with integrated sensors and switches as well as remote cameras to enable performers to "play" the new building. Cook and his three ToneTree compatriots started work on the Birch prototype after he realized that doing away with the wires and creating a free-form instrument would heighten the experience.

The new instrument allows users to draw shapes on a flat surface and program them to correspond with waveforms like notes or percussive sounds. (Check out this demo.) This free-form interface makes Birch's expressive possibilities almost endless. The team has also talked about developing mats printed with a piano keyboard or guitar fretboard in the future.

The group has made rapid advances on Birch since undertaking the prototype in November 2014. They've just recently achieved data processing fast enough to eliminate most of the lag between finger movement and sound production and are still experimenting with faster optics. Birch renders a user's table or wall touch-sensitive, like a piano, by tracking finger velocity and producing tones that are proportionately forceful or gentle. The backend is powered by a combination of OpenCV for computer vision, MIDI for the musical portion, and Qt for the user interface.

The Birch is a patent-pending design and will be sold commercially in the coming year. Cook and software designer Ronald Sardarian discussed expanding research to a future device that would enable users to transform any flat surface into a touchscreen using a simple mobile device. It might be a useful addition to other promising augmented reality (AR) technologies such as Microsoft's HoloLens.

Classroom application of Birch seems like a natural progression, and the inventors are already partnering with local school STEAM (STEM+the arts) programs. Thinking back to my own school days, I imagine that most would rather play with this neat new technology than study the recorder. And if its future AR applications allow tapping walls rather than waving my hands in the air, well, sign me up for that too.

Image credit: Quickmeme