Music technology – Matan Berkowitz: Two weeks before coming here, I shared a different stage with four very inspiring musicians: a paralyzed electronic producer, who creates music using his eyes, a blind and autistic pianist, who uses artificial intelligence to enhance his left-hand playing, a multi-instrumentalist in a wheelchair, that turns him into a one-man band, and a singer, born with one hand, who could now, for the first time in her life, play the guitar.
That event was DisCoTech, an a-first-of-its-kind hackathon, dedicated to the creation of music technology, for people with special needs. I often get asked, “What does that mean and why did you choose to do this of all things?” The truth is, it pretty much chose me. When I was 17, I spent five weeks in India with my father. This is not my father, by the way. (Laughter) He was just taking the picture. But at the time, I already had my first band, and we were recording and performing regularly back home in Tel Aviv. So I was constantly thinking about music. But it was only during a 30-hour train ride in India that something in me clicked.
You see, 30 hours is a very long time, especially in the age before smartphones. So all I could do was read my book, listen to my CDs, talk to my dad, sleep what felt like forever, and then find out we still had 10 more hours to go before we reached our destination. There was nothing to do but wait, just be. That was when I first started realizing what I was listening to all along. (Train sound) The train had its own rhythm, a mechanical beat. And my mind started focusing on the relationship between that beat, and the sound of dripping water from a leaking tap nearby.
What was previously an annoying background noise, started becoming a musical part in a symphony, the symphony of the present moment, and I realized – (Sound) everything is musical and music technology. I devoted the next 10 years of my life to music and film, I built my career and my art and my sense of self around these mediums, and I traveled the world with them as much as I could. But a decade after that first realization on the train, I started feeling like something was missing.
An inner voice kept telling me I could spend my time and energy doing something more meaningful, something that would directly affect other people’s lives. It was around that time that I discovered hackathons. Hackathons are events where people from different disciplines form groups and create together around a specific theme or technology. Hackathons usually last 24 or 48 hours straight. They include very little sleep and quite a lot of pizza, and they end in every group’s presentation of what they have made. For me, hackathons were the gateway to the world of music technology.
You see, this was the perfect context to take the “everything is musical” principle and apply it on a whole new level. So, I started making (Beatboxing) prototypes. (Laughter) Seriously, I started turning hats, and glasses, and gloves into instruments and converting head movements into sound effects, and heartbeats into rhythms, and brainwaves into melodies. I was getting hooked, and the lines were beginning to blur. Are we playing the instrument, or is the instrument playing us? Perhaps we ourselves are becoming an instrument, assigning sounds to the biological symphony that is our physical body.
And how could these ideas serve a deeper purpose? The answer came from the mind, literally, it came from the brain, as I started working with a technology called NeuroSteer, developed by professor Nathan Intrator and Lenny Ridel. What they were doing was to take the EEG readings of electrical activity in the brain and convert them into cognitive data, making them into numbers. And what I was doing was to take that cognitive data and convert that into musical values in music technology.
So, basically, translating electrical activity in the brain into melodies: On music technology, Musical Neurofeedback. I’m going to play you for a few seconds, so you can hear what it sounds like. (Music) The recording you are listening to is of Sefi on stage at an amazing event called TOM. Sefi is a paraplegic, in a wheelchair, and watching him on that stage performing music live for the first time since his injury opened my eyes to the realization that these technologies and these inventions could do much more than empower professional musicians and hobbyists.
They could help improve the lives of people with disabilities. They can make music accessible to everyone. A seed was planted. When some of these prototypes started winning awards and gaining recognition, I started receiving emails and calls from other people who were working on similar projects.
One of them was Erez Simon, a design student who was trying to build a musical instrument for a nine-year-old girl from Jerusalem. That girl is Raheli. Rachel is both physically and cognitively impaired. She can only move her right hand at will and can do almost nothing independently. Taking this into account, we tried to create an instrument and decided to make a glove that will act as a one-button machine, triggering an event whenever it hits a surface.
My challenge was to turn such a simple device into a musical toy that will enable Raheli to feel the joy of self-expression. So, I assigned three different modes to that glove. The first mode is drum mode, triggering a kick sound (Kick sound) followed by a snare sound (Snare sound) in a one-two-one-two pattern so that she could play like this. (Imitating kick and snare sounds) Now, Raheli could try and become a drummer.
For the second mode, I divided Raheli’s favorite song into sections, so that she could queue in the next part in time. Now, Raheli was becoming a DJ. For the third model, I invited Raheli’s mother to my studio and recorded her reading two different children’s books.
I then cut those readings into sentences, so that Raheli could control the pacing and overall experience while keeping her mother close by even when she wasn’t. Erez made an amazing job designing and creating the glove itself and the project was a success, not only at school but also for Raheli and her family, which brings me to the present day and the beginning of this talk.
About two weeks ago, I had the privilege of producing a hackathon myself: DisCoTech, the first event of its kind dedicated to music technology for special needs. This was produced in collaboration with an Israeli NPO called “Imagine”, founded with the mission of making music accessible to the special needs population, and my own company “Shift” which focuses on innovation for positive impact. Together, we worked for four months to assemble teams of designers, developers, and makers around four different musicians and four different challenges.
You may also like:
- Sirius XM Satellite radio for auto sound system
- CD duplication versus replication
(Video and music starts) Ofer. He is suffering from a rare neurological disorder similar to ALS. He is also a talented electronic producer. His team built a unique interface with a camera that translates his eye movements to music. Roy is a gifted pianist, autistic and blind. He can play virtuously with his right hand but is limited to using one finger on his left hand.
His team built a pedalboard and an algorithm that compensate for this motoric limitation by adding in the notes that he can’t play himself, creating a fuller, richer sound on the keys. Liron is a brilliant professional filmmaker and musician. He was injured in a ski accident and became paralyzed from the chest down. His team turned his wheelchair into a portable studio with a guitar, an harmonica, a microphone, and a built-in looper, turning him into a one-man band in a wheelchair.
And last but not least, Kineret, a singer born with one hand, but with energy and talent to spare. Her team designed a 3D printed prosthetic arm that allowed her for the first time in her life to accompany her own vocals without needing to depend on anyone else by playing the guitar. (Video and music ends) (Applause) Thank you. (Applause) Thank you.
At DisCoTech’s closing event, all four prototypes were demonstrated live, making it clear that music technology has the power and potential to change lives. For me, this journey into the future of music has become about more than technology or special needs. It has become about our basic rights as human beings to express ourselves freely and fully, regardless of any limitation. Thank you very much.