20th November 2025
MusiKraken Case Study – The jaw flute
Occasionally we’ll have to find solutions for students with severe physical disabilities.
I visited a recent music session at one of our schools, specifically to see if we could set up an instrument for a student (Robin*) with extremely limited movement – just able to move their head and mouth.
MusiKraken is a general purpose musical instrument – able to remap any of the iPad’s sensors and many other inputs to make musical output. We decided to try using it to make a quick and easy to use instrument!
After establishing Robin’s comfortable ranges of motion, I decided to use the ARKit Face tracking features to track the position of their jaw – left to right. In Musikraken, this simple setup looks like this:
In simple terms, we are just mapping the “Jaw Sideways” output to a musical note scale, and sending it out to another piece of software, ThumbJam, to actually play the instrument (as it has better synthesis options). We tried a flute sound, as it felt it worked well in terms of response and feel. This worked great! After only a few minutes of setup and testing, Robin was able to play and express themselves.
However – Robin was not able to choose when to play. A note was always playing – and it seemed like a good idea to give Robin some agency over when not to play. With mouth control already being established, we set the “Mouth Open” parameter to allow a note to play or not, which in MusiKraken looks something like this:
Clearly, this looks a little more complicated – but it’s still straightforward! You can still see the output line from “Jaw Sideways” is being mapped in the same way, but this time is fed into a “Gate” block, rather than straight to Thumbjam.
You can also see a line from “Mouth Open” to another block – the “Threshold” block. This will play a note when the value is over a certain level. This allows us to fine tune how much Robin’s mouth needs to be open to play a note.
This note output is then used to “allow” the main jaw output through the gate – and we’re done.
Robin was able to play along with others, solo, and also stop playing between jams or allow others to solo.
It does need some tweaks though – when Robin tries to communicate normally the flute would play too. Perhaps we’ll change the mouth open trigger for something else – an eyebrow raise, eye position, or even using the ipad touchscreen to trigger a note. Either way – something that doesn’t affect their other abilities.
*Name changed
Other OmniMusic News
View News Archive
