Has anybody experiences using mediapipe (or similar) to create an ‘air keyboard’ and send OSC messages to SC? Sending OSC messages to SC from eg. python is pretty trivial, what I am mostly interested in for now is the ability to reliably produce midinotes (or information translated as such) with hand gestures, so that it is possible to actually play a preconceived melody. @jordan - I saw an older post where you are praising mediapipe, I was wondering if you have any (new) experiences using mediapipe in this way?
I only did the head/face tracking, it worked very well for my use case. Here’s what I learnt…
Use a good webcam.
Make sure your scene is well lit with good contrast between the background and your object.
Writing a custom GPU thing in c++ is: one, a pain; two, has terrible latency. Use the python code, it uses more CPU, but has better latency.
Using something like pycharm to download all the packages was easiest more me, and makes it a one clock solution.
You access the individual joints/anchors of the body, so you will need to write some heuristics to actually recognise the gesture/interaction.
You can do actual gesture recognition on mediapipe, but you need to train it - I have no experience of this.
Good luck!
1 Like
Oh, if you didn’t know, you get the coordinates of each joint in 3d space - I believe the camera is at 0,0,0. This would make placing a virtual keyboard in space pretty trivial.
Yes, I am working together with a grad student who is trying to implement different designs. I theory everything is trivial but in reality being able to faithfully reproduce a desired pitch with proper noteons and noteoffs is more tricky because of various types of signal noise similar to what we experience in DSP (smoothing control data etc). I myself have no experience yet with either Python or mediapipe but I might have go get my feet wet to help the guy.
The aim is to build an interface which enables people who suffered a stroke to play my Thormulator SC app and thus train their weak side (typically only one side of the body is severely affected by the stroke) in an inspiring way. More about thormulator here:.
The weak arm could eg. be used to set the velocity of notes triggered by the strong arm/hand or for filtersweeps or other cc-like stuff. The important thing (from a medical point of view) is the training of the arm (bicep curls etc) and the hope is that a performance system (like Thormulator) can inspire and motivate the patients to actually get it done - just telling the patients to exercise proves to be quite difficult - they fall of the wagon quickly. Similar attempts have been tried with specialized video games, but apparently they quickly become too boring for the patients.