A few weeks ago, I achieved one of my dreams: being on a team that publishes a paper from doing something cool. Here it is in full (shoutout NIME for being open-access):
You can see my cameo as a hand model in Figure 5.
Link to this section Our Vision
The project was pitched to me by my advisor roughly like this:
- Boards like the Raspberry Pi, Bela, etc. are excellent tools that enable people to learn audio programming and make creative instruments using sensor data.
- These boards can be expensive, and are especially difficult to find in certain parts of the world like Latin America.
- A commodity smartphone, even an older one, has plenty of sensors, processing power, and audio capabilities to support this use case.
- Old smartphones, specifically Android ones, are quite readily available in most parts of the world.
- We just need to find a way to get a software environment low-latency audio I/O on these devices and package it into something easy to use.
Link to this section My Work
Our approach was to circumvent the JVM entirely by rooting the phones, building binaries for them on a host computer, and executing these binaries directly. This gave us more direct access to the hardware, at the cost of a more difficult setup process. I focused on two main goals: our sensor I/O and our build system.
Link to this section Sensor I/O
This work involved working with libandroid, Android's internal library used for handling device functions. I spent a ton of time in Android Code Search to investigate how everything fits together, and started to put together some test functions for adjusting sensor sampling rates and reading data, which my advisor then turned into LDSP's user-facing sensor API, providing functions like sensorRead -- taking heavy inspiration from the Arduino IDE's simple functions like analogRead.
Link to this section Build System
Our build system draws heavily on the Android NDK, or Native Development Kit. This toolchain is designed to build smaller native components within a larger JVM-based app, so we needed to circumvent the intended way of doing things a bit.
Initially, we just used manually invoked the g++ binary included with the NDK to compile our executables. However, as the project grew and we began pulling in more and more libraries, it became clear another solution was needed. We decided to start with Makefiles, due to their popularity and simplicity. This worked for a while, and even let us include some other scripts in the file (e.g. we set up make push to push the binaries to the phone).
This solution worked well as we continued development, but started to stretch when we wanted to make our build system more user-friendly. We wanted to provide a configuration file for each phone we supported, so users would not need to configure compiler settings for their phone manually. Many phone model names contain spaces. Furthermore, we wanted to let users specify a path to their project, and compile some of their files alongside our files. Allowing user project names to contain spaces was a hard requirement for us.
As it turns out, Makefiles handle spaces extremely poorly -- not only do they have to be escaped differently in different places, but many functions treat a variable with spaces implicitly as a list, causing issues. Many make builtins simply could not handle spaces in filenames, so we were forced to use $(shell ...) and rely on shell commands (which aren't portable across operating systems). Our solution was still plagued with bugs.
Eventually, we gave up on pure Makefiles and I started to investigate other build systems. Ninja seemed promising, and has the backing of projects like Chromium and LLVM. We rolled out some CMake files across the repo, and set the output format to Ninja files, and... success!
Of course, CMake isn't necessarily built for running random commands in the way pure Makefiles are, so I created a few wrapper scripts: ldsp.sh for Linux and macOS and ldsp.bat for Windows. Each script allows configuring the project, building (with incremental builds), pushing the build to the phone, and running the build.
We got to see everything come together in an all-day workshop we did with many students from NU Sound. While the install process was a bit convoluted, most participants were able to get a toolchain up and running on their computer!
Link to this section Future Goals
The biggest hurdle with our current setup is complexity: participants need the Android NDK, CMake, Ninja, ADB (Android Debug Bridge), and more installed on their computer. If we could package everything into an easy-to-install application, we could make this project more accessible to those unfamiliar with the command line or programming in general, and make it much more fitting as an educational tool. Furthermore, if we could install this package onto the phone itself, we could remove the need for a host device altogether, making our work accessible to those without a laptop or desktop.
I've been working on cross-compiling LLVM to run natively on an Android host. Admittedly, I haven't had much luck so far, but it's still early days. Once we have this crucial portion working, we can pivot to developing a web UI which runs on the phone to support editing and running code from any connected device, then package everything up into an .apk for people to install.
Link to this section Thanks
I want to thank my advisor, Victor Zappi, for giving me the opportunity to work on this amazing project.