The main source of inspiration for PhonHarp is the desire to create a hybrid musical instrument that embodies digital technology (such as a software running on a smartphone), analog electroacoustic devices (that is the phone loudspeaker), and physical components (such as the musician's mouth), thus breaking the boundaries between analog, digital and phisical instruments.

From a technological point of view, this work is similar to a talk-box, a pedal effect born in the ‘60s. The goal of the talk-box is to let the performer modify the sound of a musical instrument with her mouth, thanks to a plastic tube that connects the instrument to the oral cavity.
While the tube injects sound in the oral cavity, the musician can mimic words with her mouth, without actually pronouncing them. In this way, simply using the vocal tract, the spectral composition of the signal is modified, giving the audience the illusion of listening to a “speaking instrument” (this is basically a physical version of the Vocoder).

To turn the smartphone into something similar, we designed PhonHarp.

The goal of PhonHarp is letting users transform their smartphone into a hybrid musical instrument simply by installing a piece of software, with no need for external devices. Then they should be able to shape the sound through their vocal tract.

To this end, two key aspects must be considered: first, the smartphone must be held like a flute in order to bring the loudspeaker as close as possible to the mouth; secondly, the timbre has to present spectral richness in order to support articulated effects.

Concerning interaction and playability, we opted for a monophonic instrument, with the possibility to perform melodic lines spanning across at least two octaves, playing all the notes of the chromatic scale.

Both left- and right-handed users are considered.

The designed interface presents a keyboard consisting of five buttons, indicated by white circles in Figure.

To handle the instrument users must rotate the smartphone bringing the speaker towards the mouth. You can see here two alternative ways to hold the phone.

Although PhonHarp may recall a flute, the relationship between the fingering and the played notes makes it more like a trumpet.
To identify the note triggered, we use a 4-bit binary encoding of numbers in the interval 1 to 15. These notes are mapped onto a diatonic musical scale, thus determining an extension of the instrument equal to 2 octaves. During the configuration phase, it is possible to specify the type of scale, choosing between major and natural minor, the first grade, and the reference octave

Concerning the sound production, we choose to let the user manage and customize a list of available timbres.
In particular we provided control over: harmonic content, vibrato, tremolo, equalization, portamento, volume, and envelopes.
Many of these parameters can also be dynamically adjusted during the performance, thus making the instrument more expressive.
Finally, a feature for sharing timbre settings with user contacts is provided via standard Android sharing practices (long-press the timbre name in the timbre list).

The core of the app is the software synthesizer.
It consists in a semi-normalized subtractive architecture, composed of a variable pulse-width square-wave oscillator, followed by an amplitude-modulation block, and a simple equalizer.
Typically subtractive synthesizers also carry a filter module, but, in this case, the filter is provided by the mouth of the musician, thus involving a physical process in the sound-generation chain.
Using a variable PW oscillator allows the user to choose in a simple way among odd-only harmonics, typical of wind instruments; all harmonics, typical of plucked string instruments; and a PW-modulated sound, similar to bowed string instruments.

You can check our paper at Audiomostly 2021 for further details.