Could Machine Learning Become a New Instrument for Musicians?

(T) When I am listening to Bach’s Brandenburg Concertos, I have always been fascinated by how much I have the wrong certitude of listening to a concerto for violins from Vivaldi. No doubt that Bach learned a lot from Vivaldi, and so did Mozart from Haydn. In 2014, research scientist Sander Dieleman analyzed those similarities in the audio signals to develop a recommendation system for Spotify listeners.

In 2017, guitarist and music deep learning pioneer François Pachet led “Hello World“, the first music album composed with the help of neural networks. This paper from Mr. Pachet (and two other researchers), and this GitHub repo provide a deep dive into the initial deep learning techniques and projects to generate music.

Like any other fields in machine learning, GANs and transformers have more recently created huge opportunities to provide better tools to musicians and artists (but note that you can be contrarian and also make good (pop) music with a simple probabilistic model)

GANs such as Google’s GANSynth have been used to synthesize high-fidelity audio, and transformer-based architectures such as OpenAI’s GPT-2 have been used in MuseNet to imagine how Chopin would have composed Mozart’s Rondo alla Turca (OpenAI released after MuseNet, Jukebox which learns from raw audio, while MuseNet learns from MIDI data).

NeurIPS 2020 had a workshop on “Machine Learning for Creativity and Design” where many papers were presented. In particular, I found the presentation from Jesse Engel from the Google’s Magenta project part of that workshop quite interesting.

Following is another Jesse’s presentation of Project Magenta at Google I/0 2019:

In a recent blog article, the Google Magenta team investigated the challenges encountered by (computer science) musicians in their quest for songwriting with machine learning tools:

“We found that teams faced three common challenges when co-composing with AI because AI was not easily decomposable (not easy to tweak individual musical components), was not context-aware (not fully aware of the musical context it was generating for), and also not easily steerable (not easy to request for the music to bear a certain mood or effect). Rather than using one giant end-to-end model, teams often used a combination of smaller, independent models that aligned well with the building blocks in music (i.e. melodies, chords, etc), and then stitched those smaller pieces together by hand. Teams often generated hundreds of samples and then manually curated them post-hoc, or used another algorithm to filter and rank the model outputs.”

So could machine learning becomes a new instrument for musicians? It could! But we are not there yet. We have not found the “killer app”, and we need to make the instrument widely available for non-computer scientist musicians.

Note: The picture above is the first non-online and last non-online concert for the academic year of 2019-2020 of the Stanford Wind Ensemble playing the music of Giovanni Gabrieli, Frank Ericson, and Gordon Jacob in the outdoor patio of the Bing Concert Hall on June 2nd.

Copyright © 2005-2021 by Serge-Paul Carrasco. All rights reserved.
Contact Us: asvinsider at gmail dot com