Leave the Guitar, Smash the Laptop: The Story of Music-Making AI

Music-making artificial intelligence isn't that great... yet. Even though these AIs are limited and are most often used to produce background tracks for commercials, that hasn't stopped a dedicated group of technophiles from using them to produce everything from riffs to entire songs. We're still in the early days, and only a handful of artists are using AI, but as the programs improve, it's only a matter of time before computers produce entire tracks that are indistinguishable from what a real artist is capable of

Leave the Guitar, Smash the Laptop: The Story of Music-Making AI

Ten years ago people asked: What can artificial intelligence even do? Today, the better question might be: What can’t AI do? The answer to that may be surprising. Even though AI has excelled at difficult tasks — e.g., beating Chinese champion Ke Jie at the game of go — there are a few skills that not even the smartest computer has yet been able to master. This is the story of AI and music. A review of technology’s influence on what we listen to, a critical look at AI’s current role in music production, and a prediction for the future. The year is 2066, do we have an AI Hendrix? 

A brief history of technology changing music  

On a fine July evening in the year of 1965, Bob Dylan brought to the stage not his usual acoustic guitar, but a Fender Stratocaster. The fans, it can be said with only mild exaggeration, went nuts. People cried, booed and cheered as the face of folk music plugged in and electrified the crowd. In a single night, one man broke with tradition and introduced a new style of playing music. Several decades later, autotune was introduced. 

Cher, a singer of international note, used autotune on her 1998 song, “Believe.” In that song, we hear for the first time the musical style that would become synonymous with pop in the 21st century. For better or for worse, wires and transistors had again reshaped the sound of music. Not everyone was pleased. 

But there is one particularly sinister invention that has been putting extra shine on pop vocals since the 1990s. It is a piece of software called Autotune. Essentially, it takes a poorly sung note and transposes it, placing it dead centre of where it was meant to be.

Neil McCormick, Music Critic

One can feel his disdain for any person unrefined enough to consider autotune a good idea. What happened to natural talent? Where did the real musicians go? Neil may wish he was born half a century earlier, until a band called BTS was invented. 

Yet, here we are — so accustomed to technologically enhanced music that nobody gives it a second thought. Entire songs are produced on computers, voices corrected so that they’re unrecognizable, shower singers shot to stardom in the age of filters and synthesizers. Computers have been sitting co-pilot in music production since the turn of the century, so is it surprising that the next logical step is for AI to try its hand at flying plane? 

How AI is making music today, sort of… 

Technophobes ought not to lose sleep over the state of the AI-generated music industry. It’s young, plagued with issues and really only works when humans have a heavy-handed final edit. Even the best artificial intelligence systems are not yet capable of creating full length, catchy songs that people actually want to listen to. Yet, within its domain of expertise, AI is doing some pretty cool stuff. 

Amper is a company at the forefront of AI’s bid for dominance in the musical arts. Founded in 2014, its sale’s pitch was simple: Use our program to help you create music. You choose the genre, length of the song and any number of other small details. Our AI will compose a completely original piece of music for your use. Amper’s production quality was nothing to turn up the radio for, but in a limited number of circumstances, it proved beneficial. 

A singer who does not know how to compose a melody, for instance, could use Amper to create a background track. She could tweak a number of variables to get the sound she desired, maybe send it to a sound engineer for the final tuning, and then release a song without ever having worked with a band. 

In this example Amper’s artificial intelligence really did compose something, a completely original computer creation. Yet, almost never is that product good enough to stand on its own. It only works because of the heavy editing and creative input that happens afterward. Amper fizzled. It was used here and there — notably in one case, which we’ll discuss later — but as a whole, it did not offer a groundbreaking solution that bands were clamoring to use. Then the company pivoted… 

The commercial use of music-making AI

Amper Score, an enterprise program designed to, among other things, produce background music for commercials. Featuring a foolproof design, a boardroom suit just has to pick a genre of music and the mood that he would like the song to be composed in. Then, Amper Score builds the track from scratch. The great composer listens, changes the tempo, swaps the instruments, speeds it up or slows it down. Then, it’s off to production, ready to be incorporated into whatever is being produced that day. In this context, Amper proves invaluable, as it allows people with no musical instinct to produce entire tracks. Here’s how it works. 

To create a song, Amper looks at a body of music within a certain genre and takes note of which instruments are used, the tempo, melodic patterns, commonly used keys and so forth. Amper’s creators even described the program as being able to read the mood of a piece. Whether that’s true or it’s just marketing speak, Amper analyzes the data and uses patterns to create a new song, similar to others in the genre. Boom, presto, creation! 

Music without the hassle seems to be a good selling point, if Amper’s $4 million seed funding in 2018 is any indication. While Amper Score is only available at an enterprise level, before the company cut off its individual users, there was one person in particular who took advantage of it.  

Those who compose with computers 

Singer and songwriter Taryn Southern can’t play the piano — at least, not well enough to produce a rich melody. Traditionally she might have partnered with a musician to help her make an album, but Southern felt that perhaps tradition wasn’t the best choice. She chose instead to work with Amper in order to create the instrumentals for her song, “Break Free.” The music-making AI did all of the work, crafting chords and harmonies, and then Southern tweaked it to her taste. The result was a perfectly adequate, if somewhat flavorless, background track. 

When asked about the experience of working with an AI-music program, Southern responded, “I find that really fun, and because I’m able to iterate with the music and give it feedback and parameters and edit as many times as I need, it still feels like it’s mine in a sense.” She went on to claim that Amper made the process of writing a song 20 times faster. Presumably, for those well-versed in music theory, the program might not be nearly as useful, but for someone like Taryn, it was a great solution. So great, in fact, that she decided to employ several different music-making AIs to compose every background track on her entire album

Making use of not just Amper (before the company pivoted to corporate only), she also worked with Google’s Magenta and IBM’s Watson Beat. Reflecting on whether she could properly claim credit for songs that were in no small part made possible by artificial intelligence, Southern stated, “Because I’m able to iterate with the music and give it feedback and parameters and edit as many times as I need, it still feels like it’s mine in a sense.”

That the music is its own, despite an AI aiding and abetting, is a claim Yacht would also be likely to make. The Los Angeles-based dance-pop group has been around since 2002, but that didn’t stop it from experimenting with decidedly newer technology. “(Downtown) Dancing,” the lead single on the band’s new album, Chain Tripping, is largely the work of AI. While the group retained some creative control over the final product, a so-called “neural network” crafted the melodies, wrote the lyrics and contributed in a dozen other small ways to the production. Here’s how. 

First, the band took 82 of its past songs and chopped them up into short segments that could be read by a computer. They fed those into Magenta’s MusicVAE. which analyzed Yacht’s samples and, based on their unique sound, created an entirely new melody. The band then took those suggestions from the AI and put them together to create a coherent sound — as a full composition was beyond the means of the computer. 

Claire L. Evans, Yacht’s lead singer, confirmed this when she told a reporter from Spin magazine, “There’s a lot of tools that can help you generate fragmentary things like melodies or text, but we’re not quite at a point now with machine learning where it can help us make structured pop songs.” 

In another interview, this time with music blog Brooklyn Vegan, she went on to say that the benefit of working with AI wasn’t the sound, but it was the interactions that it fostered. “It’s not so much about collaborating with the machine as it is about collaborating with people through the medium of the machine. Creating this kind of work requires that creative and technical people work together, and that’s a good thing in our increasingly siloed culture.” 

Interestingly, although the composition of “(Downtown) Dancing” was heavily influenced by artificial intelligence, there is no dead giveaway that this is the case. It sounds like a normal indie dance song, and people familiar with the group’s work say that it is in keeping with Yacht’s style. The real headlines, in other words, are the headlines themselves, not the actual product. 

When computers can sing 

Although Yacht drew the headlines, it’s not the only group experimenting with computer generated vocals. Amnesia Scanner, a Finnish group based in Berlin, has created a virtual vocalist that the band has named “Oracle.” The voice is generated by what Amnesia Scanner refers to as, “a stack of software.” Interestingly, the group has gone the opposite direction of other musicians working with AI, and by producing the background tracks and tasking a computer with the vocals, the group has helped to show what the potential of a digitally led music revolution could be. 

Yet, it’s worth noting that while Oracle sounds like it could refer to the AI from “2001 a Space Odyssey,” it isn’t actually a true AI. Instead, it’s a finely tuned audio program that gives Amnesia Scanner a chance to tweak vocals in a very specific way. This can be done consistently, and the group hopes that in time, people will come to know this trademark sound as one associated with Amnesia Scanner. For those interested, the entire album is available on Soundcloud, including “As Spectacult,” featuring Oracle. 

Ash Koosha is another musician using a computer-powered vocalist. For the last couple of years, he has spent his time creating Yona, an artificial singer. Speaking on the work required to create such a program, Koosha stated in a Dazed Digital interview: “I think Yona will become the ultimate musical computational intelligence or knowledge base. Over the last year or so I have made more than 100 trials, but very recently Yona’s output started sounding like music, and Yona’s voice is shaping in a way that creates emotional impact.”

The result is an intriguing sound, one that would be difficult to label as artificial on the first listen. It’s not that it sounds natural — far from it, actually — but so many singers tune their voices toward the robotic that Yona ends up sounding eerily similar to a heavily distorted pop singer. Later in the interview, Koosha spoke on the mechanics behind the artificial songstress, “Yona’s engine is comprised of a series of generative software that generates sentences, melodies and singing via a complex text-to-speech process.” 

Koosha claims that although he does the mixing and final edit, the majority of each song is inspired by chords, melodies and lyrics created by Yona. To do so, Yona derives her creativity from Koosha, as the music-making program analyzes a backlog of his music in order to create a new sound. However, as the technology advances, it’s hard to say what Yona could be capable of. Koosha is hoping for the best, saying, “She performs the music, and as she improves, her unique voice is on the path to becoming a full electronic pop musician.”

Anti-chart topping AI

Daddy’s Car” was one of the first songs to have ever been created by AI. It was produced in a more primitive time, back when man led a simpler, more natural life: 2016. To create the song, the producers used Flow Machines, a program that’s still around today. While the AI was responsible for crafting the score, as we’ve seen before, it took a talented human — in this case, the French composer Benoît Carré — to bring the pieces together into a melodic whole. 

The song is intended to be stylistically similar to the Beatles, a goal that it accomplishes to some extent. However, the most interesting aspect of this composition is that “Daddy’s Car” doesn’t really sound all that much better or worse than other, more recent AI compositions like “Break Free” or “(Downtown) Dancing.” Three years in the world of silicon and bits is like a decade in the real world, so for there to be little noticeable progress in the field of music-making AI is a good indication of how slowly the technology is advancing.  

What we have not seen yet is the killer app — the use case that only an AI can fulfill. Aside from a few avant-garde artists, the music-making AI of today is most widely used to aid in the creation of background tracks. For that small subset of the population that has a use for this, music-making AI could be a godsend. It can reduce costs and speed up production times. That’s fine and good, but the AI is not doing something a human can’t do. It’s not creating a song completely unique to AI, a sound that is impossible to get otherwise. 

In researching this article, what became clear is that music-making AI is not on a significant growth curve. Look at the number of people who own smartphones, the number of Bitcoin wallets or the growth in Instagram users. These are technologies that, when viewed from an annual perspective, are clearly growing. When we look at music-making AI, there appears to be little progress. Every year, a couple of bands experiment with it, some articles get written and concert tickets are sold, but nothing phenomenal is taking shape. There is, at this point, a lack of brilliance in the field of computer-generated music. 

Welcome to the world of tomorrow 

Having said that, there are some people who believe that regardless of what’s happening today, AI’s future influence is going to be tremendous. “It may be as big a deal as the original shift from people making music primarily with their own bodies to crafting instruments that had their own unique properties and sounds,”communications manager at Google Jason Freidenfelds predicted, believing that AI’s impact in music will exceed any one technological advancement. 

If he’s right, music-making artificial intelligence may move music in a direction that no human could have conceived of. As autotune is ubiquitous today, so may AI’s influence on music be taken for granted in the coming decades. The results could be surprisingly beautiful, an unexpected combination of digital and analog giving us music greater than the sum of its parts. 

But it could turn out differently… Maybe, just maybe, music will prove to be a unicorn, the lost city of Atlantis, the ineffable art that humans remain outdone by emotionless thinking machines. AI has conquered chess and poker, cancer diagnosis and insurance claims settlement. There is no human at the controls when SpaceX’s rockets touch down. Every year, AI takes on more. Yet, if there is any hope for humanity to hold its edge, that hope lies in music. Emotion-driven, soul-gripping music. It could be that in 20 years, people will tuck themselves into their robot taxis, turn on their earphone implants and listen to something a human made — a soulful sound no computer can replicate.