Music Curation with ML

Desktop HD.jpg

Music curation with Machine learning

Thesis, MFA of Interaction Design
Individual Work, Academic
In Collaboration with Soundtrack Your Brand
Umeå Institute of Design
Feb - June, 2019 | 18 Weeks

Exploring ways for Machine Learning to work with music curators to find new perspectives of understanding music and detailing the current music classification models.With background music curation as an intervention point, this project explores ways for human raters to collaborate with machine learning to augments human intelligence.

Guides: Niklas Andersson, Patrik Axelsson
My Role: Research, Concept Ideation, Visual Design, Testing



Go Through the Project with Zhi in 3 Minutes



My Beginner’s Guide of Designing for Machine Learning Raters

Machine Learning Raters are those who train, teach and validate for machine learning applied products. With music curation as the intervention point, I summarized my own beginner’s guide of designing for machine learning raters.



Digital Music Curators and Their Current Workflow

With the emerging of digital music streaming and distribution services, a new profession comes into being - Digital Music Curators - those who create “Romantic Dinner” playlist on Spotify or the music experience for McDonald's all over the world.

Group 8.png


Conflicts between Human and Music

Group 25.jpg

Conflict 1

Sheer Amount of Music vs. Limited Human Power

“Facing the sheer amount of music makes me not feeling close to music anymore.
It just feels very industrial - Where's the human behind this? Am I just a machine doing this?”
—— Jenny, Music Curator

Group 24.jpg

Conflict 2

The Fluidity of Music vs. The Solidity of Music Decisions

“Sometimes we organized music in a certain way and then we changed our mind.
But since we have already tagged 70,000 songs, it is just impossible for us to go back and clean up all the music.”
—— Amanda, Music Curator

Group 23.jpg

Conflict 3

Subjective vs. Objective

“It is hard to bring the team on the same page because every person has his own way of understanding music.
In stead of saying ‘Guys, we need 80s Rock’, there are normally tons of meetings to define what exactly we are talking about.”
—— Stefan, Music Curator



Project Positioning in Intelligence Levels and Learning Types

With the goals of “Finding new ways of understanding music” and “Detailing the current music classification model” in mind, me and the machine learning engineers from Soundtrack Your Brand identified Unsupervised Learning as the suitable learning type of the project. Based on our understanding of current progress of Artificial Intelligence, we positioned the project as Augmented Intelligence.
In short, Machine Learning would act like an exoskeleton to human for music curators in this project.

Group 8.jpg


The Mechanism - How it works

This project will be using Clustering technique within the unsupervised machine learning field.
Clustering is a technique of grouping data into clusters based on their similarity. Since it will figure out relationships of different attributes in a data set without any predefined input, it would be able to “Find new dimensions of looking at music” and “Detail the current music classification model”.
However, the decision making process of the algorithm is in a black box, we, as human, can not really understand the true reasons why the algorithm thinks these songs sound similar. That is why we need human music expert to review, valid, identify and fine-tune the clusters, for example, deciding if the cluster has one or more universal themes, e.g., Christmas music, happy songs or R&B. 

Group 33.jpg


Challenges in the New Way of Working

Group 35.jpg

Challenge 1

HMW Identify Meanings in the Unknown?

The current workflow of music curation is very straight forward. The music curators always have a clear goals in their mind.
”What feeling does he want the music to be? “
“What genre does he want to work on?”
“What kind of intensity does he want the music to be?”

However,  with the new work model , the music curators will be presented clusters of music without any idea of what this cluster is.
It is like shooting in the dark or traveling without a direction, which leads to the first challenge of the new work model: How might we identify meaning in the unknown clusters?

Examination presentation.001.jpeg

Above: Herrada, O.C., 2009. Music recommendation and discovery in the long tail, Doctoral dissertation, Universitat Pompeu Fabra


Challenge 2

HMW Manage the Music Clusters with Different Objective/Subjective Levels?

The notion of “Making Sense” can have very different subjective/objective levels in music.
With the clustering algorithm, there could be clusters of all kinds.
Theoretically, there can be a music cluster with cat meowing in the background.
How are we going to manage, communicate and make use of all the clusters and tags we are getting?



Co-Creation and Testings

With challenges I found from the research on music curators and machine learning, I did a co-creation workshop and multiple early concept testings with music curators, machine learning engineers and the playlist consumers. I was able to gather ideas and validate them through the real users.



A Full Workflow of Applying Machine Learning in Music Curation

The final outcome of this project is a full workflow of applying Machine Learning in music curation, which are communicated in the format of wireframes and a click through prototype.
Click HERE to get the high resolution of the wireframes.




As part of the final deliverables, the project was presented in the UID 19 Degree Exhibition. The Exhibition was separated into two stations. 

The first station was to communicate the background of the project, including the four design principles of designing for human raters of machine learning algorithms and how these were realized in the context of music curation. Two posters and a click through video were presented in this part.


The second station was to invite the audience to experience life as a music curator.
In this station, the audience could participate in crowd-curating the music experience for UID19 by adding one to two songs in a collaborative Spotify playlist. By going through how diverse music was added to the playlist, we can again see how different people understand music as well as a certain event. Click HERE to join the Crowd-Curated Experience.
The audience could also go through the simple version of the tagging flow by listening to a machine learning generated playlist and answer a printed questionnaire. The questionnaires were collected and used as the research material of how we understand music. Click HERE to experience a music cluster generated by Machine Learning.



Read More…