Last.fm is a service which tracks (scrobbles) the music you listen to no matter where it is stored or how you listen to it (local files, streaming serivces, YouTube, whatever else). It is also a social network of sorts and you can connect with other people and see what they’re listening to, but I’m there primarily because I love to have my music and listening habits gathered, summarized and quantified. My twin calls it OCD, I prefer to think of it as curiosity. Anyhow, just recently, it was her who was showing me some visualizations of her Last.fm library which were pretty cool. She offered to run those same visualizations against my library, too and I was baffled to find out that the data export (which was done via widely spread lastexport.py script whose origin is unknown to me) took forty-five minutes! That’s duration of an entire halftime of a football match, mind you. Surely, it could go faster than that?
Lets do some quick maths. Last.fm API imposes rate limit of maximum 5 requests per originating IP address per second, averaged over a 5 minute period. This means that the best execution time we can hope to achieve for retrieval of some 80K scrobbles by making 1600 requests (and if we assume a single request takes about half a second) is 160 seconds, aka just over two and a half minutes. There, we can go from duration of a single halftime to a duration of only the added time at the end of it!
Unnecessary football references aside, and in all seriousness, why not to do it? Since I’ve stated using Go recently, this was my chance to utilize the awesomeness that are goroutines to speed things up. Goroutines, in brief, can be thought of as lightweight threads. They are functions that run concurrently with other functions and they are very cheap, compared to threads. That’s why it is common for Go applications to have thousands of goroutines running concurrently.
I’ve re-implemented this Last.fm data export in Go, spinning up a goroutine for each HTTP request. I’m doing this in batches of 30 requests, in order to prevent getting rate-limited too often. The result: execution time does indeed go down from forty-five to three minutes. It is not the ideal two and a half minutes we caclulated, and I attribute that to the nature of network where some requests will take longer than half a second and the fact that some reqests are inevitably bound to get blocked and will then have to be resent. All in all - export now works way better than it used to.
I’ve also also expanded functionality a bit to enable export of your loved tracks, artists you’ve listened to and tags of your music. You can download the executable from GitHub, or build it for your desired platform directly from source with go build. Run it simply by passing your Last.fm API key (which you can get here), username and flags specifying which data to fetch. It will then export your data to csv files in your working directory.
lastfmuserexport.exe -key=<LASTFM-API-KEY> -user=<LASTFM-USERNAME> -[scrobbles|loved|artists|tags]
For (slightly) more detailed usage explanation, run
lastfmuserexport.exe -help