Intro

Hello!
In this post I plan on talking about how I receive artwork data and store it. Speed is paramount when creating successful websites, and making multiple API calls can really slow down a website. Since Museum Passport searches multiple Museum APIs, optimization is key to create a positive user experience. I will explain my strategies for making Museum Passport feel as fast as possible, and how I hope to make it even faster in the future.

Go Concurrency

One of the main reasons I chose Go for Museum Passport was its great concurrency features. One of the newer concurrency features of Go is the errgroup. Errgroup is basically a high level way of running multiple goroutines at once, rather than making channels to handle errors can get hard to manage. Instead, errgroup cleanly handles possible errors for you. Although errgroup returns only the first error it receives, it ends up being far cleaner than standard channels.

artworks := make([]*models.SingleArtwork, len(currentSearch))
g := new(errgroup.Group)
g.SetLimit(10)
for i, id := range searchIDs {
g.Go(func() error {
artwork, err := artworkAPICall(id)
if err != nil {
return err
}
artworks[i] = artwork
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}

*NOTE: Go 1.22 fixed loop variable capture issues*

The very first line of code involves preallocating an empty slice to store the returned data. This is a safe way of allowing multiple goroutines to write to the same slice. There is no fear of resizing issues when each goroutine is preassigned its own spot in the slice.

Another great feature of errgroup is the ability to easily set the maximum number of goroutines occurring at once. Without SetLimit, Go could theoretically start running a hundred API calls at once, which would probably not end well! SetLimit(10) ensures at most 10 concurrent calls are made, preventing you from overwhelming the API.

Caching Artwork

Making concurrent API calls is great, but would it be efficient to make the same calls repeatedly? That is where caching comes in. I decided on first focusing on using local memory for artwork data, as it is relatively low in size. The main downside of storing in local memory though is that when the application is restarted, all that saved data is lost.

artwork, exists := m.Cache.GetArtwork(fmt.Sprintf("met-%d", id))
if exists {
return &artwork, nil
}

Before deciding to make a call to a museum’s API, the function first checks to see if the artwork already exists in the cache, therefore eliminating an API call. Say you’re searching for artwork by Van Gogh, and 3 of the 10 returned IDs are already stored in the cache, only 7 API calls are necessary to return the needed data, speeding up the search experience.

When working with a relatively small cache (50 artworks at roughly 16KB total is negligible), local memory works well. However, I plan on implementing Redis or a database in the future to handle cache persistence across restarts and create a more production-scale application.

Conclusion

By combining Go’s errgroup for concurrent API calls with in-memory caching, Museum Passport can serve data to users quickly without overwhelming external APIs. These optimizations create a responsive search experience while using APIs responsibly.
Thank you for reading!