Exploring Themes and Bias in Art using Machine Learning Image Analysis.

Researchers

Journal

Modalities

Models

Abstract

Machine learning and computer vision have been applied for image recognition of art objects such as paintings, sculpture images etc. In particular, deep learning methods for image classification in art have been used to improve user engagement by providing access to accurately labelled and classified art objects. As an increasing number of notable museums turn towards creating open access collections, alternatives to the use of laborious human annotating methods are needed. This paper focuses on the Open Access initiative of The Metropolitan Museum of Art (The Met) which was launched in 2017 in an effort to expand The Met’s reach and presence. The museum now provides a select dataset of information on more than 470,000 artworks in its collection for unrestricted commercial and noncommercial use. However, with a widely accessible collection, the Met now faces the problem of how to enhance the user experience via access to accurately labelled art. This paper focuses on machine learning methods with applicability to automated classification of images obtained from The Met’s online collection. We aimed to: 1) Compare three different convolutional neural networks ResNet 50, ResNet 101, and Inception-ResNet-V2 using human annotated data, 2) Add transparency and interpretability to our models by using Gradient-weighted Class Activation Maps (Grad-CAMs) and to explore bias in gender labels and 3) Implement a multi-label classification model using ResNet 50. Future work would include the use of unsupervised clustering methods/auto-encoders to explore additional themes in the data. Other extensions of this work would include exploring methods to implement fine grained visual categorization, to mitigate bias, and to address the limitations associated with culture and stylistic interpretations. Deep learning techniques for art image classification may also help detect consistent features of bias in human annotated art.