Quantifying Qualia in Art
Can Machine Learning Automate Judgment?
In our creative realm, where aesthetics and qualia converge, we're constantly driven by the pursuit of innovation and emotional resonance. Capturing the essence of human expression in our art is an elusive journey, where our subjective experiences breathe life into every creation. But hey, as artists, we're in luck! A powerful ally has emerged—deep learning algorithms.
Bridging Qualia and Algorithms
Algorithms, fueled by vast amounts of data and intricate neural networks, possess an uncanny ability to learn, adapt, and even conceptualize. They analyze patterns, recognize visual elements, and can now even predict our aesthetic preferences. It's like having a collaborative partner that opens up new artistic horizons and dares to explore the uncharted territories of creativity—taking what's subjective and giving it a tangible form.
In what follows, we dive headfirst into a world where art, technology, aesthetics, and qualia come together in a mesmerizing dance. In this article, we explore the potential of deep learning algorithms as our artistic comrades – capable of impacting our creative process and shaping our artistic judgment.
Exciting, isn't it?
So let’s get started. How can artists appropriate emerging technologies to facilitate the process of creating ‘better’ art? As an artist and an AI researcher myself, I find this to be an exhilarating intersection.
Where Aesthetics Meet Qualia
At the heart of our artistic creations lies the elusive concept of qualia, which encompasses the feeling of what it is like to have subjective experiences and sensory appreciation. On the other hand, there is the field of aesthetics, which is the formal study of the principles of beauty and artistic taste. Aesthetics and qualia form a profound connection, as our personal experiences and emotions are conveyed through the aesthetic elements we use. As artists, we aim to transmit these feelings to our audience.
For example, when you gaze upon Van Gogh's Starry Night, you perceive vibrant colors, textured brush strokes, and, reportedly, an overall sense of awe. These elements evoke unique qualia, meaning deeply personal experiences for each individual. The introspectively accessible sensations that are elicited within us, the qualia, are a byproduct of the formal elements used in the painting: the aesthetic form of the color palette, brushstroke placement, and overall flow of the painting. Essentially, when gazing upon Starry Night, we find that qualia and aesthetics, the subjective and the objective, are inextricably linked – at least for us humans.
Yet, this emotional expression poses a unique challenge for computational systems, which lack the inherent qualia essential for artistic creation. With advancements in deep learning algorithms, these systems can analyze formal aspects such as symmetry, complexity, and ambiguity within visual input. However, quantifying the "sensation" of color or the impact of a paintbrush dipped in turpentine and oil paint remains elusive.
While algorithms may achieve aesthetic success through formal elements, the question remains: can they evoke the same emotional expression as human artists?
Fusing Computational Aesthetics and Deep Learning
In recent years, I've been fascinated by the fusion of deep learning algorithms and artistic creation, which has unlocked exciting opportunities to explore the intricate relationship between qualia, aesthetics, and the ever-evolving landscape of art.
Yet, I must admit that marrying technology with art poses a "grand challenge" since art is traditionally perceived as a domain exclusive to human beings, tied to our subjective, social, and emotional traits. It's not easy for computational systems to make aesthetic judgments that truly resonate with human emotions (Colins 2019). Nevertheless, the quest to enable computational systems to evaluate universal aesthetic properties and connect them with social, cognitive, and emotional components has given rise to the fascinating field of computational aesthetics.
Computational aesthetics seeks to bridge the gap between technology and the human-like aesthetic judgments and expressions we associate with art (F. Hoenig 2005). My focus in this article lies on deep learning—the integration of sophisticated algorithms that dive into aesthetics and qualia. This convergence of deep learning and artistic expression takes us on a journey where technology can become a tool for us artists to tailor our creations, align them with our audience's aesthetic preferences, and explore novel possibilities that nurture the evolution of our art.
By embracing deep learning algorithms, we supposedly gain the ability to automate personal aesthetic judgments, allowing our creativity to flourish while enhancing the overall artistic experience. But how exactly does this work?
Infusion of Deep Learning and Aesthetic Judgment
In a recent study, conducted by Jon McCormack and Andy Lomas, deep learning models were used to predict an artist's aesthetic preferences based on genotype and phenotype data. In the context of generative systems, genotype refers to the set of parameters that define the characteristics of an artwork. Phenotype, on the other hand, represents the visual output or manifestation resulting from those parameters. By training deep learning models on large datasets consisting of images paired with corresponding aesthetic ratings, these models learn to extract meaningful features and patterns associated with aesthetic qualities.
Deep learning models, such as the widely used ResNet-50 and fast.ai's Tabular model, were used for analyzing and assessing aesthetic qualities. ResNet-50 is a convolutional neural network (CNN) that excels at image recognition tasks, while the Tabular model is designed for working with structured data, such as genotype and phenotype information.
In the case of ResNet-50, the model is trained to recognize visual features that align with aesthetic preferences, using a dataset that spans various artistic forms. Similarly, the Tabular model is trained to predict aesthetic categories and rankings based on the genotype data.
The utilization of deep learning models allows for the quantification of aesthetic judgment in a more objective and systematic manner. The models learn to identify visual cues and patterns that are indicative of the artist's aesthetic preferences, capturing both explicit and implicit aesthetic criteria. By analyzing a vast amount of data, these models can make predictions about the artist's preferences, potentially assisting in exploring the design space, categorizing phenotypes, and even suggesting transitions between aesthetic categories. But aside from predicting an artist’s preference, it would be necessary for AI “art comrade” to also express artistic judgment. Luckily, there’s an AI system that can do just that.
The Machine Meets Art in MENA
Let’s take a look at ArtEmis, an AI system that reviews artworks with the expertise of a human art expert. It was developed by Saudi Arabia's King Abdullah University and is backed by an extensive data bank of human explanations on thousands of artworks. ArtEmis represents a groundbreaking effort in overcoming the ‘grand challenge’ by quantifying aesthetics and qualia through the domain of art by bridging the gap between emotions and visual stimuli using language. By collecting and analyzing language-based explanations for emotional responses evoked by visual artworks, the system sheds light on the intricate interplay of subjective experiences, aesthetics, and how emotions are expressed through linguistic descriptions.
Deep learning approaches are harnessed to develop neural-based speakers that emulate human emotional responses, effectively translating the elusive essence of qualia into quantifiable expressions. The ArtEmis dataset serves as a valuable resource, providing rich and diverse linguistic annotations that offer insights into the affective and abstract aspects of art, transcending traditional descriptive datasets.
ArtEmis also works with ambiguous artwork. It associates the following image with the emotion of excitement and says that “the colours are bright and bold and the lines are very dynamic”.
This innovative research illustrates how the integration of machine learning techniques can not only predict emotional reactions to art but also provide plausible explanations, offering a glimpse into the potential for quantifying and understanding the complexities of aesthetic judgment and emotional responses in the artistic domain. Through ArtEmis, the fusion of aesthetics, qualia, and deep learning ushers in new possibilities for advancing human-computer interaction and enriching the artistic experience.
The Artist(s) Meets The Machine: A Perspective
As artists, the integration of deep learning techniques into our creative process opens us up to exciting possibilities. These powerful tools offer us valuable insights into what our audience truly craves in our art, allowing us to fine-tune our artwork with precision.
In the end, embracing these technologies not only helps us connect more profoundly with our audience but also fuels our own artistic growth. We're no longer limited to the known, but rather we're inspired to reach new heights of expression.
As an artist, I am confronted with a profound question regarding using AI tools in my creative process. Undoubtedly, these tools hold tremendous potential for aspiring artists, providing invaluable assistance in honing their style and mastering fundamental elements like symmetry, perspective, and color theory.
However, when it comes to my own practice, I must admit that while I acknowledge the usefulness and immense benefits AI tools offer to beginners, I find myself hesitant to incorporate them into my artistic journey. My art is a reflection of my innermost self and emotions, and any form of external interference could dilute the authenticity and uniqueness of my creative expression.
Though I may not be classified as a "professional" artist, I hold a deep conviction in preserving the purity of my artistic voice and maintaining an unfiltered connection between my creations and myself. Each artwork I produce carries a distinct message, and I am committed to presenting it exactly as envisioned, untouched by external elements.
While I am excited about the future of art and the potential of AI tools for artistic development, I'm eager to see how technology will continue to empower artists. With that said, I choose to remain true to my personal artistic process, allowing my emotions and vision to guide me on the deeply personal journey of self-expression.
References:
Achlioptas, Panos & Ovsjanikov, Maks & Haydarov, Kilichbek & Elhoseiny, Mohamed & Guibas, Leonidas. (2021). ArtEmis: Affective Language for Visual Art.
F. Hoenig, “Defining computational aesthetics,” in Proceedings of the First Eurographics Conference on Computational Aesthetics in Graphics, Visualization and Imaging, Computational Aesthetics ’05, pp. 13–18, Aire-la-Ville, Switzerland, Switzerland, 2005.
Johnson, Colin & McCormack, Jon & Santos, Iria & Romero, Juan. (2019). Understanding Aesthetics and Fitness Measures in Evolutionary Art Systems. Complexity. 2019. 1-14. 10.1155/2019/3495962.
McCormack, J., Lomas, A. Deep learning of individual aesthetics. Neural Comput & Applic 33, 3–17 (2021). https://doi.org/10.1007/s00521-020-05376-7