The Meaning of “Vision” and “Image” in the Age of AI

On the occasion of For Your Eyes Only, curator Domenico Quaranta discusses machine vision, operational images, and the future of human visual culture with Antonio Somaini, professor in film, media, and visual culture theory at the Université Sorbonne Nouvelle – Paris 3.

This conversation took place over email in November 2021.

Domenico Quaranta: Since cave paintings, the production, vision, and understanding of images has been an exclusively human area of activity and competence. All pictures were crafted by humans for the human gaze. Today, as Trevor Paglen wrote in 2016, most images are made “by machines for other machines, with humans rarely in the loop.” As you point out in recent texts, the impact of machine vision and artificial intelligence on today’s visual culture is so profound that it leads us to raise the very question of what we mean by “vision” and “image” in the age of artificial intelligence. Too often, however, I see the immense implications of this question downgraded to an attempt to recognize, explore, celebrate, or criticize such impact, by either pointing to AI’s usefulness or bias and limitations, and by exploring the aesthetics of machine-generated images. Is this what you actually mean? As a researcher in visual culture, are you more excited or frightened by this momentous passing of the baton from humans to machines as active agents of vision?

Antonio Somaini: Trevor Paglen’s quote comes from the essay “Invisible Images (Your Pictures Are Looking At You),” an essay that, when I first read it just after it was published, literally opened my eyes to the new phenomenon of machine vision and its vast implications.

Five years later, I feel the need to question this statement. Is it really true that today “the overwhelming majority of images are made by machines for other machines?” This is certainly the case for various forms of optical capture that record data that are then processed by machine vision technologies without ever producing images accessible to human eyes: for example, automatic license plate readers, cameras autonomously monitoring industrial processes, sensors installed in unmanned vehicles, surveillance cameras connected to machine vision devices that analyze the vast quantity of images they produce without ever visualizing them onto a screen. Besides cases like these, machine vision systems are normally applied to images that are at least partly or momentarily visible by human eyes. When such systems scan the trillions of images that are currently uploaded on the internet, what they analyze are images that have been in most cases produced by humans for other humans, even though such images may appear on a screen for a very short time span. What is true, though, is that in both cases—the forms of optical capture that are processed by computers without producing images that are visible by human eyes, and the machine vision systems that analyze images produced by humans for other humans—the very concepts of “image” and “vision” are put profoundly into question. Can we still use the term “image” for a digital file, encoded in some image or video format (.jpg, .png, .mp4, etc.), that is machine-readable even when it is not visible by human eyes, or that becomes visible on a screen as a pattern of pixels only for a short fraction of time, spending the rest of its indefinite lifespan circulating across invisible digital networks? And what is “vision” when the human psycho-physiological process of seeing, with all its complexity and its mental associations, is reduced, as in the case of machine vision technologies, to entirely automated operations of pattern recognition and labeling, and when the various applications of such operations (face and emotion recognition, object and motion detection) may be deployed across an extremely vast visual field, that no human eye could ever access?

As a researcher studying visual culture and the history of images and optical media, I find such questions fascinating. These latest phenomena do not frighten me, nor excite me. I want first to understand them, in order to unpack their various implications, which are at the same time aesthetic, epistemological, political, and historical. And I do not see in them “a momentous passing of the baton from humans to machines as active agents of vision,” but yet another phase in the history of the ways in which humans have interacted with technology in order to produce, process, and transmit images. We are not moving away from human vision into a realm of entirely automated machine vision. We are rather facing new intertwinings of human and nonhuman agencies, and this is what the so-called artificial intelligence is really about.

smoking-vase-1 by Petra Cortright. Image courtesy of the artist and Feral File.

DQ: Years ago, the filmmaker Harun Farocki introduced the concept of “operational images” to describe what at the time was a very specific category of images, produced by machines and meant to make something rather than to inform or instruct. Today, as you pointed out, any networked digital image has become machine-readable: this means that potentially any existing image has been turned operational. When I think about the implications of this—that any existing and future image, be it a work of art or one of the myriad of pictures we take with our smartphones, could and will be used by machines we can’t understand for purposes we can hardly imagine—I inevitably end up swaying between a sense of impotence and the moral imperative that we must make every effort to regain human control over images. The question is: how?

AS: The concept of “operational images” that Farocki introduced at the beginning of the 2000s, through a series of texts and video installations such as Eye/Machine I, II, III (2001-03) and Counter-Music (2004), has since become a key concept for visual culture studies—a concept which can be applied to any image which participates in some kind of technical operations. Images have been “operational” way before the advent of the semi-automated processes that Farocki analyzes in his video installations, and they will continue to be “operational” in new ways that we will need to analyze and understand. Machine vision systems scanning the trillions of networked digital images that exist across the internet are indeed adding a new layer of “operationality,” in the sense that they turn the entire digital iconosphere into a vast field for data extraction and aggregation, connected to a wide spectrum of processes and applications. In this way, images that are produced and uploaded on the internet for different purposes—a travel photo, a portrait, a selfie, etc.—participate in a whole series of operations, which remain for the most part hidden and inaccessible to those who produced these images in the first place.

Regaining full control of these images will be impossible, because we give up “full control” the moment in which we decide to upload an image on the internet, but we must indeed try to understand as much as possible the various operations these images become part of. We need to understand how machine vision systems work, what kind of data they can extract from images, and how these data are used. We also need to promote new forms of regulation that might prevent the digital iconosphere from turning into a massive field of surveillance fueled by machine vision systems, as it is currently happening in totalitarian or semi-totalitarian regimes. The interesting thing, in this historical phase, is that artists seem to be at the forefront of this battle for a better understanding of the implications of machine visions, and of the structural biases that often lie at the basis of such systems.

DQ: In his seminal documentary Ways of Seeing (1972), John Berger describes the power of the zoom function of the camera by showing a painting by Piero di Cosimo, and saying: “From being part of a strange, poetical world of metamorphosis, a dog can be turned into a pet.” In my opinion, this sentence sheds a light on machine vision that makes it more understandable and less threatening. After all, we didn’t teach machines how to see: we simply instructed them to do, on a larger scale, what we learned to do with images when the camera made them reproducible. Labelling or tagging is just like zooming or cropping: it’s an act of manipulation that follows an act of understanding, which excludes other ways of understanding. Piero di Cosimo’s work can be turned operational; machine vision can turn that dog into a pet, it can even use it as a pet, but it can’t fully capture that “strange, poetical world of metamorphosis” that makes that painting meaningful and alive after centuries. And yet, the problem of how human visual culture, becoming a “special case of vision” (Paglen) for the first time in our history, will evolve, still persists. If machines won’t be able to grasp the richness of human visual culture, will we?

Fractura by Francoise Gamma. Image courtesy of the artist and Feral File.

AS: John Berger’s remarks point towards an idea of “machine vision” that largely precedes the advent of machine vision systems fueled by machine learning processes, and that can be investigated through a media-archaeological approach. Since their appearance and throughout their history, the photographic camera and then the film camera have often been considered to be “machines” capable of introducing a new kind of “vision.” If we begin this media-archaeological inquiry with the 1920s—a moment in which a whole series of artists, photographers, film directors, and cultural theorists began reflecting on the profound cultural impact of visual media such as photography and cinema—we find this idea in Dziga Vertov’s writings on the epistemic and political power of the “kino-eye,” in Jean Epstein’s remarks on the camera as a “metal brain” and a “non-human eye,” in Siegfried Kracauer’s idea that photography offers us a glimpse into a world seen “in its independence from human beings,” and in Walter Benjamin’s concept of the “optical unconscious,” a previously inaccessible region of the visible that photography and cinema reveal and explore through operations such as slow motion and time lapse, the close-up, and the different forms of editing, just as psychoanalysis gives us access to the “instinctual unconscious” through the analysis of dreams or verbal lapsus.

In these writings, photography’s and cinema’s “machine vision” are not seen as a menace, but rather as a new way of seeing characterized by a whole aesthetic, epistemic, and political potential that needs to be explored and understood in order to be then mastered and exploited. This same attitude could be applied, today, to the new kinds of “machine vision” we are faced with. Machine vision systems are not intrinsically destined to turn the digital iconosphere into a massive field of visual surveillance. We can use them for entirely different purposes, in order to foster a new kind of research in various fields across the entire spectrum of the humanities and the natural sciences. Disciplines such as art history and film history are already testing the possibility of applying machine vision systems to vast corpuses of artworks and films, in order to detect, for example, the recurring presence of expressions, gestures, objects, environments, colors, light configurations, etc. What happens in these cases, is the possibility of creating new interactions and new intertwinings between human eyes and non-human machine vision systems. In the future, such new interactions and intertwinings may indeed further help us to grasp “the richness of human visual culture,” as you write, rather than relegating human vision to a marginal condition.

DQ: If it’s unlikely that machines would learn how to see like humans, we are not sure we’ll always be able to understand how they see and make images either. “Infinite Fun Space” is the term—borrowed from Iain M. Banks—used by James Bridle to describe the moment when artificial thinking becomes largely autonomous, speculative, and creative, to a point that goes beyond human understanding. Bridle discusses Google’s DeepDream as an early example of the possible outputs of the “Infinite Fun Space.” How will our visual culture evolve, side by side to this kind of imagery?

One Million Manga Pages by Lev Manovich. Image courtesy of the artist and Feral File.

AS: In his already mentioned essay “Invisible Images (Your Pictures Are Looking At You),” Trevor Paglen writes that “if we want to understand the invisible world of machine-machine visual culture, we need to unlearn to see like humans.” This is of course an impossible task, because we cannot escape the framework and the limits of human vision, even when we look at images produced by new visual technologies. This said, we can bring into our field of human vision images that materialize the outcomes of certain machine vision processes that are used in training machine vision systems. This is what Trevor Paglen has done with his Adversarially Evolved Hallucinations (2017), a series of images generated by a machine learning technology called Generative Adversarial Networks (GAN), taking as a starting point training sets composed by images extracted from the vast image database called ImageNet, related to themes such as “allegories of capitalism,” “omens and portents,” “objects and spaces mentioned in Sigmund Freud’s The Interpretation of Dreams,” “eye-machines,” etc. Works like these try to visualize, for human eyes, specific instants in the process of machine learning: a process in which one of the two neural networks interacting in the GAN, the Generator, produces images that try to resemble the ones contained in the training set, and that the other neural network, the Discriminator, has to evaluate. We find a similar attempt in works by Grégory Chatonsky such as Second Earth (2019) or Complétion 1.0 (2021), which introduce us to the inner workings of the GAN, by extracting and visualizing some of the images they produce. Even though very different from the images produced by Google’s DeepDream, the images that we find in Paglen’s Adversarially Evolved Hallucinations and in Chatonsky’s installations do have a slightly hallucinatory, psychedelic, surrealist outlook, which explains the reason why they have often been associated with the idea of a machine that is dreaming, hallucinating, or imagining, as in Chatonsky’s concept of “artificial imagination.”

During the last few years, several other artists, including Hito Steyerl and Pierre Huyghe, have begun to work with images generated by GAN, and it is important to observe on the one hand the great similarity between these images, and on the other hand the great difference between the discourses and strategies they are part of. In the coming years, it will be very interesting to see how visual artists will continue to engage with machine learning technologies, intertwining their own agencies with the ones of the machine learning process itself, developing new forms of creativity that include image-generating processes that in some cases lie beyond the limits of our understanding. The field that is opening up is a real challenge for visual culture studies. What is at stake is the possibility of understanding the new status of “images” and “vision” within a visual culture that is more and more impacted—as all areas of contemporary culture—by the development of artificial intelligence.