Page 286 - The Black Box Book

W H AT T HE M A C HI NE
VI S I O N C AN S E E ?
If we examine a painting in a gallery, we can see it contains
an array of different things. When we are looking at it from a distance, we
are likely to be drawn to its dimensions or overall composition. When we
edge closer, we discern the abstract style of the painting, our eyes follow
the lines, surfaces, edges and shapes. When the style of the painting is
representative, we start to explore a motif, a figure or the elements and
how they are spatially arranged. In the case of a portrayal of a human
being, our eyes will naturally turn to the face, the features and gestures
depicted, the position of the body or the attributes of the figure. We might
also be intrigued by the creative style, the schematic aspect of the image,
or the specifics that emerge from the artist’s handwork, such as brush-
strokes, contrasts, or the color scheme; we might even pay attention to
the unique and defining artistic signature.
Machine Vision tools and human eyes both possess numerous
ways and methods of visioning the world. Just as there are many ways
of observing with the human eye, there are a variety of machine vision
tools that can be used in the analysis. Each of these methods highlights
a unique aspect of images as well as an interpretation. Thus, by the use
of specialized algorithms, one can analyze in a different manner colors,
contrasts, shapes, search for similar images in large collections, compare
painting the handstyles of different artists, and classify artistic styles.
In my work on the Digital Curator project, I investigated these ap-
proaches. The terms capture a collection of software tools, experiments
and applications that I created during my PhD at the Academy of Arts,
Architecture and Design in Prague. Ultimately, these tools aim to bring nov-
el perspectives in respect to the study of artistic creations. Even though
some of these outputs are online and accessible at digitalcurator.art and
atlas.digitalcurator.art, there are several others that do not presently
have a user interface, and thus my descriptions and examples explored
here will be derived from a limited sample of my work.
Three different experiments will be presented in this text. The first
one is called Curator of Art Techniques and focuses on the automated
classification of various drawings, paintings or prints. Here, an algorithm
takes on the role of an expert, who classifies whether a particular canvas
is an oil painting, watercolor, a woodcut, or an engraving. The second
experiment titled Curator of Taste, focuses on collections that have
appeared under the guidance of a curator. It is a speculative attempt
to extrapolate the formal tastes of curators, in this case illustrated in
the painted collections of Rudolf II, and apply them to a look at later works
of art produced outside the given circle. This experiment created special-
ized neural networks that reconstructed the tastes of long-dead curators
based on works that once belonged to their collections. The provocative
question was whether the aesthetic preferences of people from the past
could be reconstructed in this way.
286
The final section Curator of Symbols analyses iconographic anal-
ysis and genre classification. In this case, the neural network algorithm
has been trained to classify motifs and themes, attempting to capture
their significance, identify the works on which they appear, and conduct
frequency analysis to determine why their popularity reigns over time.
D A T A B A S E
Despite the fact that the principles of computer vision are rooted in linear
algebra, the data itself is more than merely mathematics. Arguably, it is
the responsibility of the curator or art historian to continuously use their
critical eye concerning the specific data, take an active role in its selection
and to also suggest changes to its composition, form, and processing
techniques. Here, the composition of sample datasets for machine learn-
ing can influence how the algorithm comes to evaluate and potentially
shape the outcome of the research. We will therefore proceed to outline
the data that the Digital Curator handles.
The Digital Curator database can be compared to the collection
of an art museum or gallery. More specifically, it is a collection of dig-
ital reproductions of artworks that an algorithm has at its disposal to
search, filter, sort, label, group, and in orderlink, to create datasets and
shape their context. The works included in the database are accessible
and processable by the program; the others, although they can be pub-
lished online, are left out by the Digital Curator. Even so, this is probably
the largest collection of reproductions of Central European art that can
now be handled by algorithms.
From the summer of 2022, the Digital Curator’s collection con-
tained 196,000 works, mostly paintings, drawings and prints from
the holdings of 90 museums in Austria, Bavaria, the Czech Republic, and
Slovakia. It included reproductions of exhibits from the Albertina in Vienna,
the Belvedere, the Alte Pinakothek and Neue Pinakothek in Munich,
the Prague City Gallery, the National Gallery in Prague, the Moravian
Gallery in Brno, the Slovak National Gallery and a number of smaller
collections – from the Benedictine Abbey in Ottobeuren, West Bavaria,
to the East Slovak Gallery in Košice.
The database’s focus on Central European institutions responded
to the shared cultural history of the region, where artists have traveled
for centuries between cities, aristocratic courts, workshops, schools and
academies, sharing their artistic style, formal elements, genres, themes
and motifs. It was not only the artists themselves who traveled, but also
paintings and entire collections that gradually changed owners and lo-
cations. Once related works of art are now spread across hundreds of
museums in several independent countries, and to get a handle on even
a small fraction of this heritage is beyond human capacity.
Regarding the database’s composition, it should be highlighted
that despite its large size, it could not and cannot be statistically repre-
sentative. There are no collections in museums that represent period art
287