Page 310 - The Black Box Book

C U R AT O R O F S YMBO L S
One of the methods of machine vision is the detection and classification
of objects in the image. In the language of engineers, an object is prac-
tically any visible element that can be identified and bound in the image
by a so-called boundbox. It can be a vase, a flower, and a human face as
well as an angel, a halo, or the Virgin Mary holding the baby Jesus in her
arms. While computer science does not distinguish between the meanings
of these elements in its terminology, art history goes a little further in this
direction. At the very least, there are terms with established meanings,
such as motif, symbol, and genre.
Typically, a motif is an observable element in a piece of art; it can
be specific or abstract, and its definition is closest to that of an infor-
mation object. Some motifs have no special meaning, do not represent
anything in particular, and only relate to themselves. Symbols, or icons,
are visual references linking phenomena and images, in turn shaping the
message and playing an intrinsic part in how the story is told. Whether it
is a Pop Art print or a pre-modern painting, the effect an image can pro-
duce can be said to be created through the relationship of symbols. This
is especially true in medieval art, where a complex system of meanings
emerged that is the subject of interest to many scholars of iconography.
They have even released a number of specialist handbooks in an effort
to organize the branching collections of interconnected references and
messages. Thus, the reader might discover the symbolic meaning of
the goldfinch (Christ’s sacrifice), the color blue (God’s knowledge, heaven),
or yellow (social isolation) in a medieval picture (Bartlová, 2013, p. 91–92).
(Bartlová, 2013, p. 91–92).
Compared to motif and symbol, defining genre is somewhat more
challenging. Within the visual arts, two meanings are associated with
the term. Firstly, it can be a category of painting depicting scenes of
everyday life of ordinary people – in this sense, the term was used, for
example, by the theoretician Andre Felibien, who in the second half of
the 17th century established a hierarchical scale among the subjects of
a work and began to use it to classify paintings. (In his eyes, historical
scenes were the highest ranked, followed by portraits and just genre
paintings.) In a broader sense, then, “genre” can relate to the subject
matter or content of a painting (Riechert, 2017).
For us, it is crucial that themes, symbols, and genres constitute
a highly visible layer of the artwork, and it is therefore reasonable to be-
lieve that these aspects will also be discernible using machine vision. If
these themes can be reliably searched, categorised, and demarcated, this
technology might be used to examine cultural and historical data and to
automatically capture, search, and link databases containing hundreds
of thousands of replicas.
310
MOTIF AND SYMBOL DETECTION
Object classification typically uses pre-trained neural networks drawing
their visual experience (in particular) from photographs of 21st century
America. However, if these algorithms are employed to classify pre-mod-
ern photographs of Central Europe, we will inevitably encounter their
limitations. The digital curator therefore uses proprietary neural networks
designed to classify motifs and symbols, with their skills extracted di-
rectly from historical paintings, prints and drawings. To this end, a set
of 3,950 digital reproductions was created. These works came from
the collections of Central European galleries and dated from 1300 to
1800, and the aim was to spread the sample as evenly as possible across
the centuries. These paintings were labeled with 4 167 objects represent-
ing 13 symbols associated with Christian iconography. These included,
among others, the ‘Madonna’, ‘angel’, ‘white dove’, ‘bishop’s scepter’, ‘Latin
cross’, ‘halo’, ‘crucifixion’ and ‘crown of thorns’.” This input data served
as a key differentiator between the capabilities of the Digital Curator and
those of other neural networks that had been trained to recognize other
objects and features on otherwise composite datasets.
The Digital Curator’s neural network gradually became better at rec-
ognizing 13 common iconographic symbols, but this set was not sufficient
for a broader exploration of motifs or genres. It was therefore necessary
to supplement its cognitive capabilities with additional objects, for which
the pre-built Resnet V2 Object detection model was used (TensorFlow,
2022). Although it was already a universal algorithm trained on contem-
porary visual materials, it might be used to identify motifs whose rep-
resentation had not altered significantly over the centuries. For example,
a vase, a flower, a tree, a dog or a horse have the same visual features
in today’s photographs as well as in Renaissance paintings. This brings
the overall number of detectable components to approximately 300.
At this point, it was possible to begin to compose the motifs into
individual genres; to define landscape painting as a picture in which
“trees” and “plants” are present, still lifes with flowers through the pres-
ence of a “flower”, “vase”, “pot” or “table”, portraits of nobles or religious
311