Andrew Yeckel > Art Gallery > Quantifying art

## Quantifying art — can it be done?

*Everything shown here is *Copyright © 2019-2021
Andrew Yeckel, all rights reserved, except as noted.

I have no formal training in art. I enjoy visiting museums,
but my intellectual understanding of the artwork is
non-existent. I have been studying educational materials
online, trying to grasp some of the basic terms people use to
discuss art, such as *abstract* vs. *figurative*,
and *linear* vs. *painterly*. I will say more
about these later. First I take a look at some studies on
characterizing visual art by quantitative measures. This
appears to be a relatively new area, the development of which
has benefited from access to a large number of digitized
artworks and the computing power to analyze them.

### The complexity-entropy plane — is it actually a plane?

Many statistical measures have been proposed to characterize
artwork, often drawn by analogy to some other discipline. Our
point of departure is this
paper, which explores some simple metrics borrowed from
information theory to quantify order and disorder in
paintings. Order is characterized by a complexity measure
denoted C, and disorder by an entropy measure denoted H (their
choice of symbol, I would have preferred S to avoid confusing
H with enthalpy). Figures 1 and 2 in the paper give the reader
a good idea what the authors are purporting to accomplish.

Before discussing their work, I show a plot of C vs. H that I computed for some of my own digital artworks and landscape photographs.

The red symbols are complexity C and the blue symbols are
disequilibrium D. Complexity is related to entropy
by C(P) = D(P) H(P) where P is a probability distribution
related to the spatial variation of shades in neighboring
pixels. I describe how these quantities are computed in
greater detail below. For the mathematically inclined,
Equations 1 to 4 in their paper define these quantities more
precisely.

The authors describe the left end of entropy scale (H = 0, C
= 0) as representing the linear style and the right end as
representing the painterly style (H = 1, C = 0). The leftmost
point in my plot (H = 0.03) was computed from my knockoff of a
Mondrian painting, clearly in the linear style. The rightmost
point (H = 0.93) was computed from a photograph taken in
Venice, a fine example of the painterly style.

Both of these images have low complexity by the statistical
measure used here. Complexity is approximately parabolic,
reaching a maximum near the middle of the plot. The
highest complexity I have computed is for this pair of
co-rotating vortexes, which is the isolated point in the
middle of the plot (H = 0.47).

Yes, that does look like a complex image in some sense, but
the photograph from Venice looks pretty complex to me, too. I
have found it often difficult to guess which of two images has
the greater statistical complexity, so I think we should avoid
equating this measure with our colloquial understanding of the
term.

An issue of greater concern is how the authors of the paper
describe C vs. H as the complexity-entropy plane. This is
wrong because C and H can only form a plane if they are
independent variables, which plainly they are not. Entropy and
complexity are functionally related and exist only in a narrow
range of values along an arc. Nearly all of the plane is
excluded as a valid solution, which substantially reduces the
value of complexity as an independent discriminant.

Nevertheless, the overall approach seems promising based on
this small set of sample images, but let's see what happens
when it is carried out on a broader set of data.

### Complexity and entropy of digitized paintings — can they
predict style?

The authors have harvested thousands of digitized paintings
from WikiArt.org and used the pixel values of these images to
compute C and H of each image. These two variables are
compared for a large number of paintings from different
artists, eras, and styles to see whether artworks cluster into
recognizable groups. Clustering is analyzed in terms of
qualitative characteristics of the art that the authors
associate to these measures of order and disorder, such as
linear vs. painterly. In some sense the authors are trying to
construct a machine learning classifier capable of
automatically categorizing an artwork in terms of style and
era, and they claim to have found one:

"Our research shows that simple physics-inspired metrics that are estimated from local spatial ordering patterns in paintings encode crucial information about the artwork. We present numerical scales that map well to canonical concepts in art history and reveal a historical and measurable evolutionary trend in visual arts. They also allow us to distinguish different artistic styles and artworks based on the degree of local order in the paintings."

Let's unpack this claim to see how much we agree with it.
Reducing da Vinci's Last Supper to a single point on a curve
seems unlikely to preserve the essence of this painting to any
meaningful extent. The most we can hope to extract by
comparing its complexity and entropy to other paintings is
some measure of distance between them. It is unrealistic to
hope that many different styles would each map to its own area
and maintain separation from other styles. This can be seen in
Figure 2 of their paper where 92 different artistic styles all
fall within the same narrow arc. The authors admit as much
towards the end of their paper:

"Our approach represents a severe dimensionality reduction, since images with roughly 1 million pixels are represented by two numbers related to the local ordering of the image pixels. In this context, an accuracy of 18% in a classification with 20 styles and more than 100,000 artworks is not negligible."

Had only they stopped with the first sentence. An 18%
accuracy rate is terrible, indicative of almost nothing, even
when the underlying data is sound. But here the underlying
data is unsound, rendering moot any claims about accuracy. To
explain why, I need to diverge here for a moment to explain
how they compute entropy, which is described in the materials
and methods section of their paper.

The entropy measure is based on 24 possible spatial
permutations of the relative shades of each pixel in a 2x2
group of neighboring pixels. The shade value in each of the
four pixels is computed as the average of the R, G, and B
values of that pixel, essentially mapping it to a gray scale.
The pixels are numbered 1 to 4 in an arbitrary but consistent
manner, and the permutation is computed by reordering them
from darkest to lightest shade, for example 4132, 1342, etc.
This is repeated for all 2x2 pixel groups in the image and
each of the permutation types is counted.

In an image made of random noise, each spatial permutation is
equally likely to occur. In pictures that show *something*,
be it a landscape or an abstract geometric object, certain
permutations of shades will occur more frequently than others.
By counting the frequency of occurrence of each permutation
type, we can compute a quantity called the Shannon entropy,
ranging from 0 for perfect order and equal to 1 for complete
disorder. This concept is borrowed from information theory
where it is used to characterize how much information is
contained in a string of digital bits called a time series.
Don't worry if you don't understand what all this means. I
don't understand it either.

If this approach seems ad hoc, that's because it is. Its
fundamental flaw is that the human eye cannot distinguish
variations in shade across a 2x2 pixel box, unless the change
in shade is large, for example at an edge. Most paintings are
dominated by areas of gradual change. Any reasonable
digitization of a painting for viewing and analysis should
have sufficient resolution that we do not perceive individual
pixels. Changes of visual relevance occur on larger scales
than are captured by this definition of permutation entropy,
which focuses instead on local variations too minute to
visually detect.

### Paintings are not made of pixels — are pixel-based measures
appropriate?

This brings forward an important point about the entire
enterprise of quantifying art. Paintings are not made up of
pixels. Any digitization of a painting will have artifacts at
the scale of its pixels that neither exist nor connect in any
way to the original painting. Any useful method of analysis
should, at a minimum, exclude this local scale, rather than
embrace it. The authors try to escape this trap with a flurry
of data intended to prove that their local method of computing
entropy is independent of image size. They show scatter plots
of entropy vs. image size for thousands of *different*
artworks of all different sizes (figure S4 in the appendix to
their paper). Finding that entropy values compared in this
manner are uncorrelated to image size, they conclude their
entropy measure is fundamentally independent of image size.
But we *expect* different artworks to have different
values of entropy whether they are different in size or not.
The sizes of the images harvested by the authors have no
connection to the content of the images, they are simply the
sizes chosen by their web curators, so we have no reason
whatsoever to expect any correlation between image size and
entropy among different images.

Curiously, the authors have failed to do the simplest, most
obvious, and only valid test of their hypothesis, which is to
compare entropy between different image sizes for the *same*
painting. Conveniently, WikiArt.org provides many paintings in
several different image sizes, or we can easily prepare them
ourselves using any image editing software. I have tested this
myself and find that entropy depends strongly on image size,
so strongly that two images of the same painting in different
sizes can appear much further apart in complexity-entropy
space than two very different paintings in the same size. For
the Mondrian knockoff I show above, the entropy approximately
doubles each time the resolution is halved, without changing
the appearance of the image at all. For the vortex pair image,
the entropy changes from 0.47 to 0.63 when the resolution is
halved, and to 0.76 when halved again. Both of these picture
types have color filled regions that are areal in extent
punctuated by hard edges that are linear in extent. As the
resolution increases, hard edges are represented relatively
less in the count of permutations, causing the entropy to
decrease.

These aren't the only pixel-based artifacts that plague these
statistical measures. The image shown next was down
sampled from an original resolution of 5700 x 5700 to a final
resolution of 570 x 570 by three different interpolation
methods: bicubic, bilinear, and hard edge. These give values
of H = 0.915, 0.947, and 0.990, respectively. The original
image has an entropy of 0.883.

These drastic changes in entropy are caused by ordinary image
editing procedures that often have little or no effect on the
visual appearance of the image. Any useful statistical measure
should remain nearly invariant under these transformations.

Another problem plagues this approach. No distinction is made
between small and large changes in shade from pixel to pixel
when computing the permutations. In any physical painting,
small variations in paint color and texture occur even in
areas intended by the artist to look uniform. Photographically
digitizing an image inevitably introduces more small
variations. My digital knockoff of Mondrian's work is made of
pure color. This purity gives it a very low entropy level, H =
0.03. An actual Mondrian painting, after digitizing,
invariably has a high entropy level, placing it on the
opposite side of the complexity-entropy plane with H = 0.91.
The situation is illustrated in the next figure, showing how
three different but similar images place along the entropy
axis. By adding 3% Gaussian noise to my image I can change its
entropy from 0.03 to 0.99 while changing its appearance only
slightly, in fact making it closer in appearance to an actual
painting.

In the paper, none of the digitized paintings are found
anywhere near the linear end of the entropy scale. Everything
is clustered at the painterly end. Mondrian's style,
Neoplasticism, is shown ranging from H = 0.82 to 0.85 (figure
S6 of the appendix), where it overlaps with several other
styles. These observations rule out permutation entropy as a
meaningful discriminant for quantifying painted art.

There are other reasons to dislike the approach studied here. Color information, specifically hue and saturation, is discarded. Use of color is a strong style element for many artists. Converting three color channels to a single gray channel also causes a devastating reduction in detail in many images. An elaborate drawing made in two colors of equal gray level will appear uniformly gray; literally all the information in the image has been discarded before it is analyzed.

### The paper is a wreck — can we learn anything from it?

The criticisms I've presented here invite an important
question: If the entropy data is really that bad, how are the
authors able to find meaningful clustering by style and era?

The answer to this question is easy: I don't think they have
found any such thing. Given the overall high level of entropy
of digitized paintings and the close overlapping of many
styles, there is little rhyme or reason to the ordering of
styles shown in their figures. In figure 1 of their paper, the
boxes meant to represent discrete eras in art history, and the
line meant to show the temporal evolution of entropy and
complexity, are exercises in wishful thinking.

We can learn some important things from this exercise,
however. Foremost, any statistical measures computed from the
artwork should be largely invariant to differences in image
size, and insensitive to details of the digitization process.
Meaningful statistical measures must be derived from
extraction of features on a visible scale much larger than the
pixel scale. Undoubtedly reduction to one or two
characteristic parameters is inadequate to characterize
hundreds of styles. Successfully identifying one style out of
dozens will likely require a much higher dimensional
representation, perhaps based on some sort of principal
component analysis in a vector space representing the image,
or by training a neural net. But these approaches are
significantly more difficult to implement than the simple
scalar reductions employed in this paper. Easy solutions like
this are not merely useless, they are also harmful for
creating the perception of major progress when little has
actually been accomplished. Similar problems plague other
areas of scientific computing familiar to me.