Visual Search Interface by Computer Vision Group, Heidelberg University

Abstract

The search for motifs and objects is essential to art history as it helps to understand individual artworks and analyze relations between them. Digitization has produced extensive art collections that hold much information and promise new knowledge for art history. However, in order to extract this knowledge, scholars need sufficient tools to analyze them. This article presents a novel visual search algorithm and user interface to support art historians in analyzing extensive digital art collections. Computer vision has developed efficient search methods for photos. However, applied to artworks they show severe deficiencies due to massive domain shifts induced by differences in styles and techniques. We present a novel image representation based on a multi-style feature aggregation, which reduces the domain gap and improves retrieval results without additional supervision. Furthermore, we introduce a voting-based retrieval system with efficient approximate nearest-neighbor search, which enables finding and localizing small motifs within an extensive image collection in seconds. Our approach significantly improves state-of-the-art in terms of accuracy and search time on various datasets and applies to large and inhomogeneous collections. Besides the search algorithm, we developed a user interface that allows art historians to apply our algorithm directly. The interface enables users to search for single regions, constellations of multiple regions, and integrates an interactive feedback system. With our methodological contribution and easy-to-use user interface, this work manifests significant progress towards the machine-supported analysis of visual art.