Image Recognition Technologies for Digital Humanities
One of my first assignments as a postgraduate was to identify a jug within a digitised collection catalogue without any existing metadata. With only the visual information provided by the 3D model I was asked to retroactively document the life-history of the object, which proved to be a thrilling challenge that called upon me to problem-solve creatively.
On the bottom of the jug there was a diamond pattern with different numbers and letters I assumed corresponded with some type of code within the pottery trade. Unsure of how to proceed, I decided to change my investigative approach.
Earlier this year (2020) I attended Brendan Ciecko’s online presentation on emerging computer vision applications and their uses within museum collections.[1] While Ciecko primarily explored the potential promises and pitfalls of the automated generation of tags and metadata by computer vision technologies, his talk also alluded to other similar studies conducted on the possible utilization of image similarity analysis. With my curiosity piqued, I looked into the work of computer programmer and digital humanities expert John Resig. Resig conducted tests and collected data on the effectiveness of TinEye MatchEngine’s online application to analyze, identify, and match similar images within museum collections.[2] This is profoundly different from earlier reverse image searching technologies that were only able to recognise identical images, thus severely limiting their functionality with museum collections. Improvements on this technology enable it to analyze similarities which gives rise to a whole new host of features, such as discovering works before and after conservation, copies of the same artwork, different but related artwork, locating artworks that have been inefficiently tagged, cropped or incomplete shots of the artwork, and cataloging errors.
Returning to Resig’s blog, I followed his methodology and uploaded my images of the diamond marking into TinEye MatchEngine. Unfortunately, my attempts garnered no matching results. Shelving TinEye MatchEngine for the moment, I decided to test out another computer vision technology, preferably a newer version. TinEye was released in 2008, making it one of the pioneers of computer vision. Referring back to Brendan Ciecko’s session and the data he shared from a recent third-party industry evaluation on the accuracy of the top six leading image recognition technologies, I decided to test what results I could get using a Google application.[3]
I returned to the internet search and quickly found a free smartphone application Google had created to allow users to ‘search what you see’.[4] By snapping a photo within the application, users would be provided relevant information about the content of their image. Unsurprisingly this application is utilised predominantly among users to match and locate the content of the image with online shopping websites. Yet despite the fact that Google Lens was not created with cataloging collections in mind, my first attempt led to immediate success. The strange diamond marking I could not identify was, according to all the top suggested results, an English registry mark used by patent offices since 1842. This demonstrates the potential value of utilizing image similarity analysis technology during the identification process. While none of the suggested results I found on my first attempt were an exact match to the content of my image, Google Lens suggested other English registry marks because it could recognise the basic pattern. I now understood that in order to continue my search, I needed to decipher the details contained within the jug’s English registry mark.
[1] Ciecko, Brendan. ‘AI Sees What?’. MW20: MW 2020 (January 2020). https://mw20.museweb.net/paper/ai-sees-what-the-good-the-bad-and-the-ugly-of-machine-vision-for-museum-collections/
[2] Resig, John. ‘Using Computer Vision to Increase the Research Potential of Photo Archives'. Published 28th May 2015. https://johnresig.com/research/computer-vision-photo-archives/
[3] CapTech Consulting. ‘Accuracy of Six Leading Image Recognition Technologies’. (22nd June 2017.) https://captechconsulting.com/news/accuracy-of-six-leading-image-recognition-technologies-assessed-by-new-captech-study
[4] ‘Google Lens’. https://lens.google.com/