AI Hunting: Can Maybe's AI Art Detector Identify Machine-Made "Photographs"?

 “Is this photograph taken by a human, or is it a digital image file made by a machine?” This question is coming up more often these days. AI image generators are becoming more sophisticated so that fake “photos” can even get past humans with vast experience. It has happened to me with increasing frequency as an admin of “I Love Black & White Photography.” It is frustrating to see good, real photos taken by photographers who are careful and passionate about their craft being shunted aside by hyper-real deepfake images. We need AI detection capabilities!

Even worse: Authors of these faux photos are passing them off as real. That’s deceptive and dishonest.

This has caused problems for photo editors, photojournalists, competition judges. Especially photojournalists. When some images are convincing-looking lies, other true photos are placed under suspicion. 

So, I once again dive into evaluating a new software category, AI Detection apps. Today, we look at the AI app clearinghouse huggingface.co's Mabye’s AI Art Detector. It was created by Matthew Mabye, who wrote: “the open release of Stable Diffusion has not been good news for a different group: working artists, graphic designers, and illustrators. While the output of DALLE-2 is more passable as plausibly human-made, Stable Diffusion comes close — and is free. This creates almost impossible price competition for the services of trained professional human artists.”

Mr. Maybe used machine learning to detect AI in an image, and after some hit-or-miss results, he came up with HuggingFace, which he admits is not perfect but rather an experiment.

But at least it’s free. For now, I’m sticking with free. If you are interested in comparative test results of the fee-base models (which might be more reliable), I may need to start a Patreon page. Should I do it? Please leave a comment!

Maybe, Maybe Not: Let’s Dive Dive In

In the meantime, let’s see how close or far off Hugging Face is in evaluating my four sample photos. Two are real, and two are machine-made fakes that I prompted. Can it distinguish which is which?

Image 1: Realistic—But It Was Created By A Machine

This one’s AI, and it fooled Hugging Face. Frankly, it could have foold me, too, if I wasn't the person who created it through a prompt. I used the words “Palestinian and Israeli shaking hands” to fabricate this image. And yet, Huggingface gives this a 69% Human rating. Geopolitics aside, the image does look rather photo-realistic, but the idea is to find a program that can find the traces that say “yes, this is AI”. We’re not off to a great start here.

Image 2: Clearly AI

This machine-created image is more blatant. My prompt was: create an image that looked like a photograph taken by Garry Winogrand in Manhattan in the 1970s. From the headless body to the missing legs, an impossibly short tie and pants that can’t decide if they are pants or shorts, the “tells” are strong here. And yet, Hugging Face isn’t quite sure.

The “70% artificial” rating is an improvement over the previous shot, but the machine behind this app still has a lot of learning to do. This one should be much closer to 100% artificial.

Image 3: A Real Photograph, But I Threw A Cruve 

I took this picture in New York City in 2021, when most people on the streets were still wearing masks; I deliberately chose this shot to see how the app would handle masked faces. In my previous evaluation, a different app thought this was machine made. The good news here is that Huggingface wasn’t thrown off. Indeed, it rated this one 93% taken with a camera by a human. Well done!

Image 4: So Real, I Want That Pizza

I chose this as a "control" because the faces are clearly visible. This should have been…and was…easy to identify as a human-made photograph. 

Summary: Not there yet

While Huggingface was able to confidently identify human-made photographs, it wasn’t as successful with both the less-obvious and painfully obvious machine-made images.

Huggingface is clearly presented as an experiment. (Go here for a deep dive into how Matthew Maybe created the software.) Maybe admits it is far from perfect but is a good starting piont, and I hope more AI people can use the program as a starting point.

As I said before, Huggingface is free. Would a fee-based AI detector app do better? What’s it worth for you to find out? Let me know in the comments!