IsitAI.com Evaluation: Can This AI-Powered App Distinguish Human-Made Photos from AI-Created Images?

If it takes one to know one, can AI detect AI? 

Artificial Intelligence-based images that look like photographs are becoming more convincing. It is becoming increasingly challenging for editors and online photo forum moderators to pick them out from legitimate photos (in other words, images made with a camera). What is real? What is not? The lines are blurring, which is really, really bad for photography, if photography is about truth.

We need help!

Fortunately, help is on the way, although it is not foolproof. Illuminarty, AI Or Not, and Is It AI are web/app-based programs that claim to be able to tell you if a photo is a human-made-with-a-camera photo, or the result of a machine-learning-based process. In other words, real or fake photos. There are, no doubt, others and I'll be adding them to my list. If you are using an AI detection app not listed above, feel free to leave a comment about it below.

As the founder and main admin for the Facebook group "I Love Black & White Photography," I've had to deal with this issue more frequently in recent months. It came to a head recently when someone posted this photo: 

Looks like a street photo, right? Nope. It's fake. Look carefully: Do you see the telltale signs that it was generated by a really smart computer? For some viewers, the clarity of the image is a red flag but doesn't necessarily mean it wasn't made with a camera. Nevertheless, this image looks almost too good. But the "tell" is in her fingers. See the extra digit, in shadow, under her right forefinger, that doesn't seem to belong to either hand? Look at her thumb—it is badly distorted. The bindi on her head looks like it is floating in air. Her backpack straps don't match, and one stops in the middle of her shoulder.

Once you see these defects, you can't unsee them. My team of half a dozen admins came to a consensus that it was AI only after one sharp-eyed member pointed out the problems. When we go through dozens of submitted images a day, images like this can slip by. 

After determining this one was an AI-generated fake photo, we looked at the same member's other posted photos, and saw more AI fakes. But we weren't sure about some of them. One of our Admins, Shane Balskowitz, has been battling AI fakery for some time now, and he recommended trying "Is it AI?". I decided to give it a try.

Introducing "Is It AI?"

Here is how Is It AI describes how it works: "Our AI detection tool analyzes images to determine whether they were likely generated by a human or an AI algorithm. The tool works by using machine learning models to examine various features of the images, such as color patterns, shapes, and textures, and then compares them to patterns typically found in human-generated images or AI-generated images. 

It continues: "The machine learning models were trained using a large dataset of images that were labeled as either human or AI-generated. Through this training process, the models were able to learn to recognize patterns that are indicative of either human or AI-generated images."

How It Works

To use the app, simply upload an image and click the "check" button. The image is then processed, and the confidence results are displayed. These indicate the likelihood that the image was generated by AI or a human. However, the site cautions that "the results should not be considered final as the tool could have some false positives or negatives" and that it could produce inaccurate results. As we shall see.

Here's an image that we suspected was AI:

Clearly, Is It AI thinks this is quite likely an AI-generated image. This was helpful, since there were no obvious "tells" in this one other than the fact that it looks almost too perfect to be a real photo. Given the context that the same person was "busted" for other AI-fake images, we tended to agree with Is It AI's analysis.

Evaluating the App

I decided to run a few tests on Is It AI. I expected mixed results, and I was not disappointed. As a control, I ran a couple of photos of my own— I am 100% sure these are not AI-generated images, because I took them with my camera. The results were enligtening as to the app's limitations.

Here's one that I shot in 2021, and Is It AI's analysis:

Is It AI apparently is uncertain whether or not this is a real photograph, and is leaning towards not! Is it possible the face masks faked out the detection algorithm? That's very possible. So, I tried again. I chose a photo where faces were visible:

Is It AI correctly indicated that this is a photo that was taken by a human being. (Although, as the human being in question, should I be worried about that 3.58% chance that I'm an AI?)

Now let's see what it does with 100% AI images. I make no apologies that my approach to street photography is inspired by Garry Winogrand. So, I hopped onto Deepai and prompted it to create a fake Winogrand photo. My prompt was: "Black and white Street photo in new york city in the 1970s in the style of Garry Winogrand"

It delivered this monstrosity: 

Is that a headless body to the left and behind the man in the suit? Where's the woman's left leg? Or the man's right leg? Also, his hand is blurred. Looks like the AI tried to use motion blur to fake us out. Do you see other anomalies? Let's see what Is It AI found. Should be a no-brainer, right?

Ruh-roh! This isn't great--in fact, this photo got almost the same "somewhat likely to be AI-generated" as my real photo with people wearing masks.

So...is this thing effective, or not? I tried again. The next image prompt, ripped from today's wishful-thinking headlines, was "a Palestinian man and an Israeli man shaking hands." It came up with this: 

Hmm...doesn't look too bad, although I see possibly wierd finger sticking up behind the taller man's thumb. But otherwise, I'm hard-pressed to find any "tells" here. (A somewhat philosophical side note: Which one's the Palestinian, and which is the Israeli?) Let's see what Is It AI's verdict is:

I'm actually a little surprised, based on previous results, that Is It AI got this one right. I wonder if it took the fraught geopolitical situation into consideration and "thought" to itself that this is an extremely unlikely scenario (although in fact it happens all the time in real life, according to my Israeli friends). Naah. It probably detected a problem with the shirts' fabric.

Conclusion: Imperfect but Helpful

So, what can we learn from this limited sample? Like its human overlords (for now), this AI-based AI detector isn't perfect, but its robotic opinion may help undecided humans in their efforts to determine if the image in front of them is, indeed, a photograph. The waters are still muddy. 

Is It AI is free, but they do encourage you to sign up, so I'm guessing eventually there will be a limited free version and a paid version somewhere down the road.

I give this program a moderately likely chance that it can correctly identify whether or not an image is AI-generated.

Stay tuned for more evaluations of AI detectors!