So-called artificial intelligence (AI) has one thing above all else: it can recognize what can be seen in pictures. It is a role model when it comes to describing the potential of future technology. The software now recognizes cell phone sinners at the wheel, filters out photo fakes and guides self-driving cars through traffic. But researchers at the Massachusetts Institute of Technology (MIT) are now showing AI its limits - and causing the most powerful AI programs to fail in a row.
The scientists found that AI can recognize stereotypes fairly well. But if the motif deviates from the norm and no longer fits into the context, then it becomes tricky. If a hammer is in a woolen blanket, a chair lies on the bed, and dirty laundry covers the armchair, most artificial neural networks give up.
While in classic photo databases the accuracy of AI object recognition is sometimes over 90 percent, according to the report, the values collapse when objects that are rotated or skewed have to be detected. The researchers have stored such images in their photo collection called ObjectNet. The values will only reach just over 50 percent if AI software is tested with it. In practice, this would be about as helpful as if software were guessing what can be seen in a picture - which would make it worthless.
WITH
02_statistik_objectnet
It took around four years to build ObjectNet. The scientists had developed an instruction app and hired amateur photographers to take the 50,000 pictures. They had to photograph more than 300 objects such as eggs, flashlights, chargers, alarm clocks and woks. Andrei Barbu, one of the researchers on the ObjectNet team, told SPIEGEL: "We explained which object to put where in your apartment, how to turn it and how to hold your smartphone."
The result is motifs that are mostly easy to recognize for people - but not for an AI. You shouldn't start with human object recognition, says Barbu. It is no problem for people to see an object in a photo and later to recognize it in the real world. "That doesn't really apply to object detection," says Barbu. "Now we have good test material to crack a system without endangering people."
"We have to be very careful"
First and foremost, it's about making AI software better - also to avoid risks like accidents. A software error can quickly become dangerous when used on the road. When asked what the high error rate means for the development of self-driving cars, the researcher says: "This means that we have to be very careful." Object detection is not yet perfect. "As long as we don't test them properly, we won't discover the inaccuracies either." The pictures do not want to help to reduce the accuracy of the AI programs. "We want to show how you can test the performance of image recognition more reliably."
However, the researchers expressly point out that the images are not a training database, but test material. So you can and should only check how well an already trained model deals with unknown material. ObjectNet is far too tiny for an AI training database. Compared to photo collections such as Google Open Images and ImageNet, each with ten million images, the 50,000 ObejctNet images are a tiny selection.
However, the motifs are not as expected as pictures in the ImageNet database, which is fed from Flickr and other photo services. Because when it comes to sharing photos with friends, the images from ObjectNet would not have a good chance, as it says in a report at "MIT News". And that is exactly the crucial point: AI should also recognize objects that are upside down, lying in untidy rooms and taken from unfamiliar perspectives.
