Feline or ostrich? The neural network that deciphers humanity's memes on Twitter

Capture of two posts from @ResNeXtGuesser

Glen prefers to keep his professional life separate from his personal.

That is why he omits his last name in this interview conducted by mail because he also prefers written communication to oral communication.

The 23-year-old engineer based in San José, California, dedicates his professional life to the development of embedded computer systems.

The staff employs her, among other things, in running a Twitter bot as young as it is popular: @ResNeXtGuesser.

In less than six months, Glen and his automaton, a neural network that tries to decipher all kinds of memes, have managed to surpass 350,000 followers. "There are engineers, scientists and people in technical branches who are interested in seeing an unconventional application of machine learning and then there are those who simply enjoy the memes that the bot publishes," he reasons. These systems are, in short, algorithms trained to process certain information, in this case images, and produce a result, which in the example at hand is the bot's predictions about what is in the photograph: the same thing sees a frog arboreal where there is indeed one, than an ostrich where there is a cat with its face covered in milk. "In my opinion,the best projects are those that mix a professional or academic theme with a bit of immaturity ”, sums up the baby's father.

On August 12, the neural network made its first performance.

He predicted with 99.99% certainty that a pizza was indeed a pizza and got a retweet and three likes.

“The reception was a bit slow the first few months, but every so often, the bot would tweet a particularly funny prediction,” recalls the engineer.

Those early hits, in which the model mistook a tower of cheese for a tropical pineapple or a fridge full of eggs with ping-pong balls hovered and even surpassed 20,000 retweets.

More information

The laughing stock of neural networks

Image prediction: pineapple

Confidence: 99.3% pic.twitter.com/w6to1e8Vxk

- neural net guesses memes (@ResNeXtGuesser) November 2, 2021

Glen, who completed his computer engineering studies last year, explains that he was always interested in machine learning systems. "I did some courses on the subject, and a good part of it is working with images: trying to identify them, edit them or even compress them more intelligently," he explains. "While doing homework for problems like this, I found it fun to run memes through the system."

The neural network used by the bot, ResNeXt, is one of the candidates submitted in 2017 to the competition organized every year by the ImageNet project, focused on creating a database of tagged images that can be used, for example, for the research on artificial vision systems. “After I graduated, when I finally had time to work on some personal projects, I was experimenting with the ResNeXt architecture, it was interesting to see how even by passing random memes through the system you could understand the logic that the neural network was using in its classification. ”, Says the engineer. Those tests gave him the inspiration to create a Twitter bot that would post a meme and prediction every few hours.

“Getting it up and running was a bit of a challenge.

I had never done anything like it, ”admits Glen.

What took him the most time was learning how to use Twitter's application programming interface (API) to connect the neural network to the platform.

“Believe it or not, the easiest thing was to write the code for the neural network, because the Pytorch library for Python has a trained and ready-to-use ResNeXt model.

Just download it and run it ”, he assures.

Image prediction: French bulldog

Confidence: 14.5%

Submission by @EMTSLA pic.twitter.com/EOuk9uNj1q

- neural net guesses memes (@ResNeXtGuesser) November 20, 2021

For now, Glen does not consider improving or expanding the coverage of his neural network, which makes its predictions from a list of a thousand categories that include some as specific as "garden spider" and others as broad as "shoe store. ”. "It's not that I don't want to, it's that modifying the architecture of a neural network and retraining it takes a lot of time and work," he argues.

What the engineer has been adapting is the page to which the followers of the bot can send their own proposals of memes to decipher. Or rather:

. "I had to block it after a few months because the volume of shipments was simply becoming too difficult to manage," says Glen, who currently has about 8,000 images pending moderation and another 1,500 queued to be published. It has also relaxed some predictions to offer more than one category. "Any meme with a lot of text or solid colors tended to be classified as a comic or a book dust jacket, which makes sense, but it's not very funny."

Ultimately, the accuracy of @ResNeXtGuesser's predictions is almost secondary.

“The most popular tweets are those in which the bot is wrong but you can see what the logic is that the neural network was following.

The case of the cheese tower and the fridge full of eggs are good examples of that ”, explains its creator.

There are also people who, against all odds, have sided with the automaton and get excited when it succeeds.

"It's fun and admirable at the same time."

Would this Twitter account still be funny if the neural network was always right?

“It would be a pretty big technological achievement, but I don't think it would be very interesting.

Part of the appeal of the bot is seeing a new technology pushed to its limits in a funny way. "

You can follow EL PAÍS TECNOLOGÍA on