Robots that sense what they touch

The famous anthropologist Ashley Montagu wrote many years ago that "the communication we convey by touch is the most powerful means of establishing human relationships." And here they are, one day in early May 2023, a group of researchers from the University of Bologna trying to provide the sense of touch to a machine. They are using two technologies to give a robotic hand two different types of sensitivity: one, a little more thick stroke, which will cover most of the surface of the palm, and another that will offer much richer and more complete information about the hardness, roughness or softness of objects, such as that transmitted to us by fingertips. and that the device will have precisely there.

Today, in the Robotics Laboratory of the Faculty of Engineering, predoctoral researcher Alessandra Bernardini not only strives with infinite patience to explain to visitors how the latter works – small hemispherical sensors are placed, made of a malleable material similar to rubber, which send information about the properties of the object it holds to the computer. Instead, she's willing to give up two of the strawberries she's brought for lunch to illustrate the demonstration. As the robotic hand squeezes the red fruit, the sensors' phototransmitters translate the rubber disturbances into data displayed on the screen in the form of graphs of lines that come together and separate.

Next to her, Roberto Meattini, another member of the team, continues with the demonstration. Once he has a rectangular structure securely attached to his right wrist – at one end there is a hole to put his hand and at the opposite is attached the robotic prototype – he begins to shrink and slowly extend his fingers. What he is doing, with the help of Bernardini at the controls of the computer, is to teach the prosthesis to read the movements of the muscles of the forearm that direct what his hand does, so that he can then repeat them; A bracelet placed a little below the elbow is responsible for collecting and transmitting that information to the robot through wireless sensors.

Researcher Roberto Meattini demonstrates how to use the robotic hand prototype they are creating. Manuel Vazquez

The two engineers, together with half a dozen other colleagues and led by Professor Gianluca Palli, are just taking the first steps of the IntelliMan project, formed by a consortium of 13 universities, companies and research centers from six European countries, with the aim of developing a new manipulation system powered by artificial intelligence that allows robots (whether prosthetics such as the hand we have seen or independent machines) to learn so much. of the environment as well as the interaction with people. The European Commission has chosen it among the first 42 artificial intelligence and robotics projects launched within the Horizon Europe program, the continent's major research and innovation plan. They have been selected for their potential to "improve the society in which we live, addressing important technological challenges," according to the document that compiles the initiatives. Among those chosen are from a large center that will try to ensure safety in artificial intelligence research to proposals for water recovery, recycling or drone development to support field workers and infrastructure maintenance employees in dangerous places.

In the IntelliMan case, Professor Palli explains in his office in the historic Italian city what fruits they will try to obtain from the 4.5 million euros they will receive from the Commission between September 2022 and February 2026. First, its goal is to create prosthetic limbs with which its owners are able to easily address their daily activities, such as holding a glass or opening a door or drawer. The next step will be to transmit those capabilities to a standalone machine that can sooner rather than later become a robotic home assistant that puts the dishwasher on its own or picks up the table after eating. The idea is that they can interact with people, but that no one can imagine chatting with them yet; It is about teaching them new tasks – showing them once how to do it, so that they can repeat them later – but that, in addition, they are able to adapt that work learned to a changing context: an obstacle in the way that was not there before, an object more slippery than the previous one.

For that they need the sense of sight – which can be solved with cameras and sensors – and the more complicated sense of touch, in which they are working. But not only to give it to the machine, but also so that, in the case of prostheses, they also make it felt, in some way, to the person who uses them: "For example, we are working on transmitting the grip force through vibrotactile motors, which are small motors that vibrate at different frequency amplitudes [depending on whether that force is greater or less]", explains Meattini, back in the lab. "It's something that even the most advanced prostheses on the market don't do yet," he adds.

Part of the team operating from Bologna in the IntelliMan project. On the right, researcher Alessandra Bernardini, who works on the sensors that will give robots sense of touch. Third from the right is assistant professor Roberto Meattini.Manuel Vázquez

The researchers move in a space in which there are computers and modern prototypes of cutting-edge machinery, but also many cables, cardboard, some roll of electrical tape and a large metal cabinet from which engineers are taking all kinds of large, medium and small tools. Along with that world of ultramodern, white and aseptic technology, with harmonic and rounded shapes that presides over our collective imagination, this laboratory reminds us that advances – at least part and at least in part – continue to occur in places full of plugs and angles that puncture, with objects that must be touched to assemble and that must be tested again and again to find a way to make them work. Places where students, doctoral students, budding researchers intermingle with others already consecrated, who work on simultaneous projects that converge and sometimes overlap. Where, perhaps precisely because of all that, it is essential to break down each objective into smaller tasks to make them manageable and move forward step by step, but on firm steps, towards the final result.

Right now, when it comes to IntelliMan, the engineers who make up the Bolognese group are focused on understanding how a robotic grip works: first you have to get to the object; then, establish the contact and understand the context in which it is established to know whether or not it can increase the grip force, explains Meattini. All this can be done in different ways, the most classic of which is to create mathematical models that describe how this process can be undertaken and repeated without failure. "The problem with this is that, in theory, it works perfectly, but in real life you don't know exactly where it's going to play, how it's going to do it... So we're working on a different approach, probability-based artificial intelligence, so that we can take the measurements from the sensor and describe them in our probabilistic model and have it tell us what the grip should look like. The real world is probabilistic, it's not like a formula," adds the researcher.

That real world – with its failures, its unforeseen obstacles and its unusual solutions – is very present in this research, in the form of factory workers who explain the hidden secrets of their functions and people with amputees who tell scientists what exactly they need from a prosthesis so that it really works for them and hinders them (the specialized rehabilitation center that the National Institute for the Prevention of Accidents of the Italian Government has near Bologna, in Budrio, is part of the consortium). But there is also very present — in fact, it is at the basis of all this collective effort — that idea of looking for different approaches to old problems. For example, the one that supposes not only that the machine senses what happens around, but, above all, that it knows what to do next with that information, something that they are trying to solve "by mixing two approaches that are already fully established, but whose connection we are still investigating," Palli points out. And he stops at the explanation, because it is one of the great singularities of his project.

Bracelet that, by means of sensors that send information to the robotic hand, allows to move a prosthesis reading the movements of the muscles of the forearm. Manuel Vazquez

On the one hand there are the most classic methods, which consist of mapping and preconfiguring all the activity that the robot will develop in a time and space, that is, it will take a specific object at one point and leave it at another. "It is difficult to apply these solutions in practice if you do not have absolutely all the information of the context in which it moves," he says, which is simply impossible at the moment when the unpredictability of everyday life and interaction with people comes into play. Which brings us to the second possible approach, the fundamental one of machine learning and artificial intelligence: collecting a gigantic amount of context information that is given to the machine through millions of examples so that it learns how something is done. This also has a great limitation, according to Palli: it requires such an amount of evidence, such an amount of data, that it may be possible to collect if you are researching, for example, in the field of economics, on which there are countless references available; But for a robotic application like the one they have in mind, it neither exists nor is it within reach, since collecting that information is very expensive: it takes time, equipment...

"In a way, we are trying to define what the robot's plan is and, based on very few experiments, collect enough data to adapt the plan to the real conditions of the environment. But, in addition, we want the human being, the person, to be actively present in that environment, to interact with it, to also teach it", summarizes the project coordinator.

Palli, 46, speaks with the aplomb of someone who has been looking for solutions to create more complete and independent robots for several decades and does so from an institution that was a pioneer in this field; Specifically, in the development of anthropomorphic robotic hands that were increasingly reliable, simple and cheap. Its history can be traced through the prototypes exhibited in several showcases in the university's robotics laboratory: from the first, funded in 1988 by IBM Italia – a rudimentary device with two parallel fingers and an opposable thumb controlled by artificial muscles, tendons fed by several motors and a system of calculators and electronic equipment, to one of the most recent: a robotic hand, moved by a spectacular arm with braided string tendon transmission, with which researchers have already begun to take the first steps to introduce tactile sensors. On all that basis of previous knowledge – thanks to everything that was achieved, but also thanks to everything that was discarded along the way – we are now moving towards a new generation.

Prototype created some years ago by the Laboratory of Automation and Robotics of the University of Bologna.Manuel Vázquez

Detail of the Robotics Laboratory.Manuel Vázquez

The project directed by Palli goes, in any case, far beyond the robotic hand and prostheses in general. Not only for domestic robots, but for the applications for the industry in which they are working. On the one hand, they collaborate with the British online supermarket chain Ocado in a robot that is capable of removing fruits from a large box to place them in smaller ones, for which it is important that you can change the grip force depending on whether you do it with apples, with oranges or strawberries. In addition, they are trying something even more difficult for the automotive industry (the Slovenian components company Elvez is another partner in the consortium): assembling cables and connectors, i.e. deformable materials whose handling requires fairly fine skills.

They're on it. And the next leap forward will depend on breaking down barriers in many fields of science: from mathematics — to best represent the robots' pre-assigned plan and how it perceives its environment — to parallel technological developments — touch sensors are still in the process of advancing and improving. But Palli is especially concerned about how the interaction between machine and human can be, for which the project has the support of a team of psychologists.

The role that touch plays in this path will depend to a large extent on its development, which is currently very rudimentary. But let's not forget the words of British-American anthropologist Ashley Montagu about the power of this sense, perhaps the most complex of all, for human communication. In his classic book Touch: The Importance of Skin in Human Relationships, the Princeton University professor who died in 1999 also made statements like these: "In the evolution of the senses, touch was undoubtedly the first to exist. Touch is the father of our eyes, ears, nose and mouth. [...] as Bertrand Russell pointed out some time ago, it is the sense that gives us a sense of reality. "Not only our geometry and our physics, but every conception of what exists outside of us is based on the sense of touch."

Gianluca Palli, IntelliMan project coordinator.Manuel Vázquez

"I agree that touch will become increasingly important, because it's what completely changes the game when you interact with objects and the environment. I think it will really open up the ability of robots to interact with the environment in an effective way," Palli said. At the moment, in any case, what he and his team propose for the relationship between human and machine to work well is a kind of shared autonomy, in which the person who begins to relate to the machine treats it like a child who has to be taught and gradually giving him more room to make his own decisions. "That means the robot starts from a point where it is completely dependent on the human, because it doesn't know how to do something. And so the human will teach it what to do, which gives the robot more autonomy, because it will be able to do it independently in the future," explains the researcher. This process will also help the person trust the robot.

When Roberto Meattini is asked in a social context what he does, he usually chooses the short version: "I work in the interaction of robots and humans; I study how to connect humans with robots and vice versa." Because he insists that it is not about creating robots, but about that relationship, which is in fact what sustains the theoretical bases of the main disciplines on the engineering of automatic systems with which they work, explains this engineer who mentioned in his doctoral thesis that toy robot that his parents gave him on his fifth birthday. He is also convinced of the importance of touch in the improvement path: "The ability to sense and respond to touch will allow robots not only to better perform handling tasks and autonomous operations, but also to understand human intentions and be useful to their needs, leading to more intuitive and effective interactions."

Sounds good, no doubt. But at this point, as much as we are talking about robots with basic capabilities in principle, it is inevitable to face the fears that emerge with increasing violence around the possible undesired consequences of the development of artificial intelligence; After all, we are talking about robots capable of learning new things. "We apply ethical principles to all our research," Palli replies when the issue is raised. But do they have in mind the idea of transmitting some kind of ethics to robots? "No, in principle we are not in that. We are working on security, that all the work they do is done safely, "he answers. He adds: "We could say that we are working on a very basic ethical requirement."

75% discount

Subscribe to continue reading

Read without limits

Read more

I'm already a subscriber