Estimating object hardness with GelSight
Humans easily estimate object hardness upon contact, and we want to make the robot to have the similar capability. We use GelSight to contact the target objects in loosely controlled conditions: either a human tester holding the sensor and pressing on the objects, or an open-loop robot gripper squeezing on the objects. In both situations, the contact trajectory and velocity are unknown and incontrollable, but GelSight can still get a sequence of images of the contact states. In the sequence, the contact force increases as press proceeds, and the contact area increases. Soft objects will deform under the force, and the softer the object is, the more it deforms, making the geometry of the contact surface flatter.
Here is a demo video shows how GelSight obtains different data when pressing on hemispheres with different hardness levels:
GelSight measures the gradient of contact surface by the change of image colors, and the contact force by the motion field of the markers. For the hemispherical samples, using the hand-crafted features of the color change and marker motion, we can predict the hardness of objects with R squared higher than 0.98.
To measure the hardness of objects of unknown shapes, we design a neural network combining CNN and RNN: the CNN (VGG16) measures the spatial information of the images from GelSight, and RNN (LSTM) compares the 5 representative images in the pressing sequence. We collect a dataset of sample objects, either made by silicone with known hardness, or the natural objects that human can compare their hardness. The silicone samples include basic shapes, like hemisphere or cylinders with different radii, flat surfaces, edges and corners; it also include some irregular shapes, such as the shape of counting cups, or sorted chocolates. The examples of dataset is shown in the following image:
The estimation of the neural network on measuring the objects with seen and simple shapes with unseen hardness has an R squared of 0.96; the estimation of the objects with seen shapes but under different loading procedures has an R squared of 0.80.
The neural network can estimate the hardness of natural objects as well. This could be very useful in many ways. For example, it helps a robot to feel the hardness of different fruits!