Taxim: An Example-based Simulation Model for GelSight Tactile Sensors

Taxim is a realistic and high-speed simulation model for a vision-based tactile sensor, GelSight. Our simulation framework is the first to incorporate marker motion field simulation together with the optical simulation. We simulate the optical response to the deformation with a polynomial lookup table. This table maps the deformed geometries to pixel intensity sampled by the embedded camera. We apply the linear elastic deformation theory and the superposition principle to simulate the surface markers’ motion that is caused by the surface stretch of the elastomer. The example-based approach requires less than 100 data points from a real sensor to calibrate the simulator and enables the model to easily migrate to other GelSight sensors or their variations.

Taxim can simulation the illumination characteristics based on contact geometry, including fine textures. It can also simulation the marker motion under external normal or shear loads. We illustrate the objects (first row), real data collected from a GelSight (second row), and simulated tactile images with Taxim (last row).

Paper link: https://arxiv.org/pdf/2109.04027.pdf

Codes link: https://github.com/CMURoboTouch/Taxim

Bibtex:

@article{si2022taxim,
  title={Taxim: An Example-based Simulation Model for GelSight Tactile Sensors},
  author={Si, Zilin and Yuan, Wenzhen},
  journal={IEEE Robotics and Automation Letters},
  year={2022},
  publisher={IEEE}
}

Supplementary Material

We show more data collection and results with Taxim simulation.

Usage tutorial

A usage tutorial of Taxim about how to use Taxim alone and how to use Taxim within a robot simulator.

Optical Simulation

Data collection: To get examples to calibrate the simulator, we use a metal sphere with a diameter of 4 mm and a metal pin with a diameter of 1 mm to collect tactile data with various indentation depths and locations with a GelSight sensor.

Optical setup for data collection. An indentor (metal sphere, pin or objects) is mounted on a vertical linear stage. We control the pressing depth by adjusting the linear stage. Gelsight is placed on a horizontal XYR stage. We control the XY displacement by adjusting the XYR stage.

Calibration: The mapping function from the shape gradients to the image intensities is built with a polynomial table. And we use the shape-location-intensity pairs extracted from the examples to fit the polynomial table.

We calibrate the polynomial mapping table with intensity-shape-location pairs. We also extract shadow masks under different indentation depths.

Simulation: 1) Approximating the soft body deformation. 2)  Mapping the shapes to the tactile intensities. 3) Attaching shadows around the contact area.

We first approximate the soft body deformation with contact. Then we mapp the shape gradients to the image intensities. Finally we attach the shadows around the contact area.

Marker Motion Field Simulation

Calibration: We set up the gelpad model in ANSYS and simulate the simple deformation case with pin pressing. The mutual elastic influence between any two nodes is gained from the ANSYS FEM simulation. 

We set up the gelpad model in ANSYS and simulate the deformation under pin pressing. The elastic mutual influence between two nodes are outputed from the ANSYS simulation as calibration parameters.

Simulation: We apply the linear elastic deformation theory and superposition principle to accumulate the elastic deformation over the surface under certain boundary conditions.

We apply the initial displacements uk on the active nodes, amend the active nodes’ displacement uk by superposition principle and calculate the resultant displacements uj at each node using the superposition principle with the amended motions of the active nodes.

Simulation results

Optical Simulation results. We simulate different contact geometries and compare the simulated tactile images with the real data.
Marker Motion Simulation results. We apply different normal and shear loads on the objects aganist the gelpad.

Taxim with Digit sensor

We apply our Taxim simulation on a digit tactile sensor without shadow attachment. Shadows are more unevenly distributed on the contact surface of the digit sensor, which we are exploring for more generalized solutions.

Taxim simulation on a digit sensor. The illumination matches well between the real and simulated images. However, we have are exploring a good solution for the shadow attachment to simulate the digit’s uneven shadows well.

Taxim with GelSight 1.5 sensor

We apply our Taxim simulation on a GelSight 1.5 tactile sensor.

Taxim simulation on a GelSight 1.5 sensor.

Taxim with Sim-to-Real Shape Mapping

We propose an incremental shape mapping method using a GelSight tactile sensor and a depth camera. The local shape is recovered from tactile images via a learned model trained in Taxim simulation. Through efficient inference on a spatial factor graph informed by a Gaussian process, we build an implicit surface representation of the object. We demonstrate visuotactile mapping in both simulated and real-world experiments, to incrementally build 3-D reconstructions of household objects.

Paper link: https://arxiv.org/pdf/2109.09884.pdf

Video link: https://www.youtube.com/watch?v=nFaWtjanQXQ

Webpage link: http://www.cs.cmu.edu/~sudhars1/shape-map/

Bibtex:

@inproceedings{Suresh22icra,
  author={Suresh, Sudharshan and Si, Zilin and Mangelson, Joshua G. and Yuan, Wenzhen and Kaess, Michael},
  booktitle={2022 International Conference on Robotics and Automation (ICRA)}, 
  title={ShapeMap 3-D: Efficient shape mapping through dense touch and vision}, 
  year={2022},
  pages={7073-7080},
  doi={10.1109/ICRA46639.2022.9812040}}
We perform incremental 3-D shape mapping with a vision-based tactile sensor, GelSight, and an overlooking depth-camera. We combines multi-modal sensor measurements into our Gaussian process spatial graph (GP-SG), for efficient incremental mapping. The depth-camera gives us a partial noisy estimate of 3-D shape, after which we sequentially add tactile measurements as Gaussian potentials into our GP-SG.

Taxim with Sim-to-Real Grasping

We integrate Taxim with a physics simulation engine, PyBullet to enlarge the application field of the tactile simulation to more robotic perception and manipulation tasks. 

Codes link: https://github.com/CMURoboTouch/Taxim/tree/taxim-robot

Taxim intergrated into the PyBullet to realize grasping simulation with tactile feedback.