Tomato Gantry: T-REX and Disease Detection

Tomato Gantry T- Rex Disease Detection

Robot for Extracting DNA Samples from Tomato plants

The Gantry was the platform for multiple projects: T-REX, Disease Detection and ...

T-REX: Robot for Extracting DNA Samples from Tomato plants

This project is a collaboration between CMU and Virginia Tech to automate the process of extracting DNA samples from tomato plants for disease study. The gantry like platform is a custom-built robot with 6 Degrees of Freedom robotic arm, custom imaging systems, gripper, growth lights, and growth bed to grow, monitor, 3d modeling for intelligent robot-canopy interaction and data collection.

Gantry Robot Platform

Research Effort:

My research effort was in the design and development of the 3DoF end effector attached to the Gantry robot. Once a working prototype of the grab-and-cut end effector was made, the objective shifted to 3D reconstruction of plants (on a Gantry robot by taking multi-view images using a stereo camera) and grasping control. The stereo images were processed from the Rosbag files to create disparity images using a Deep learning-based approach called RAFT-Stereo. These disparity images were in turn used to create point cloud models of each disparity. These models were fused to provide a more detailed point cloud, capturing all 6 different views for each plant. Leaf segmentation was done to identify the top N leaves of each plant sorted by size. These were filtered to get individual leaf point clouds. Ransac best-fit plane method was employed to obtain the plane of the leaf and estimate its pose. This pose was used to compute desired end effector pose.

Current Status:

Currently, the pipeline created is able to process rosbag files containing the stereo images, produce the disparity images using RAFT-Stereo and produce the raw point clouds from the disparity. These point clouds undergo pre-processing and filtering. Segmenting the leaves and obtaining the end effector pose is also done. The next stage is to control the robot grasping and refine the process.


Partners: VirginiaTech, who are concerned with disease detection of the sampled leaves

Project Objective: The goal of the project is early-stage disease detection in young tomato plants. The plants are grown on a “Gantry” table, where the Gantry robot is a Cartesian robot capable of traversing in X, Y and Z axes. In addition, a 3DoF end-effector is attached to the Z-axis. This end effector consists of a stereo camera and a “gripper” that can grip and cut tomato leaves for sampling purpose. These leaves are then dropped to a specific location on the table, where (genome sequencing?) is carried out to analyze the leaf for (leaf spot?) disease.

○Sponsor: NIFA
○Partner: Virginia Tech (PI Song Li)
○Overall Goal: Early-stage plant disease detection in young tomato plants using autonomous Gantry robot with image collection, leaf sampling and genome sequencing capabilities
○CMU Goal: Robotic platform capable of autonomous multi-view imaging, point cloud processing and leaf grasping and cutting capabilities
■Build robot: Jim, Abhi
■Build End effector: Abhi
■3D Reconstruction: Sohan
■Leaf Sampling: Abhi, Sohan
○People: Abhi (lead), Jim, Sohan