The Idea
This is one of those ideas that's been sitting in the back of my head for years.
My brother-in-law runs a greenhouse with a couple hundred plants, and a big chunk of the work is just pruning. Over and over. It's repetitive, manual, and honestly feels like the kind of problem a robot should be able to help with.
Back around 2020, I started thinking about building a robotic arm that could identify where to prune and make the cut. At the time, my approach looked like this:
- TensorFlow for training models
- OpenCV for image processing
- PlantCV for plant-specific analysis
- Unity for simulation
That stack made sense then. It was very much computer vision first, robotics second. The idea was to analyze images, figure out what to cut, and then translate that into movement.
Fast forward to now, and I would approach this very differently.
The Stack
Today, I'd build this as a robotics problem first, not just a vision problem.
Software
- ROS 2 + MoveIt 2: motion planning + control
- PyTorch: segmentation model for plant structures
- OpenCV: calibration, geometry, cleanup
- NVIDIA Isaac Sim: simulation + testing
- ONNX Runtime: edge model deployment
Hardware
- Robotic arm: 6-DOF for reach + precision
- Depth cameras: real 3D positioning
- Servo-driven scissors: cutting mechanism
- NVIDIA Jetson: edge compute
PlantCV is still interesting, but I'd treat it as optional. More of a helper for plant structure analysis instead of the core of the system.
The biggest shift is this: I'm not trying to detect leaves in an image anymore. I'm trying to understand a plant in 3D space and safely move a physical object through it. That's a very different problem.
The Approach
This won't start as a fully autonomous system. That's a good way to break plants and waste time.
The first version will be more of a pruning assistant:
- Identify candidate areas to prune
- Highlight them visually
- Let a human confirm
- Eventually guide or position a robotic arm
If that works reliably, then autonomy becomes a real conversation.
For now, this is still in the planning phase. No hardware yet, no code running. Just a much clearer picture of how I'd build it today versus how I was thinking about it a few years ago.
It's one of those projects where the tech finally caught up to the idea.