CherryPicker: Semantic Skeletonization and Topological Reconstruction of Cherry Trees - Meyer, Lukas; Gilson, Andreas; Scholz, Oliver; Stamminger, Marc (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 2023, 2023)
In plant phenotyping, accurate trait extraction from 3D point clouds of trees is still an open problem. For automatic modeling and trait extraction of tree organs such as blossoms and fruits, the semantically segmented point cloud of a tree and the tree skeleton are necessary. Therefore, we present CherryPicker, an automatic pipeline that reconstructs photo-metric point clouds of trees, performs semantic segmentation and extracts their topological structure in form of a skeleton. Our system combines several state-of-the-art algorithms to enable automatic processing for further usage in 3D-plant phenotyping applications. Within this pipeline, we present a method to automatically estimate the scale factor of a monocular reconstruction to overcome scale ambiguity and obtain metrically correct point clouds. Furthermore, we propose a semantic skeletonization algorithm build up on Laplacian-based contraction. We also show by weighting different tree organs semantically, our approach can effectively remove artifacts induced by occlusion and structural size variations. CherryPicker obtains high-quality topology reconstructions of cherry trees with precise details.
Keywords: Point Cloud Skeletonization; Plant Phenotyping; 3D Reconstruction; Digital Horticulture

Enhancement of Vision-Based 3D Reconstruction Systems Using Radar for Smart Farming - Meyer, Lukas; Gedschold, Jonas; Wegner, Tim Erich; Del Galdo, Giovanni; Kalisz, Adam (2022 IEEE INTERNATIONAL WORKSHOP ON METROLOGY FOR AGRICULTURE AND FORESTRY (METROAGRIFOR), 2022)
Digital field recordings are central to most precision agriculture systems since they can replicate the physical environment and thus monitor the state of an entire field or individual plants. Using different sensors, such as cameras and radar, data can be collected from various domains. Through the combination of radio wave propagation and visible light phenomena, it is possible to enhance, e.g., the optical condition of a fruit with internal parameters such as the water content. This paper proposes a method to correct sensor errors to perform data fusion. As an example, we observe a watermelon with camera and radar sensors and present a system architecture for the visualization of both sensors. For this purpose, we constructed a handheld platform on which both sensors are mounted. In our report, the radar is analyzed in terms of systematic and stochastic errors to formulate an angle-dependent mapping function for error correction. It is successfully shown that camera and radar data are correctly assigned with a watermelon used as a target object, demonstrated by a 3D reconstruction. The proposed system shows promising results for sensor overlay, but radar data remain challenging to interpret.<br
CRIS-Team WoS Importer:2023-03-24
Keywords: Smart agriculture; Three-dimensional displays; Systematics; Systems architecture; Radar; Sensor phenomena and characterization; Sensor fusion

For5G: Systematic approach for creating digital twins of cherry orchards - Meyer, Lukas; Gilson, Andreas; Uhrmann, Franz; Weule, Mareike; Keil, Fabian; Haunschild, Bernhard; Oschek, Joachim; Steglich, Marco; Hansen, Jonathan; Stamminger, Marc; Scholz, Oliver (43. GIL-Jahrestagung, Resiliente Agri-Food-Systeme, 2023)
We present a systematic approach for creating digital twins of cherry trees in orchards as part of the project “For5G: Digital Twin”. We aim to develop a basic concept for 5G applications in orchards using a mobile campus network. Digital twins monitor the status of individual trees in every aspect and are a crucial step for the digitalization of processes in horticulture. Our framework incorporates a transformation of photometric data to a 3D reconstruction, which is subsequently segmented and modeled using learning-based approaches. Collecting objective phenotypic features from individual trees over time and storing them in a knowledge graph offers a convenient foundation for gaining new insights. Our approach shows promising results at this point for creating a detailed digital twin of a cherry tree and ultimately the entire orchar
Keywords: 5G; cherry tree; deep learning; digital horticulture; digital twin; knowledge graph; orchard; phenotyping; photogrammetry; precision farming; UAV

FruitNeRF: A Generalized Framework for Counting Fruits in Neural Radiance Fields - Meyer, Lukas; Gilson, Andreas; Schmidt, Ute; Stamminger, Marc (2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024)
We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit count. The use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant fruit. We evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and mangoes. Additionally, we assess the performance of fruit counting using the foundation model compared to a U-Ne

PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DOF Object Pose Dataset Generation - Meyer, Lukas; Erich, Floris; Yoshiyasu, Yusuke; Stamminger, Marc; Ando, Noriaki; Domae, Yukiyasu (2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024), 2024)
We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DOF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting. Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. PEGASUS allows the omposition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that captures images from both object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.
Keywords: Novel View Synthesis; Robotics; Dataset Generation; Pose Estimation; Gaussian Splatting; Sim-to-real transfer