[Muhammad Faizan Ahmed]

Hello everyone,

I am currently working on a robot-to-robot needle handover project in RoboDK using two JAKA cobots (Zu12 and Zu7). I have attached an OnRobot Eyes camera to the end effector of the Zu12 robot alongside a custom two-finger gripper.

The goal is to perform a synchronized needle handover through a narrow opening. I have already implemented the basic handover sequence using predefined targets and synchronization logic, but I am now exploring vision-based decision making.

My idea is that as Robot A approaches the handover position, the camera monitors Robot B and determines when it has successfully grasped the needle. Only then would Robot A release the needle and move away. Later in the cycle, Robot B would return the needle and Robot A would use vision again to determine when the needle is in a suitable position for re-grasping.

I am also considering a second approach where the camera is mounted above the workcell, providing a top-down view of both robots and the handover area, instead of mounting the camera on the robot itself.

My questions are:

• Is this type of vision-guided robot-to-robot handover possible using the simulated OnRobot Eyes camera in RoboDK?
  
• Would an eye-in-hand setup (camera on the robot) or an overhead camera be more suitable for this application?
  
• Can the simulated camera provide useful depth information for detecting successful grasping and object transfer?
  
• If I want to use machine learning or object detection, can RoboDK be used to generate training data from the simulated camera, or would I need to train and validate using images from the actual OnRobot Eyes camera?
  

I am relatively new to RoboDK and still exploring its vision capabilities, so any guidance, recommendations, or examples would be greatly appreciated.

Thank you.

[Albert]

You can simulate cameras in RoboDK (both color cameras and depth cameras). With RoboDK you can obtain the color image, grayscale image, or depth data through the API.

But the vision logic itself needs to be handled through the API, using the simulated camera image or depth map. RoboDK can provide the simulated camera view, depth data, and the robot motion sequence, but the decision making, object detection, or machine learning inference should be integrated using the API given the images obtained from RoboDK.

If recommend you to use the in-hand camera for the object transfer as you'll have a closer view to adjust the movement.

For machine learning, you can use the simulated camera to generate training images and depth maps, then train and validate your model externally. For final validation, we still recommend testing with images from the real OnRobot Eyes camera, since the real sensor will have lighting, noise, and calibration differences.