[Michael D]

I am working on a project to see if a simulation could be used to train an AI model for a robot in the same way that would be done manually. To test this, I have been developing a simulation environment in RoboDK, however, when trying to record using the virtual camera and OpenCV, I am experiencing extremely low frame rates. RoboDK itself, as well as the 2D camera window run fine, but as soon as I try and either display, or even record without displaying in an OpenCV window, the frame rates drop to around 2 fps for the recording or display. Is there any other way to save the recording data from the RoboDK camera?

[Albert]

You can follow one or more of the following steps to improve the simulation speed and have a faster frame rate with your simulations, including your simulated camera views:

• Ignore displaying small objects: Select Tools-Options-Display and check the option Don’t display objects smaller than and set it to 2% or greater.
  
• Simplify object geometry: Select Simplify Object… in the same Display tab. This action does not alter the appearance of 3D objects. Selecting the root of the station (the first item) will apply the simplification to all objects in the station.
  
• Remove small objects: Select Remove Small Objects… in the same Display tab to remove small objects and triangles. This will delete the objects and triangles smaller than a given size.
  

You can find some tips to improve the frame rate of your simulations here:
https://robodk.com/doc/en/General.html#FastFPS

If you can share your RoboDK project we can help you better.

[Michael D]

The code has gone through several iterations, but here is the project with the current code.

I have done all of these steps and the built in RoboDK camera view runs smoother, but the RDK.Cam2D_Snapshot code does not run faster.

Since posting that, I have found that grabbing the robot joints continuously is giving me the same issue.

[Albert]

It looks like you are using the same Robolink object on both your main process and the camera thread. The Python API is thread safe. However, RoboDK API operations in one thread can be blocked by another thread. Therefore, I recommend you to create a second instance of Robolink in your Camera thread. This should run your loop smoother.

I attached an example modification as a proof of concept:

Code:

class CameraThread(threading.Thread):
    def __init__(self, camera_id=0, fps=10, frame_queue=None):
        super().__init__()
        self.camera_id = camera_id
        self.fps = fps
        self.interval = 1.0 / fps
        self.frame_queue = frame_queue or Queue(maxsize=10)
        self.running = True
        self.RDK_t = rl.Robolink()
        self.RDK_t.Cam2D_Close()
        cv.destroyAllWindows() #destroys all windows still open
        self.cam = self.RDK_t.Item('Vision') # connects to the 2D camera
        self.cam.setParam('Open', 1) # turns on camera

    def run(self):
        mini = 'Camera Feed' # names window Camera Feed

        # Create a live feed
        while self.cam.setParam('isOpen') == '1':
            # Camera Vision
            img_socket = None # will later hold image
            bytes_img = self.RDK_t.Cam2D_Snapshot('', self.cam) # captures a snapshot and returns it as bytes
            if isinstance(bytes_img, bytes) and bytes_img != b'': # checks for error
                nparr = np.frombuffer(bytes_img, np.uint8) # creates an array for the bytes
                img_socket = cv.imdecode(nparr, cv.IMREAD_COLOR) # converts the array into a picture
            if img_socket is None: # breaks to prevent crash if there is no image
                break

            small = cv.resize(img_socket, (0, 0), fx = .5, fy = .5)

            # Stream
            cv.imshow(mini, small)


            if cv.waitKey(1) == 27:
                break  # User pressed ESC, exit
            if cv.getWindowProperty(mini, cv.WND_PROP_VISIBLE) < 1:
                break  # User killed the main window, exit

    def stop(self):
        self.running = False