Background
As part of the capstone project for my bachelor's degree, my team and I were tasked with creating a device that can help increase the spatial awareness of our paraplegic client. He suffered from complete paralysis of his body below the neck, which meant that his only source of mobility was the power wheelchair. He specifically needed a device that could assist him in visualizing the back half of his wheelchair.
My team took inspiration from the bird's-eye view feature that some car models provide. The working principle behind it is to use multiple wide-angle (fisheye) cameras attached to the sides of the car and computationally combine all video streams to provide a top-down view of the car.
Figure 1: Placement of cameras for bird's-eye view feature.
Figure 2: Bird's-eye view feature being used during parking.
The system is especially useful during parking since it allows you to see everything in close vicinity around your car, which fit our needs exactly. Our client wanted to a system that could help visualize the region close to his back wheels so that he does not accidentally run over someone in a crowded area and the bird's-eye view system was perfectly suited for the job. Additionally, we realized that the bird's-eye view perspective is the most intuitive way of visualizing your surroundings compared to, for example, putting four video streams (front, back, left, right cameras) on a single screen.
Although the system did not look difficult to implement at first, we quickly realized that making a system that is both durable and reliable for long-term use while being attached to a moving wheelchair was extremely difficult. Thus, we decided to divide the team and I was in charge of developing the GUI software.
The purpose of this blog post is to explain the detailed inner workings of the GUI application.
Implementation
For demonstration, I will use the Gazebo simulator.
Figure 3: Robot model used for simulation. The grey circle is the wheel. The red squares are the cameras.
Because our client could see in front of him, we decided that installing 3 cameras at the left, right, and back sides of the wheelchair was sufficient. The image processing pipeline is undistortion --> projection --> combination. The raw image frames from each camera goes through all three steps to produce the bird's-eye view.
Undistortion
Wide-angle cameras have the benefit of being able to capture a wider field of view, but they come with the cost of producing heavily distorted images.
Figure 4: Normal (pinhole) camera in Gazebo.
Figure 5: Wide-angle camera in Gazebo (180 degree field of view).
The reason for using a wide-angle camera instead of a pinhole camera, which has much less distortion, is because pinhole cameras do not provide the field of view required to make the bird's-eye view.
Figure 6: Regions highlighted in red indicate the overlapping fields of view of the wide-angle cameras.
The fields of view of the cameras need to overlap to produce a continuous view of the surroundings.
Because of the inherent distortion in wide-angle cameras, first step in the image processing pipeline is undistorting the raw image frames. This is a fairly simple task using OpenCV's fisheye calibration module.
Figure 7: Finding the corners of a chessboard for calculating the distortion coefficients.
Using the cv::findChessboardCorners() function, you can find the corner points of a chessboard (a corner point is defined as a point of intersection of two white tiles and two black tiles). The coordinates of these corner points contain the information about the severity of the distortion and using the cv::fisheye::calibrate() function, you can find the 4 distortion coefficients that fully describe the distortion mathematically (details about the math is at https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html).
Figure 8: Image in figure 7 undistorted.
Using the 4 distortion coefficients, cv::fisheye::initUndistortRectifyMap(), and cv::remap(), you can produce the undistorted image frames. Notice that the field of view decreased due to the undistortion process. However, it is still sufficient to produce the bird's eye view.
An important note about computing the distortion coefficients is that the chessboard must be shown to the camera in as many orientations as possible for accurate results, which is why I made a moving chessboard in Gazebo.
Figure 9: Moving chessboard in Gazebo to increase accuracy of distortion coefficients.
Projection
The next step in the image processing pipeline is the projecting the image frames into a top-down view. This means transforming the perspective of the cameras from a side view into a top-down view. OpenCV provides the functions cv::getPerspectiveTransform() and cv::warpPerspective() for this specific task.
Figure 10: Clicking on the 4 corners of the rectangular tile for perspective transformation.
The cv::getPerspectiveTransform() function takes in 4 source points and 4 destination points to calculate the 3x3 matrix that describes the intended perspective transform. The 3x3 matrix can then be fed into the cv::warpPerspective() function to produce the image frame with transformed perspective.
Figure 11: Left, right, and back cameras perspective transformed into top-down views.
Combination
With the projection matrices calculated, the final step is to combine all projected images into a single bird's eye view. This is a simple process where you generate a blank image matrix with the specific size that can fit in all the projected images then drawing the images onto it.
Commenti