AI Robot - Object Tracking | Object following Robot using TensorFlow Lite

This projects can also be built using pure image processing techniques. Such techniques work well for a particular object for which image processing logic is written. However, if the object is replaced, the image processing part of the code needs to be modified for the new object. Also, if multiple overlapping objects are present in a frame, then such a techniques would be difficult to implement.

Now, the approach used in this Object Tracking project involves harnessing the benefit of Machine Learning. Which allows the robot to track 90 different types of objects just by changing one line of code.

Object Tracking Overview

Raspberry Pi camera captures the picture of the object. This picture frame is pre-processed to make it compatible with the pre-trained Object Detection ML model. The model is trained on COCO dataset which can detect 90 different objects in a picture frame. The processed frame is fed to the ML model via TensorFlow Lite interpreter to perform inference. The output of inference comprises of 04 parameters. These parameters are:-

1. Location of the object in frame
2. Name of the class the object belongs to
3. Confidence score
4. No of objects in the frame

The first parameter i.e. Location information of the object has been utilsed to build the logic of object tracking. The video feed with live tracking parameters is streamed over LAN and can be viewed on a browser. This real-time monitoring of tracking parameters enables appreciation of the tracking mechanism. A snapshot of the video feed with all the parameters is shown below. The frames (video feed) along with all the overlays is generated using OpenCV.

We need to specify the name of the object that needs to be tracked. It is done in the beginning of the code. It helps the code to discard other objects present in the frame and only consider the specified object (s) for tracking.

When a frame is fed to the ML model, the model returns the names of all objects present in the frame. Now, the code can check whether the specified object is present in the list of objects returned by the model. If present, then the code moves on to the tracking part.

The important aspects involved in the tracking logic are described below.

Locating the object in the frame

The location parameter returned by the model provides the top left and bottom right coordinates of the object. Which are used to calculate the center of the object as shown below. The center is marked with a red dot. The dot follows the center as the object moves.

Defining the tolerance zone

Tolerance zone is the area around the center of the frame within which the center of the object must fall in order to stop the tracking. As long as the red dot (center of the object) is outside the tolerance zone, the robot continues to move and track the object. The tolerance value determines the size of the tolerance zone. 

Calculating the deviation of the center of object from center of the frame

The distance of center of the object from the center of the frame can be calculated along the horizontal and vertical axis. This distance is denoted as 'X' (horizontal deviation) and 'Y' (vertical deviation) on the frame. Both these values are compared with the tolerance value. If any of these values is more than than the tolerance value, the robot continues to track.

Moving the robot to minimise the deviation

The aim of the object tracking algorithm is to move the robot to make both X and Y values less than the tolerance value.

The horizontal deviation or X value can be reduced by moving the robot in Left or Right direction.

The vertical deviation or Y value can be reduced by moving the robot in front or back direction.

Every time a new frame is processed, the X and Y values are calculated. Depending upon the location of the object in the frame, one of the following cases is handled by the code:-

• X and Y < Tol Value

    The robot does not move.

• X > Tol Value and Y < Tol Value

    The robot moves left or right to minimise X.

• X < Tol Value and Y > Tol Value

    The robot moves forward or backward to minimise Y.

• X > Tol Value and Y > Tol Value

    The code further checks which one is greater among X and Y and moves the robot in that direction first.

The camera capture and inferencing part of the object tracking code is similar to the object detection code discussed in previous article. Only the tracking part is elaborated here.

The output returned by the interpreter contains a list of objects present in the frame. This list along with the labels is passed to a function 'track_object()' for further processing.

The function track_object() starts with checking the number of objects present in the frame. If the list is empty, the functions returns without any further processing.

    if(len(objs)==0):
        print("no objects to track")
        ut.stop() #stop the robot motion
        ut.red_light("OFF") 
        arr_track_data=[0,0,0,0,0,0] //initialse the track parameters to zero
        return

It is possible that certain objects are present in the frame but the object that is required to be tracked is absent. Following code checks the same.

    k=0
    flag=0

#go through each object present in the list
    for obj in objs:
        lbl=labels.get(obj.id, obj.id) # check the label of individual object
        k = arr_valid_objects.count(lbl) # non zero count indicates presence of selected object in the frame
        if (k>0):
            x_min, y_min, x_max, y_max = list(obj.bbox) # obtain the location parameters of the object
            flag=1
            break
    
# if the selected object is not present in the list, the flag will remain 0 and further processing halts
    if(flag==0):
        print("selected object no present")
        return
 

Following code will only execute if the selected object is present in the frame. Here, we calculate the center coordinates of the object and it's deviation from the center of the frame.

    x_diff=x_max-x_min
    y_diff=y_max-y_min
    print("x_diff: ",round(x_diff,5))
    print("y_diff: ",round(y_diff,5))
        
       
    obj_x_center=x_min+(x_diff/2)
    obj_x_center=round(obj_x_center,3) 
    
    obj_y_center=y_min+(y_diff/2)
    obj_y_center=round(obj_y_center,3)
    
        
    #print("[",obj_x_center, obj_y_center,"]")
        
    x_deviation=round(0.5-obj_x_center,3) #The horizontal deviation value
    y_deviation=round(0.5-obj_y_center,3) #The Vertical deviation value
        
    print("{",x_deviation,y_deviation,"}")
  

At this stage, all the necessary parameters required for generating motion commands for robot are available. Now, the function that controls the robotic movement is invoked. You can notice that the 'move_robot' function is executed inside a thread. This is required to maintain overall high FPS.

The tracking parameters are stored in an array which will be utilised for updating the tracking GUI.

    #move_robot()
    thread = Thread(target = move_robot)
    thread.start()
    #thread.join()

    #print(cmd)
    
    arr_track_data[0]=obj_x_center
    arr_track_data[1]=obj_y_center
    arr_track_data[2]=x_deviation
    arr_track_data[3]=y_deviation

The function that controls the movement of robot is described below. 

def move_robot():
    global x_deviation, y_deviation, tolerance, arr_track_data
    
    print("moving robot .............!!!!!!!!!!!!!!")
    print(x_deviation, y_deviation, tolerance, arr_track_data)
    
   #don't move if X and Y within tolerance zone
    if(abs(x_deviation)<tolerance and abs(y_deviation)<tolerance):
        cmd="Stop"
        delay1=0
        ut.stop()
        ut.red_light("ON")
    
   # movement starts here
    else:
        ut.red_light("OFF")
        # movement in left / right direction if X > Y
        if (abs(x_deviation)>abs(y_deviation)):
            if(x_deviation>=tolerance):
                cmd="Move Left"
                delay1=get_delay(x_deviation,'l') #more the deviation, more is this value
                
                ut.left()
                time.sleep(delay1)  # delay1 decides how long to move 
                ut.stop()
                
            if(x_deviation<=-1*tolerance):
                cmd="Move Right"
                delay1=get_delay(x_deviation,'r')
                
                ut.right()
                time.sleep(delay1)
                ut.stop()
        # movement in forward / back direction if Y > X 
        else:
            if(y_deviation>=tolerance):
                cmd="Move Forward"
                delay1=get_delay(y_deviation,'f')
                
                ut.forward()
                time.sleep(delay1)
                ut.stop()
                
            if(y_deviation<=-1*tolerance):
                cmd="Move Backward"
                delay1=get_delay(y_deviation,'b')
                
                ut.back()
                time.sleep(delay1)
                ut.stop()
    
    arr_track_data[4]=cmd 
    arr_track_data[5]=delay1

You can download the code from github.