Mean Average Precision (mAP) for Object Detection

Everything you should know about mean Average Precision

Figure 1: Object Detection (Image credit)

In this article, I am going to give you an idea about the Mean Average Precision (mAP) for object detection. If you have already worked on a computer vision project like object detection, Segmentation and so on, probably you may know about a metric called Mean Average Precision. If not let’s get a complete idea about that.

mAP is a metric used mainly to evaluate the performance of object detection models. In computer vision, mAP is a popular evaluation metric, and many object detection algorithms such as Faster R-CNN, MobileNet SSD, and YOLO use mAP to evaluate their models. Before understanding the mAP, we need to know about the following sub-metrics.

• Intersection over Union (IOU)
• Confusion Matrix

What is Intersection over Union (IOU)?

Intersection over Union (IOU) is a value used in object detection to measure the overlap of the predicted and actual bounding boxes (Ground truth) for the object. When the predicted bounding box values ​​are closer to the actual bounding box values, the intersection is larger. Therefore the IOU value also would be larger. You can simply understand that by using Figure 2.

Figure 2: Intersection over Union (Image credit)

What is the Confusion Matrix?

A confusion matrix is made up of 4 components called True Positives (TP), True Negatives (TN), False Positives (FP) and False Negatives (FN). To define all these components for object detection tasks, we need to define a threshold based on IoU which we have discussed above. Let’s take that threshold as alpha (α). Then,

• True Positive (TP) — Predicted as positive and the truth is indeed positive. (IoU ≥ α)
• False Positive (FP) — Predicted as positive and wrong positive detection. (IoU < α)
• False Negative (FN) — An actual instance that is not detected.
• True Negative (TN) — Actual instance is negative and predicted as negative.

In object detection, True Negative (TN) metric does not apply. Because it includes all possible wrong detections that were not detected. That means there are many possible predictions that should not be detected in an image.

Let’s consider an example for better understanding.

If we take 0.5 as our IoU threshold, and the IoU value for a prediction is 0.3, then we classify that particular prediction as False Positive (FP). On the other hand, if IoU value for a prediction is 0.8, we classify that particular prediction as True Positive (TF).

Figure 3: Explanation of confusion matrix (Image Credit)

So, what are Precision and Recall?

Precision: How many predictions are really positive from all positive predictions?

Figure 4: Precision

Recall: How many cases are predicted as positive from all really positive cases?

Figure 5: Recall

For object detection tasks, we calculate Precision and Recall using the IoU value for a given IoU threshold.

Precision-Recall Curve

High precision indicates that most of the predictions made are correct, while a high recall indicates that we actually found most of the objects. We can visualize the relationship between precision and recall using a precision-recall curve. A precision-recall curve is constructed by calculating and plotting precision against recall for a single model at different confidence scores. Here the curve is drawn by averaging the Precision across all Recall values between 0 and 1. (See Figure 6)

If the prediction model is of high quality, the Precision-Recall curve remains high with increasing Recall values. This would indicate that many objects were detected while not making many errors.

Figure 6: Precision-Recall Curve (Image Credit)

Average Precision (AP)

When it comes to our application it would be better to have a single numeric value to indicate the quality of a model instead of comparing every model by its curve. Therefore AP is the area under the Precision-Recall curve. Here, the higher the Area Under Curve will give the better the performance.

Mean Average Precision (mAP)

However, we calculated the Average Precision only for one class. The mean Average Precision is the average of APs across all classes.

Figure 7: Formal mean Average Precision formula (Image credit)

In some related research papers, you may find abbreviations like mAP@0.5 or mAP@0.75. This notation represents the IoU threshold used to calculate mAP.

The IoU threshold can significantly affect the final value of the mAP. This IoU threshold introduces variability into the model evaluation. This is not a good thing because there can be a scenario where one model performs well under one IoU threshold and lags significantly behind another.

To overcome this weakness, researchers proposed that first measure the AP for each class for different IoU thresholds, and then calculate the average from the obtained scores. Let’s consider an example for better understanding.

Figure 8: PR curve by object class (Image credit)

In the example in Figure 8, the difference between each IoU threshold value is 0.5, This measure is called a step. According to the literature, the abbreviation like mAP@0.6:0.5:0.8 means that the mAP was calculated for each IoU threshold in the range [0.6, 0.65, 0.7, 0.75, 0.8] and each class called A, B and C. Then the obtained values were averaged to get the final result.

In this article, I have summarized all the important points you should know related to mean Average Precision. If you felt that you learn something new by reading this article hit the clap icon below. :)