Mismatch Annotations / Labels Space between `gt_labels` and paper supplement materials

[OuYaozhong]

Hi, STF teams and @MartinHahner @MarioBijelic ,

It seems that the provided annotations space is (ref)

- Obstacle
   - Vehicle
      - PassengerCar
      - Large Vehicle
      - RidableVehicle
   - Pedestrian
   - DontCare

However, In the supplement materials of your published paper,
in Figure 2, or the 2nd paragraph of Sec 1.2, you wrote that,

We label objects according to the following types/classes: Pedestrian, Truck, Car, Cyclist, and DontCare.

Meanwhile, the Figure 2 is also do the statistics based on the classes named Pedestrian, Truck, Car, Cyclist, and DontCare.

The annotations space between the given data and the one in supplement materials is different.

Is there any mistake, or there is somewhere misunderstanding ?

Thanks and hope for your reply.

[OuYaozhong]

One more thing,

It seems that there are some annotations named xxx_is_group, and there are a few is labeled as train, person, in the annotations file cam_left_labels_TMP.

For example,

1. Pedestrian_is_group, 1134 objects, one of them can be found at gt_labels/cam_left_labels_TMP/2018-02-03_20-53-15_00200.txt

2. Vehicle_is_group, 7 objects is found, one of them can be found at gt_labels/cam_left_labels_TMP/2018-02-06_16-27-03_00000.txt

3. PassengerCar_is_group, 238 objects, one of them can be found at gt_labels/cam_left_labels_TMP/2018-02-03_20-48-35_00500.txt

4. RidableVehicle_is_group, 4 objects, one of them can be found at gt_labels/cam_left_labels_TMP/2018-02-05_12-37-24_00100.txt

5. LargeVehicle_is_group, 1 object, which can be found at gt_labels/cam_left_labels_TMP/2018-02-10_13-23-06_00100.txt

6. train, 1 object, which can be found at gt_labels/cam_left_labels_TMP/2018-02-06_16-40-12_00300.txt

7. person, 3 objects, all of them is located at gt_labels/cam_left_labels_TMP/2018-02-06_16-40-12_00300.txt

So, what is the meaning of the xxx_is_group, train, and person, which is not mentioned in the paper (include supplement) and the README.md ?

[MarioBijelic]

Hi @OuYaozhong ,

thank you for pointing out those differences. Those are changes in-between the paper acceptance and data publication. For the published dataset we merged the classes motorcyclist, cyclist and everything that can have a rider on top. We wanted to minimise the confusion between those classes.
The substring '_is_group' tough is a class specific don't care label for those cases where a group of objects existed that could not be labeled individually and is not counted in the individual class histograms. We use them during evaluation as don't care label for that specific class only and don't supervise object detectors for that class in those regions.

Best,
Mario

[OuYaozhong]

Hi, @MarioBijelic

Thank you very much for your quick reply.

So, based on the given explanation, you means that the xxx_is_group refer to the xxx (prefix) class in fact, while multiple xxx objects are overlapped each other, making hard to separate and precisely located them.

And if I understand correctly, in the relevant implementation described in your publication, handling objects (xxx) with this suffix (i.e., _is_group) in a way that deliberately avoids detector to detected it.

Then, not only to reduce misunderstanding and ambiguity, but also some additional classes which are not labeled before is added to annotation space, such as motorcyclist (which is not in the original annotation space Pedestrian, Truck, Car, Cyclist, and DontCare, and seems different from cyclist depending on your response

we merged the classes motorcyclist, cyclist and everything that can have a rider on top

), your team remaps the original annotation space to the new annotation space, right ?

Annotation Space Remapping (Original + Addition = New)

Original	Additional	→	New
Car	Other Cars that can take human	→	PassengerCar
Truck	Other vehicles that can carry goods	→	LargeVehicle
Cyclist	Others that can have a rider on top	→	RidableVehicle
Pedestrian		→	Pedestrian
DontCare	Overlapped object with same classes (xxx) while unable to seperate (i.e., xxx_is_group)	→	DontCare (include xxx_is_group)

🌟Question 1 -> Did I restate or understand correctly above ?

🌟Question 2 -> What is the meaning of train and person ? Sorry for repeatedly asking since it has not been reply by the above response.

As my statistics, the train and person objects are shown in cam_left_labels_TMP by only 1 times and 3 times.

Meanwhile, both of these 4 (1 train + 3 persons) are also located at the same frame (or said as annotation file) (i.e., gt_labels/cam_left_labels_TMP/2018-02-06_16-40-12_00300.txt)

Though not a major problem, it seems strange to some extent.

---

👋Since we need to do some comparisons with previous paper, whose results are given based on the original annotations space, the precise understanding of the mapping mechanism is helpful for me.

Thanks in advance, and appreciate for the discussion.

[MarioBijelic]

Hi, yes you are right for 1.
You can plainly map the is_group labels into the don't care category. In case you want to have a more fine granular evolution with higher margins, consider mapping the is_group class only as don't care for object from the same category. That means a pedestrian_is_group is only a don't care category for pedestrians but not for cars.
You actually found a bug in the annotation files. person should be pedestrian and train I have to lookup. Can you visualise the corresponding label in the viewer?

[OuYaozhong]

Hi, @MarioBijelic,

For Question 1, thanks for your reply. I have totally understood the _is_group. This class-specific DontCare seems like a trick, and different projects will have different way to deal with it.

---

For Question 2, you remind me to check this error.

(base) user@hostname:~/Downloads/STF/gt_labels/cam_left_labels_TMP$ cat 2018-02-06_16-40-12_00300.txt
PassengerCar 0.00 0 -1 984.80 528.61 1056.50 580.46 1.40 1.84 4.10 0.96 1.98 66.62 -1.519 0.000 0.000 0.052 1.00 0.0000000000 0.0000000000 0.0260057048 0.9996617945 True True True False
PassengerCar 0.00 2 -1 982.21 513.15 1054.65 574.94 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
RidableVehicle 0.00 0 -1 556.89 522.09 587.32 586.41 1.67 0.61 1.76 -11.28 2.08 63.12 -1.548 0.000 0.000 0.023 1.00 0.0000000000 0.0000000000 0.0115202734 0.9999336394 True True True False
RidableVehicle 0.00 0 -1 621.27 528.53 634.64 560.18 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
LargeVehicle 0.00 0 -1 664.54 443.58 864.93 610.47 3.04 2.73 17.92 -4.36 1.79 50.37 -1.530 0.013 0.008 0.041 1.00 0.0066331758 0.0040289589 0.0205607333 0.9997584832 True True True False
Pedestrian 0.00 0 -1 1066.09 519.98 1086.85 568.47 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
Pedestrian 0.00 0 -1 328.98 519.70 365.89 603.61 1.71 0.63 0.52 -13.35 1.97 49.04 1.532 0.001 0.007 3.103 1.00 -0.0033538034 0.0005710441 0.9998065408 0.0193728373 True True True False
Pedestrian 0.00 0 -1 1249.81 495.95 1304.73 600.63 1.66 0.65 1.09 4.66 1.48 38.04 -1.625 0.005 0.000 -0.054 1.00 0.0026236408 -0.0000709778 -0.0270431988 0.9996308203 True True True False
train 0.00 -1 -1 652.00 439.00 868.00 615.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
person 0.00 -1 -1 322.00 530.00 353.00 598.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
person 0.00 -1 -1 571.00 510.00 600.00 579.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
person 0.00 -1 -1 1273.00 503.00 1305.00 599.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
(base) user@hostname:~/Downloads/STF/gt_labels/cam_left_labels_TMP$ cat ../gated_labels_TMP/2018-02-06_16-40-12_00300.txt 
PassengerCar 0.00 0 -1 703.00 301.00 770.00 353.00 1.40 1.84 4.10 2.02 2.57 66.55 -1.519 0.000 0.000 0.052 1.00 0.0000000000 0.0000000000 0.0260057048 0.9996617945 True True True False
PassengerCar 0.00 2 -1 707.87 275.85 768.68 336.96 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
RidableVehicle 0.00 0 -1 272.00 292.00 307.00 355.00 1.67 0.61 1.76 -10.29 2.54 63.27 -1.548 0.000 0.000 0.023 1.00 0.0000000000 0.0000000000 0.0115202734 0.9999336394 True True True False
RidableVehicle 0.00 0 -1 343.72 292.64 355.28 331.86 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
LargeVehicle 0.00 0 -1 368.00 212.00 595.00 385.00 3.04 2.73 17.92 -3.59 2.19 50.41 -1.530 0.013 0.008 0.041 1.00 0.0066331758 0.0040289589 0.0205607333 0.9997584832 True True True False
Pedestrian 0.00 0 -1 781.08 290.59 804.89 337.26 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
Pedestrian 0.00 0 -1 55.00 287.00 90.00 369.00 1.71 0.63 0.52 -12.61 2.28 49.23 1.532 0.001 0.007 3.103 1.00 -0.0033538034 0.0005710441 0.9998065408 0.0193728373 True True True False
Pedestrian 0.00 0 -1 960.00 271.00 1013.00 375.00 1.66 0.65 1.09 5.21 1.84 37.92 -1.625 0.005 0.000 -0.054 1.00 0.0026236408 -0.0000709778 -0.0270431988 0.9996308203 True True True False
(base) user@hostname:~/Downloads/STF/gt_labels/cam_left_labels_TMP$ 

From above, we can see that the annotation of the same frame 2018-02-06_16-40-12_00300.txt, are different between cam_left_labels_TMP and gated_labels_TMP.

According to my guess and the correspondence above (use the same file 2018-02-06_16-40-12_00300.txt under the gated_labels_TMP to correct the one in cam_left_labels_TMP), the train may be the LargeVehicle, and person may be indeed the Pedestrian, as what you said.

---

Also, to rule out that the bug was caused by my accidental modification, I gave the md5 of the original .zip file as well as re-unzipped it, and found that the problem remains.

(base) user@hostname:~/Downloads/STF$ mkdir tmp
(base) user@hostname:~/Downloads/STF$ cd tmp/
(base) user@ hostname:~/Downloads/STF/tmp$ ls -a
.  ..
(base) user@hostname:~/Downloads/STF/tmp$ md5sum ../cam_left_labels_TMP.zip 
3b9437abec53bea792bae4a51bce7293  ../cam_left_labels_TMP.zip
(base) user@hostname:~/Downloads/STF/tmp$ unzip -q ../cam_left_labels_TMP.zip 
(base) user@hostname:~/Downloads/STF/tmp$ cd cam_left_labels_TMP/^C
(base) user@hostname:~/Downloads/STF/tmp$ ls && cd cam_left_labels_TMP/
cam_left_labels_TMP
(base) user@hostname:~/Downloads/STF/tmp/cam_left_labels_TMP$ cat 2018-02-06_16-40-12_00300.txt
PassengerCar 0.00 0 -1 984.80 528.61 1056.50 580.46 1.40 1.84 4.10 0.96 1.98 66.62 -1.519 0.000 0.000 0.052 1.00 0.0000000000 0.0000000000 0.0260057048 0.9996617945 True True True False
PassengerCar 0.00 2 -1 982.21 513.15 1054.65 574.94 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
RidableVehicle 0.00 0 -1 556.89 522.09 587.32 586.41 1.67 0.61 1.76 -11.28 2.08 63.12 -1.548 0.000 0.000 0.023 1.00 0.0000000000 0.0000000000 0.0115202734 0.9999336394 True True True False
RidableVehicle 0.00 0 -1 621.27 528.53 634.64 560.18 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
LargeVehicle 0.00 0 -1 664.54 443.58 864.93 610.47 3.04 2.73 17.92 -4.36 1.79 50.37 -1.530 0.013 0.008 0.041 1.00 0.0066331758 0.0040289589 0.0205607333 0.9997584832 True True True False
Pedestrian 0.00 0 -1 1066.09 519.98 1086.85 568.47 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 True True False False
Pedestrian 0.00 0 -1 328.98 519.70 365.89 603.61 1.71 0.63 0.52 -13.35 1.97 49.04 1.532 0.001 0.007 3.103 1.00 -0.0033538034 0.0005710441 0.9998065408 0.0193728373 True True True False
Pedestrian 0.00 0 -1 1249.81 495.95 1304.73 600.63 1.66 0.65 1.09 4.66 1.48 38.04 -1.625 0.005 0.000 -0.054 1.00 0.0026236408 -0.0000709778 -0.0270431988 0.9996308203 True True True False
train 0.00 -1 -1 652.00 439.00 868.00 615.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
person 0.00 -1 -1 322.00 530.00 353.00 598.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
person 0.00 -1 -1 571.00 510.00 600.00 579.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
person 0.00 -1 -1 1273.00 503.00 1305.00 599.00 -1.00 -1.00 -1.00 -1000.00 -1000.00 -1000.00 -1.000 -1.000 -1.000 -1.000 1.00 -1000.0000000000 -1000.0000000000 -1000.0000000000 -1000.0000000000 False False False False
(base) user@hostname:~/Downloads/STF/tmp/cam_left_labels_TMP$ cd ..

I copy and highlight the MD5 of the cam_left_labels_TMP.zip below, which is downloaded based on these two answer1 answer2.

(base) user@hostname:~/Downloads/STF/tmp$ md5sum ../cam_left_labels_TMP.zip 
3b9437abec53bea792bae4a51bce7293  ../cam_left_labels_TMP.zip

According this answer, it seems only affect the image instead of lidar modal, beacause their height, length, width are all -1.00, and posx, posy, posz are all -1000.00.

It's not a big deal, thanks for informing me that the place is an error.

Best Regards

[MarioBijelic]

I will fix the currently uploaded txt files next week. For now please rename the person manually to Pedestrian and Train to DontCare. Thank you for pointing out this annotation mistake.

[OuYaozhong]

Glad to hear this unexpected discovery helped. 👏