Abstract

TIVE is a Toolbox for Identifying Video instancne segmentation Errors. By directly operating output prediction files, TIVE can isolate error predictions and weight each type’s demage to mAP, in purpose of distinguishing model charactors. By decomposing localization quality in spatial-temporal dimensions, model’s potential drawbacks on spatial segmentation and temporal association can be clearly revealed. TIVE can also report mAP over instance temporal length for real applications. UAP and TIDE are the formerly introduced error analyzing tools for image-level object recognition. We expect that the analysis of TIVE can give the researchers more insights, guiding the community to promote more meaningful explorations for video instance segmentation.

Error Types

TIVE bin false positives and false negatives produced by a model into 7 types, including general object recognition errors and two spatial-temporal localization errors with specific focus on spatial segmentation and temporal association quality of predicted mask sequence.

Distribution of Errors

Clear pictures are given to show errors produced by models cross all temporal length, effect of each to the evaluation metric is weighted by individually fixing oracle. Users can observe what prevents their model to achieve higher mAP.

Model Summary

Experiments conducted by TIVE show that recognizing short video instances is especially challenging, mAP corss temporal ranges and error distributions for all models selected by our paper are shown below.

Method	Training Frames	Metrics				Error Weights(ΔAP@50)
		mAP	mAPs	mAPm	mAPl	Cls	Dup	Spat	Temp	Both	Bkg	Miss
Mask Track R-CNN	T=2	36.30	10.44	35.87	43.43	7.50	0.00	6.98	6.36	0.24	0.94	6.08
VISOLO	T=3	38.49	8.58	39.31	40.94	12.35	0.00	2.74	8.97	0.09	0.86	5.84
IFC	T=5	40.27	20.44	38.98	40.98	7.03	0.02	6.93	10.99	0.38	1.20	4.72
SeqFormer	T=5	43.37	21.69	40.43	39.20	6.95	0.25	7.69	4.00	0.00	0.97	6.20
Mask2Former	T=2	48.07	25.05	43.85	44.37	5.23	0.42	5.54	6.52	0.06	2.51	2.94

Citation

If you use TIVE in your project, please cite

@inproceedings{
                To Be Updated
}