A new study by MBZUAI and Meta AI Research delves into comparing common vision models based on metrics beyond ImageNet. The goal of the research is to provide practitioners with insights into the intrinsic qualities of these models and help them make informed decisions when selecting pre-trained models.
The study focuses on four top models in computer vision: ConvNet (ConvNeXt) and Vision Transformer (ViT), trained using both supervised and CLIP methods. These models were chosen because they have comparable parameter counts and ImageNet-1K accuracy across all training paradigms.
Traditionally, models are evaluated based on metrics like ImageNet correctness. However, real-world vision problems often require considering factors like different camera postures, lighting conditions, and occlusions. To address this, the researchers explore various model properties, such as prediction errors, generalizability, calibration, and invariances of learned representations.
The findings of the study reveal that different models exhibit significantly varied behaviors, highlighting the need for comprehensive evaluation beyond a single metric. For example, CLIP models have fewer classification errors compared to their ImageNet performance. On the other hand, supervised models excel in ImageNet robustness benchmarks and calibration. ConvNeXt, compared to ViT, is more texture-biased but performs well on synthetic data.
One significant discovery is that supervised ConvNeXt outperforms CLIP models in terms of transferability and performs well on various benchmarks. This suggests that different models have different strengths depending on the task distribution. The study emphasizes the importance of developing new benchmarks and evaluation metrics that are more context-specific to ensure precise model selection.
In conclusion, when choosing a vision model for specific needs, it is crucial to consider metrics beyond ImageNet and take into account the distinctive qualities exhibited by different models. The study provides valuable insights for practitioners and encourages further exploration in the field of computer vision.
The source of the article is from the blog regiozottegem.be