A groundbreaking methodology called Q-ALIGN has emerged in the field of visual content assessment, revolutionizing the way we evaluate images and videos. Developed by researchers from Nanyang Technological University, Shanghai Jiao Tong University, and SenseTime Research, Q-ALIGN represents a significant departure from traditional approaches by educating Large Multi-Modality Models (LMMs) to rate visual content using text-defined rating levels instead of direct numerical scores.
The innovation in Q-ALIGN lies in its ability to convert existing score labels into discrete text-defined rating levels during the training phase. This approach aligns more closely with how human raters evaluate visual content, as they typically work with predefined levels such as ‘excellent,’ ‘good,’ and ‘fair,’ rather than specific numerical scores. By teaching LMMs to understand and use these text-defined levels for visual rating, Q-ALIGN bridges the gap between machine-based assessment and human cognitive processes.
During the inference phase, Q-ALIGN emulates the process of collecting Mean Opinion Scores (MOS) from human ratings. It extracts the log probabilities on different rating levels and calculates the close-set probabilities of each level through softmax pooling. The final score is derived from a weighted average of these probabilities, mirroring the conversion of human ratings into MOS in subjective visual assessments.
Q-ALIGN has demonstrated impressive performance in multiple domains, including image and video quality assessment (IQA and VQA) as well as image aesthetic assessment (IAA). It outperforms existing methods, particularly in scenarios involving novel content types and diverse scoring scenarios, where traditional approaches struggle due to poor out-of-distribution generalization abilities.
This novel methodology has the potential for broad application across various fields, as its ability to generalize effectively to new types of content offers a robust and intuitive tool for accurately assessing a wide range of visual content. Q-ALIGN not only addresses the limitations of existing methods but also opens up exciting possibilities for future advancements in the field of visual content assessment.
The emergence of Q-ALIGN marks a paradigm shift in the way we approach visual content assessment, bringing us closer to aligning machine-based evaluation with human judgment. As researchers continue to push the boundaries of AI capabilities, Q-ALIGN represents a significant step forward in accurately evaluating and understanding visual content.
The source of the article is from the blog elektrischnederland.nl