![\"\"](\"https:\/\/unbabel.com\/wp-content\/uploads\/2023\/10\/xcomet-1.png\")
<\/figure>\n<\/div>\nAutomatic metrics for machine translation evaluation have come a long way, with neural metrics like COMET (Rei et al. 2020)<\/a> and BLEURT (Sellam et al, 2020) <\/a>leading the charge in improving translation quality assessment. These metrics have shown significant advancements, particularly in their ability to correlate with human judgments, surpassing traditional metrics like BLEU (Papineni et al. 2002)<\/a>. However, these metrics, while powerful, have their limitations, as they provide only a single sentence-level score, leaving translation errors hidden beneath the surface.<\/p>\n In an era where large language models (LLMs) have revolutionized natural language processing, researchers have started to employ them for more granular translation error assessment (Fernandes et al. 2023<\/a>, Kocmi et al. 2023)<\/a>. This involves not just evaluating the translation as a whole but also pinpointing and categorizing specific errors, providing a deeper and more insightful view into translation quality.<\/p>\n Here is where XCOMET makes its grand entrance. XCOMET<\/strong> is a cutting-edge, open-source metric designed to bridge the gap between these two evaluation approaches. It brings together the best of both worlds by combining sentence-level evaluation and error span detection capabilities. The result? State-of-the-art performance across all types of evaluation, from sentence-level to system-level, while also highlighting and categorizing error spans, enriching the quality assessment process.<\/strong><\/p>\n But what sets XCOMET apart from the rest? Here\u2019s a closer look at what makes it a game-changer:<\/p>\n Let\u2019s delve into the results to see just how impressive XCOMET truly is. We\u2019ve conducted thorough evaluations, including a comparison with other widely known metrics, including recent LLM-based metrics. Take a look at these two tables:<\/p>\n These tables highlight the exceptional performance of XCOMET. In segment-level evaluations, XCOMET outshines other widely known metrics, including the recent LLM-based metrics such as GEMBA-GPT4<\/a>. When compared to AutoMQM based<\/a> on GPT-4 at the word level, XCOMET maintains its superiority, even when used without reference in a Quality Estimation scenario!<\/p>\n It\u2019s worth noting that AutoMQM based on GPT-4, while impressive at word level, relies on large and costly LLMs, limiting its accessibility and applicability. XCOMET, on the other hand, outperforms GPT-4 and thrives with cost-effective LLMs like GPT-3, making it a versatile choice for researchers and practitioners in the field.<\/p>\n To make XCOMET accessible to the community, we have released two evaluation models: XCOMET-XL, featuring 3.5 billion parameters, and XCOMET-XXL, with an impressive 10.7 billion parameters. These models are available through the COMET framework and the Hugging Face Model Hub:<\/p>\n XCOMET shares its foundational technology with UnbabelQI, with a key distinction being UnbabelQI\u2019s utilization of proprietary MQM annotations<\/strong>, spanning an impressive array of up to 30 languages, and its ability to seamlessly adapt to diverse customer domains and expectations. While XCOMET excels across various evaluation scenarios, UnbabelQI is primarily tailored for reference-free quality estimation<\/strong>. In real-world, \u2018in-the-wild\u2019 situations, reference translations are less common, making UnbabelQI an ideal choice for assessing translation quality in such scenarios.<\/p>\n To learn more about Unbabel\u2019s QE capabilities and LangOps, visit our platform<\/a> and our UnbabelQi demo here<\/a>.<\/p>\n\n
How does XCOMET Compare with Auto-MQM and other Metrics?<\/h2>\n
![\"\"](\"https:\/\/unbabel.com\/wp-content\/uploads\/2023\/10\/xcomet-table-1.png\")
<\/figure>\n<\/div>\n![\"\"](\"https:\/\/unbabel.com\/wp-content\/uploads\/2023\/10\/xcomet-table-2.png\")
<\/figure>\n<\/div>\n\n
Connecting to Unbabel Quality Intelligence<\/h2>\n