As reported by Reuters, the tool, developed by Swiss startup LatticeFlow AI in partnership with research institutes ETH Zurich and INSAIT, evaluates generative AI models on technical robustness, cybersecurity, and bias.
Models developed by major players like Meta, OpenAI, Alibaba, and Anthropic were tested, with scores ranging from 0 to 1 across dozens of categories. While most models received average scores of 0.75 or above, shortcomings were apparent in crucial areas.
OpenAI’s GPT-3.5 Turbo scored 0.46 for discriminatory output, and Meta’s Llama 2 13B Chat scored 0.42 for vulnerability to “prompt hijacking” attacks. Alibaba’s Qwen1.5 72B Chat scored even lower, with 0.37 for discriminatory output.
The EU AI Act, which is being phased in over the next two years, will impose strict rules on general-purpose AI models to ensure safety, fairness, and transparency. Companies that fail to comply risk fines of up to €35 million or 7% of global annual turnover.
Despite these mixed results, the tool provides valuable insights for AI developers, helping them identify areas for improvement as they prepare for the new regulations. The LLM Checker is freely available for developers to test their models for compliance.
As EU lawmakers finalize the details of the AI Act, which is set to be enforced by 2025, tools like the LLM Checker offer a preview of the challenges that lie ahead for tech giants in aligning with the bloc’s stringent new rules.
