LatticeFlow, a spinout from ETH Zurich, has introduced a pioneering framework aimed at assessing AI models' compliance with the EU AI Act. This initiative is crucial as the EU's regulatory framework for AI applications begins to take shape. The framework, dubbed Compl-AI, offers an open-source validation tool and evaluates major large language models against 27 benchmarks. The results reveal mixed performance across various compliance areas, highlighting the need for improvements in fairness and robustness. LatticeFlow calls for collaboration from the AI community to refine and enhance this evaluation tool.