Beijing, Dec. 19, 2025 — In a significant step forward for evaluating AI models, the Institute of Artificial Intelligence of China Telecom has launched a pioneering metric called Information Capacity. This innovation changes how large language models (LLMs) are evaluated, moving past conventional comparisons based solely on size. The new method demonstrates that a model’s genuine “talent” is determined not by its scale, but by its efficiency in compressing and processing information in relation to the computational expense.
Information capacity is defined as the ratio of a model’s intelligence to its inference complexity, signifying the inherent density of knowledge within the model. An analogy can be drawn: if a model is a sponge and information is water, then information capacity indicates the sponge’s efficiency at absorbing water. A model is considered “smarter” if it absorbs more water and does so more quickly. Experimental results indicated that models of different sizes within the same series maintain a consistent information capacity. Consequently, this metric allows for equitable efficiency comparisons between different model series and enables precise performance forecasting within a single series.
Led by Professor Xuelong Li, CTO and Chief Scientist of China Telecom and Director of its Institute of Artificial Intelligence, the research team employed information capacity as a measure to gauge an LLM’s “talent.” Inspired by the established link between compression and intelligence, information capacity provides a quantitative assessment of an LLM’s efficiency by weighing its compression performance against its computational complexity. This metric not only uncovers the density of intelligence a model generates per unit of computational cost but also supports the optimal distribution of computing and communication resources within the AI Flow framework.
With inference tasks for large models consuming ever-growing amounts of computational power and energy, accurately assessing inference efficiency is drawing more focus from LLM researchers. Through the information capacity metric, it is now feasible to evaluate the efficiency of large models regardless of their architecture or size. Additionally, this measure can effectively inform decisions during model pre-training and deployment.
This work establishes a quantitative benchmark for the more sustainable development of large models. It also enables the dynamic routing of models of varying sizes to efficiently manage tasks of different complexity levels, a capability particularly pertinent to the Device-Edge-Cloud infrastructure of the AI Flow framework. As edge intelligence rapidly advances, the AI Flow’s hierarchical “Device-Edge-Cloud” network is set to supplant the current mainstream cloud-centric computing model in the coming years.
All associated code and data from this research have been made publicly available on GitHub and Hugging Face. This open-source initiative empowers the broader community to work together towards standardizing the evaluation of large model efficiency.
Codebase:
Dataset:
Leaderboard:
CONTACT: Ziyao Tang tangzy14@chinatelecom.cn
