Large language model (LLM) inference systems have become vital tools in the field of AI, with applications ranging from chatbots to translators. Their performance is crucial in ensuring optimal user interaction and overall experience. However, traditional metrics used for evaluation, such as Time To First Token (TTFT) and Time Between Tokens (TBT), have been found to be inadequate in capturing the complete user experience during real-time interactions. This limitation is felt particularly in real-time applications, where system responsiveness is a key factor in determining user satisfaction.
Current evaluation methods, which also include normalized latency and Time Per Output Token (TPOT), provide valuable insight into aspects of system latency and throughput, but don’t offer a comprehensive view of the user experience. For instance, TTFT and TBT focus on measuring the time attributes of individual token latencies, while neglecting an assessment of end-to-end system throughput. Normalized metrics have also been criticized for obscuring critical issues such as inconsistencies in the token generation rate and scheduling delays.
Addressing this exigent need for a more robust and effective evaluation framework, researchers from Georgia Institute of Technology, Microsoft Research India, and Intel AI Lab have developed Metron. This framework introduces innovative metrics such as the fluidity-index and fluid token generation rate, which better capture the nuances of real-time, streaming interactions with LLM systems. By incorporating temporal elements of token generation, these metrics paint a more accurate picture of user-facing performance.
The fluidity-index metric adds a unique dimension to the assessment process. It sets token-level deadlines based on target TTFT and TBT values, adjusting them relative to the observed performance of the system under evaluation and the prompt length. This approach accounts for scheduling delays and variances in token generation rates, aiming to ensure an uninterrupted and smooth output stream.
Metron proves its efficacy through comprehensive evaluations of both open-source and proprietary LLM inference systems. The framework applies the fluidity-index to measure the fraction of met deadlines, dynamically adjusting deadlines based on real-time performance. This capacity to adapt allows Metron to provide a clear understanding of the system’s proficiency in handling user requests, without jeopardizing system responsiveness.
In comparative evaluations of systems such as vLLM and Sarathi-Serve, the efficacy of Metron’s metrics in capturing subtle but pivotal distinctions in user experience was made evident. Sarathi-Serve, for example, was shown to achieve fewer deadline misses and smoother functionality, with a fluidity-index greater than 0.9 for 99% of requests, and an impressive throughput of 600 tokens per second.
In contrast, the vLLM system displayed a threefold increase in tail TBT due to generation stalls, resulting in an inferior fluidity-index. Such insights highlight the value of Metron’s user-centric evaluation process in revealing performance differences, consequently helping to fine-tune LLM serving frameworks and improving user experiences in real-world applications.
In summary, the newly proposed Metron provides a novel framework for assessing the performance of large language model inference systems. Its innovative metrics, such as the fluidity-index and fluid token generation rate, offer a more especially user-centered approach. Results thus far demonstrate the significant potential of Metron in enhancing large language model serving frameworks for practical, real-world applications.
