Amazon has launched an inference optimization toolkit as a feature of Amazon SageMaker to help speed up generative artificial intelligence (AI) operations. This tool allows businesses to balance cost-effectiveness and productivity with the use of various optimization techniques. It offers up to twice the throughput and reduces costs by up to 50% for AI models like Llama 3, Mistral, and Mixtral. For instance, using a Llama 3-70B model enables users to achieve up to ~2400 tokens/sec on an ml.p5.48xlarge instance.
The kit uses AI model optimization techniques such as compilation, quantization, and speculative decoding. The compilation process uses the Neuron Compiler to optimize the model’s computational graph for specific hardware, accelerating runtimes and improving resource utilization. The quantization procedure leverages Activation-aware Weight Quantization (AWQ) to decrease the model size and memory footprint while maintaining quality. In speculative decoding, a faster draft model predicts candidate outputs in parallel, enhancing the inference speed for longer text generation tasks.
Models within Amazon SageMaker JumpStart and the Amazon SageMaker Python SDK can easily integrate the optimization toolkit in their procedures. These platforms allow users to either use pre-optimized models or create custom optimizations based on their specific requirements.
Users can deploy pre-optimized models using a simple one-click deployment. The toolkit offers pre-optimized models with superior cost-performance at scale with uncompromised accuracy. They can select a configuration that suits the latency and throughput requirements of their use case.
Those who wish to create custom optimizations can use techniques such as compilation, quantization, and speculative decoding based on the instance types. These techniques can be completed either from SageMaker JumpStart or using the SageMaker Python SDK.
By introducing the inference optimization toolkit, Amazon has significantly simplified the model optimization process. This enables businesses to accelerate the adoption of generative AI and drive better business outcomes. This move marks an exciting development for businesses heavily reliant on generative AI operations, promising improved efficiencies, cost reductions, and the ability to scale operations seamlessly.
