Amazon SageMaker Inference introduces prefix-aware routing for improved LLM performance.
“In our benchmarks on Llama 3.1 70B, this reduced P50 TTFT by up to 77 percent.”
LLM
10 tracked signals on LLM.
LangSmith CLI enables evaluation of user frustration with LLMs.
“It'll then produce some feedback, which gets assigned on the trace, and consists of a score plus a reasoning.”
LLM inference costs are rising amid growing AI usage.
“your AI needs to be put on a diet and everyone needs to start auditing and budgeting for token usage.”
Speculative decoding can accelerate LLM inference while maintaining accuracy.
Agent loops amplify LLM hallucinations by compounding errors across downstream steps
“one hallucinated fact in step two can poison step three, step four and everything downstream”
AWS demonstrates Amazon Bedrock-powered metadata harmonization workflow with human-in-the-loop validation
Yahoo replaced Word2Vec with Amazon Bedrock LLMs for DSP search keyword expansion
LLM hallucinations stem from their probabilistic next-token nature and cannot be fully eliminated, only mitigated.
“This minimizes hallucination, but it doesn't eliminate it.”
Databricks has built a proprietary inference platform serving frontier AI models at scale
“At Databricks, we've built a unique inference platform that serves every frontier”
LangChain defines an AI agent as an LLM looping through decide, act, reason, repeat.
“That loop, decide, act, reason, repeat. That's what makes an agent.”