Enterprises are increasingly relying on AI models to ensure the functioning and reliability of their applications, highlighting the discrepancies between model-driven and human evaluations. To address this, LangChain introduced Align Evals to LangSmith, aiming to bridge the gap between large language model evaluators and human preferences, reducing noise. Align Evals allows LangSmith users to create their own LLM-based evaluators and adjust them to better align with company preferences.
LangChain noted, “A major challenge teams report is that evaluation scores don’t match expected human responses, causing noisy comparisons and wasted time on false signals,” as stated in a blog post. LangChain is among the limited platforms to incorporate LLM-as-a-judge evaluations directly into the testing dashboard.
Align Evals is based on a paper by Amazon principal applied scientist Eugene Yan, who proposed a framework for an app called AlignEval to automate parts of the evaluation process. Align Evals enables organizations to iterate on evaluation prompts, compare human and LLM-generated alignment scores against a baseline.
LangChain described Align Evals as the first step in helping build better evaluators, with plans to integrate analytics for tracking performance and automating prompt optimization. Users start by identifying evaluation criteria, selecting data for human review to provide a comprehensive view for evaluators, and manually assigning scores for baseline benchmarks.
Developers create initial prompts for the model evaluator, iterating based on human grader alignment results. LangChain encourages ongoing improvement of evaluator scores, using clearer criteria for better accuracy.
As enterprises increasingly employ evaluation frameworks to assess AI system reliability, behavior, alignment, and auditability, being able to indicate a model’s performance boosts confidence in deploying AI applications and facilitates model comparisons. Companies like Salesforce and AWS offer tools for evaluating performance, with Salesforce’s Agentforce 3 and AWS’s Amazon Bedrock facilitating performance assessments. OpenAI also provides model-based evaluation services.
Meta’s Self-Taught Evaluator shares the LLM-as-a-judge concept used by LangSmith but hasn’t been integrated into any application-building platforms yet. With growing demand for easier and customized performance evaluations, more platforms are expected to provide integrated and tailored evaluation options for enterprises.