To develop AI products with confidence, you need a fast, cheap way to tell whether responses are getting better or worse. Reviewing every response by hand is slow and expensive, so evaluation is increasingly handed off to LLM-as-a-judge models. But these judges have a well-known problem: their idea of a good answer often doesn’t match that of a domain expert, someone who knows the product, the rules of the field, and what users actually need.

Today, the LLMaaJ prompt is usually tuned by hand, a couple of cases at a time. At this hackathon, we’re challenging you to automate that process.