HACKATHON - Agent Eval Harness
Wed, 4 Nov 2026, 17:30
230 Blackfriars Rd
London, United Kingdom
Registration on luma.com
Originally organized by The Friendly Data Meetup London. Hi new friends and old! The Friendly Data Meetup Hackathon is back by popular demand! 🎉 After the brilliant turnout (and the even more brilliant demos) at our first hackathon in January, we simply had to do it again. 👩💻 This evening is all about learning, building, and having fun with friendly faces. No pressure, no pitch decks, just inquisitive minds making cool things with data. With perhaps a slight competitive edge for those seeking glory 😉 📍 We're very grateful to Quantspark for kindly hosting us at their venue for the evening! 💡 The challenge: Judge the Agent ⚖️ Dawdle Deliveries' shiny new AI customer support bot goes live next week. Is it actually ready? Your job is to find out, with evidence instead of vibes! AI agents are moving into production, and everyone is asking the same question: how do you know it works? Together, we'll find out by building an evaluation harness from scratch: 🧪 Write the tests: a set of realistic (and sneaky) customer questions ✅ Build the checks: automatic pass/fail checks, plus an AI judge that marks every answer 🔧 Fix it and prove it: improve the bot, then show a before/after scorecard proving your fix worked (without breaking anything else!) The challenge is split into levels, so every team can work at its own pace, and there are bonus rounds (tracing, red-teaming, model shoot-outs) for the speedy ones. We'll finish with quick 3-minute demos: your scorecard, your most interesting failure, and the big question: do you trust your judge? 🤔 You’re welcome to choose your own team of 2-4, or teams will be formed on the evening, so don’t worry if you’re coming solo. ✅ Pre-flight checklist This hackathon is very hands-on, so it's best suited to people who already write code and/or have some working knowledge with data and AI. You'll get the most out of the evening if you: Have decent working experience writing code Are very comfortable with git, Docker and Python environments Are happy to dig in and figure things out with your team (organisers will be around to help, but there won't be step-by-step hand-holding) 🎒 Please bring A laptop (charged!) and charger; preinstalled with:Docker uv An LLM of your choice to work with, either:your own API key for a hosted provider of your choice, or free local models running via Ollama in Docker (no key needed, but allow around 10 GB of free disk space) … Venue: 230 Blackfriars Rd, London, England, GB. Registration is handled by the original organizer. Original event and organizer: https://luma.com/h8zer2ek