Senior Software Engineer - AI Evaluation / Coding Agents

Experienced software engineers can apply their code-review instincts to assessing how AI coding agents handle tasks in large, practical codebases. The role suits engineers who like judging whether someone else's code is correct and maintainable, and who can explain that judgment in writing. The work supports an AI training company that partners with frontier AI labs.

What you'll do

• Examine code and behavior produced by AI agents and decide whether each solution works
• Pinpoint where agents go wrong and turn that into feedback and evaluation signals
• Develop and refine rubrics for judging preference data
• Maintain tooling for generating, gathering and scoring data
• Report findings to researchers and engineers in concise write-ups

What you'll need

• At least five years of professional engineering work
• Strong command of Python, TypeScript/JavaScript, Go or a comparable production language
• Sound code-review judgment and clear written communication
• Familiarity with modern LLMs or AI coding tools; RLHF, rubric design or post-training background helps but is optional

Good to know

• Independent contractor engagement of roughly three months, starting immediately
• $100-$150 per hour; applicants name a specific rate
• 20-40 hours weekly, with six or more hours a day overlapping Pacific Time
• Open to candidates in North America only