Agentic Coding Evaluation Specialist (Contract)

As an Agentic Coding Evaluation Specialist, you will work with real-world codebases to design challenging coding prompts, execute them across AI models, and evaluate outputs through structured, side-by-side comparisons. Your input will directly contribute to improving AI systems for developer workflows.

• Compensation: $75/hour
• Duration: ~2-4 weeks (with potential extension)
• Commitment: Minimum 20 hours/week (weekdays only)
• Project start date: May 4, 2026 (Tentative)
• Applications will be accepted/reviewed on a rolling basis until the batch is full

Key Responsibilities

• Analyze and navigate large open-source codebases (e.g., React, Pydantic, Pandas)
• Create complex, single-turn coding prompts grounded in real repositories
• Execute prompts across multiple AI model instances
• Perform side-by-side (SxS) evaluations of model outputs
• Evalure model performance across dimensions such as Instruction following, Code quality, Tool usage, Testing & validation and Communication clarity
• Provide clear, structured justifications for evaluation decisions

Required Qualifications

• 2+ years of professional experience in software development
• Bachelor’s degree in Computer Science or a related field
• Strong proficiency in at least one of the following: Python / JavaScript / TypeScript / Java / Go / C++
• Solid understanding of Data structures & algorithms, Debugging and testing practices and Software design and code quality principles