SWE-Bench Task Auditor (Remote) at Mercor

remote
Verified Opening · Sourced from official employer portal
Share with Batch & Groups: WhatsApp Telegram
Advertisement
Mercor
Remote · Posted 11 hours ago
Apply now

Referral disclosure: the Apply button uses TechJobs360’s Mercor referral link. If you are hired through it, TechJobs360 may receive a referral bonus from Mercor. You apply on Mercor’s own site, in the same way as applying there directly.

  • Pay: $70–$90 per hour (USD)
  • Type: Contract, hourly, remote
  • Open to: India
  • Hours: About 40 hours a week
  • Field: Software Engineering
  • Posted on Mercor: 2026-08-28

About the role

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications

  • 3+ years professional software engineering
  • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
  • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
  • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)

Preferred Qualifications

  • Familiarity with SWE-Bench (Verified) or similar repository benchmarks
  • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
  • Prior code-review or task-grading experience

How applying works

You apply on Mercor’s site: create a Mercor profile, upload your CV and complete the steps the listing asks for, which usually include a short interview. Mercor decides who is selected, and pay is agreed with Mercor.

Apply on Mercor →

Browse more paid AI training roles in Worldwide Remote Jobs, or see all the platforms we list. Not the right fit? Alignerr, micro1, Ethos and Handshake AI also have paid remote AI work for professionals (referral links).

To apply for this job please visit work.mercor.com.

Get Notified for Similar Openings
Join 15,000+ tech engineers getting weekly verified off-campus & lateral hiring alerts.
Join Telegram
SWE-Bench Task Auditor (Remote) at Mercor
Mercor · Remote (India eligible)
Apply Now →