QA Engineer
We're looking for a rigorous, technical RL Data QA Engineer to review the reinforcement-learning tasks built on our platform before they reach frontier AI labs.
Every task on our platform is a small software environment: instructions, a containerized environment, a reference solution, and tests that turn an AI agent's work into a reward. Before a task reaches you, it has been run many times against frontier coding agents and audited automatically. You are the expert human judgment that decides whether it is good enough to train and evaluate the next generation of AI. Is it correct, fair, challenging and impossible to game?
What You'll Do
- Review RL tasks end to end: Examine instructions, environments, reference solutions and tests, and decide whether each task meets our quality bar.
- Break the verifier: Look for the ways an agent could pass without really solving the task, such as weak tests, leaked solutions or loopholes in the environment.
- Read agent trajectories: Analyze how frontier coding agents attempted each task, and tell a genuine solve from a lucky or illegitimate one.
- Judge difficulty with data: Interpret pass rates across models and trials, and decide when a result is solid and when it needs more evidence.
- Separate task faults from platform faults: Make sure contributors are judged on their work, never on infrastructure issues.
- Write feedback that gets tasks fixed: Give contributors precise, actionable comments, so a task improves in one iteration instead of three.
- Raise the bar over time: Turn recurring defects into rubric rules, automated checks and better task templates, working with our engineers and researchers.
What We're Looking For
- Strong engineering skills: Comfortable with Python, Bash, Linux and Docker. You can build, run, break and fix a task environment yourself.
- An adversarial mindset: You enjoy finding the hole: exploits, test gaps, edge cases, reward hacks.
- Clear, precise writing: Your feedback is specific, actionable and respectful.
- Statistical common sense: You understand pass rates, small samples and variance well enough to make sound calls.
- Ownership and autonomy: You take responsibility for quality, including the issues nobody else noticed, and you work without constant supervision.
- Ambition and drive: A highly motivated, resourceful and results-oriented person who wants to grow quickly and make a tangible impact on frontier AI.
We value demonstrated achievements, intellectual curiosity and the ability to execute over years of experience or impressive job titles.
Nice to Have
- Experience with RL environments, LLM evaluations, benchmarks or reward modeling.
- Deep technical knowledge in at least one domain: For example QA, software engineering, networking, security, data or ML engineering.
- Experience authoring tasks or benchmarks for AI agents.
- Fluency with AI coding agents (Claude Code, Codex, Cursor or similar) as review tools.
- Experience writing review guidelines or calibrating a team of reviewers.
Why Join ARIMLABS?
- Shape frontier AI directly: The tasks you approve train and evaluate the next generation of AI coding agents.
- Exceptional growth opportunities: Join a fast-scaling AI research company at an early stage, with a path to leading the QA function as it grows.
- High autonomy: Freedom to make expert calls and propose better ways of working, without unnecessary bureaucracy.
- Work with exceptional people: Collaborate with highly skilled researchers, engineers and domain experts working on challenging problems at the frontier of artificial intelligence.
- Exposure to frontier AI: See every day how the most capable AI agents succeed, fail and cheat.
- Entrepreneurial environment: Work in a company that values initiative, speed, ownership and ambitious thinking.
- Career flexibility: Grow toward QA leadership, research, or task design as the organization expands.
Compensation & Working Conditions
- Engagement: B2B contract.
- Time off: 24 days of paid time off annually.