Machine Learning Engineer – Evals

Salary: $220K – $300K

On-site work policy: 5 days in-office in New York, NY

Full-time position

Location: New York, NY

Equity: Competitive equity

Visa sponsorship details: Not open to any visas (e.g. US citizen, Green Card holders)

Hiring Count: Looking to hire 1 candidate for this role

About this role

You’ll own evaluation of Honcho end to end, the instrument that tells a research-driven team whether their identity representations are actually getting better. Plastic Labs builds the memory and identity layer for the agentic world, and the product works. It’s a complex multi-agent harness with real black-box behavior, so the job is defining what “better” means for representations that change over time, without ground truth, and building the machinery that measures it.

If you love constructing measurements from nothing, reading traces to find what’s actually broken, and shipping the fix yourself, you’ll feel at home here.

What You’ll Do

Design the evals. Define the scores and decide what “better” means for an entity representation that changes over time, and keep that definition current as the product and methods move.

Build the pipelines and harnesses. Data in, labels, versions, reruns, judges: the machinery that lets the team ask a new question this week and get an answer this week.

Run them and harvest insights. Read the traces and results, find what’s actually broken, not what’s easy to measure, and propose fixes that produce higher-fidelity representations.

Build simulation agents at scale. The ceiling on iteration speed is how many entities can be modeled and measured at once. You raise it.

Own the loop end to end. The question, the pipeline, the rerun, the writeup. Nothing gets scoped and handed off, and nothing waits on someone else’s sprint.

Requirements

Dealbreaker

Work experience

• Must have: Designed,​ built,​ and ran evals end to end (LLMS)

Hard skills

• Must have: Production-​quality Python (tests,​ maintenance,​ release cadence)

Baseline

Seniority

• Required: 3+ years of experience 3+ years designing and owning LLM evaluation

Hard skills

• Required: PyTorch,​ Hugging Face,​ Weights &​ Biases.​ Data pipeline tooling (Hydra,​ Kafka,​ SQL) a plus

Nice-to-have

Work experience

• Nice to have: Work on memory,​ identity,​ personalization,​ agent evaluation,​ or agentic optimization.​
• Nice to have: Shipped research:​ first or co-​author at NeurIPS,​ ICML,​ ICLR,​ or equivalent,​ OR self-​published open-​source work of comparable quality

Education

• Nice to have: Degree in computer science,​ mathematics,​ statistics,​ or a related quantitative field

Traits to avoid

Agent evals

• Only runs existing benchmarks — never designed a novel evaluation from scratch.​
• Pure engineers with no research background — must have published or produced equivalent open-​source research work.​

Application

Send us your updated CV/Resume to contact@remotetalents.website with the subject: “Machine Learning Engineer – Evals”

Follow us on X (ex Twitter) ; LinkedIn ; Instagram.