colours
radaremprego

AI Benchmark & Datasets Engineer / Researcher

Hoursnot stated in the ad
not stated in the adwork mode · contract · salary
posted 2026-11-01 ·

Where

54 countries incl. Portugal
More jobs: Systems analysts, remote (828)

Sign in to see the travel time

In short AI

AI Benchmark & Datasets Engineer / Researcher role at Pathway, an AI company headquartered in Palo Alto, California. Remote work, full-time permanent contract, with the option to work from offices in Palo Alto, Paris or Wroclaw.

Description

a long ad (5 007 characters): the AI summarised it; the full text is in the next tab.

Role
Design and run rigorous benchmarks and define dataset standards for evaluating Pathway's models. Identify, prioritise and curate public and client-driven benchmarks; assess their data quality and methodology; run benchmarks with baseline models; hand off benchmark-ready packages to the R&D team; maintain shared documentation and organise results and leaderboards; contribute to demos and public proof points based on benchmark results.
Requirements
  • Experience with ML/LLM evaluation, data science or technical product roles, ideally around benchmarks or experimentation
  • Ability to read papers, leaderboards and GitHub repos and turn them into clear, repeatable benchmark specs
  • Ability to talk with engineers and customers, translating technical detail into business value
  • Care for high-quality data, reproducible experiments and clear documentation
  • Fluent English
  • One of the following: first-author or significant-contribution paper at NeurIPS, ICLR or ICML; significant contribution to a newsworthy LLM training effort; at least 6 months at a leading ML research centre; ICPC World Finalist or IOI, IMO or IPhO medallist in high school
  • Cover letter of 2 to 3 lines
Preferred
  • published or open-source work on LLM evaluation, benchmarking or data quality; experience designing custom benchmarks or evaluation protocols
  • published or open-source work on LLM evaluation, benchmarking or data quality; experience designing custom benchmarks or evaluation protocols
Conditions
Contract: Full-time, permanent
Salary: Based on profile and location (no specific amount stated)
Hours: not stated
Location: Remote; possibility to work on site in Palo Alto (CA), Paris or Wroclaw. Candidates in the EU, UK, US and Canada considered
Benefits
none stated

About Pathway

Pathway builds the first post-transformer frontier model that solves AI's fundamental memory problem. While transformers wake up in the same state every time—like Groundhog Day—our architecture enables true continuous learning, infinite context reasoning, and real-time adaptation. We're not optimizing yesterday's technology; we're building what comes after transformers.

Our breakthrough architecture outperforms Transformer and provides the enterprise with full visibility into how the model works. Combining the foundational model with the fastest data processing engine on the market, Pathway enables enterprises to move beyond incremental optimization and toward truly contextualized, experience-driven intelligence. We are trusted by organizations such as NATO, La Poste, and Formula 1 racing teams.

Pathway is led by co-founder & CEO Zuzanna Stamirowska, a complexity scientist who created a team consisting of AI pioneers, including CTO Jan Chorowski who was the first person to apply Attention to speech and worked with Nobel laureate Geoff Hinton at Google Brain, as well as CSO Adrian Kosowski, a leading computer scientist and quantum physicist who obtained his PhD at the age of 20.

The company is backed by leading investors and advisors, including Lukasz Kaiser, co-author of the Transformer (“the T” in ChatGPT) and a key researcher behind OpenAI’s reasoning models. Pathway is headquartered in Palo Alto, California.


The Opportunity

You will design and execute rigorous benchmarks and define dataset standards. Collaborating closely with our R&D team, you will build the evaluation infrastructure that guides the evolution of Pathway’s post‑transformer models.

You Will

  • Proactively identify, prioritize, and curate relevant public and client-driven benchmarks across our target use cases and markets.
  • Evaluate candidate benchmarks for clarity, data quality, evaluation methodology, and fit with our model roadmap.
  • Run benchmarks with baseline models to validate setup, uncover edge cases, and de‑risk R&D runs.
  • Hand off “benchmark-ready” packages to R&D (specs, data, evaluation scripts, expected metrics, constraints)
  • Maintain a shared vocabulary and documentation around benchmarks, datasets, and evaluation formats that GTM and R&D can both use.
  • Track and organize benchmark results, model leaderboards, and “what good looks like” for different customers and scenarios.
  • Contribute to demos and public‑facing proof points based on benchmark outcomes.

You will play a key role in defining and driving the benchmarking process for AI model evaluation. Your work will directly influence what we build, how we talk about it, and how customers and the market experience BDH.

Requirements

Cover letter

It's always a pleasure to say hi! If you could leave us 2-3 lines, we'd really appreciate that.

You are expected to meet at least one of the following criteria:

  1. You have published at least one paper at NeurIPS, ICLR, or ICML - where you were the lead author or made significant conceptual & code contributions.
  2. You have significantly contributed to an LLM training effort which became newsworthy (topped a Hugging Face benchmark, best in class model, etc.), preferably using multiple GPU's.
  3. You have spent at least 6 months working in a leading Machine Learning research center (e.g. at: Google Brain / Deepmind, Apple, Meta, Anthropic, Nvidia, MILA).
  4. You were an ICPC World Finalist, or an IOI, IMO, or IPhO medalist in High School.

You

  • Have experience with ML/LLM evaluation, data science, or technical product roles, ideally around benchmarks or experimentation.
  • Are comfortable reading papers, leaderboards, and Github repos, and turning them into clear, repeatable benchmark specs.
  • Can talk comfortably with both engineers and customers, and translate between technical detail and business value.
  • Care about high‑quality data, reproducible experiments, and crisp documentation
  • Are respectful of others
  • Are fluent in English

Bonus Points

  • Published or open‑sourced work on LLM evaluation, benchmarking or data quality.
  • Experience designing custom benchmarks or evaluation protocols for novel model capabilities.

Why You Should Apply

  • Join an intellectually stimulating work environment.
  • Be a pioneer: you get to work with a new type of "Live AI" challenges around long sequences and changing data.
  • Be part of one of an early-stage AI startup that believes in impactful research and foundational changes.

Benefits

  • Type of contract: Full-time, permanent
  • Preferable joining date: Immediate. The positions are open until filled – please apply immediately.
  • Compensation: based on profile and location.
  • Location: Remote work. Possibility to work or meet with other team members in one of our offices: Palo Alto, CA; Paris, France or Wroclaw, Poland. Candidates based anywhere in the EU, UK, United States, and Canada will be considered.

If you meet our broad requirements but are missing some experience, don’t hesitate to reach out to us.

Originally posted on Himalayas

How to apply

Preferred start date: immediate. Positions open until filled; apply immediately. Cover letter of 2 to 3 lines. AI

Apply on the original ad ↗

Prepare your application

A cover letter and a short pitch for this job, written from your profile, and what to learn next.

Sign in and create your profile from your CV. It is private: we share it with no one.

Something wrong or unlawful in this ad? Report it

It stays visible while we review it. We decide within 7 days (anything clearly unlawful, such as a scam, sooner) and tell you the outcome and why.

Similar jobs

Salary compared ⓘ

This jobMedian?4 107 €
2 788 €6 138 €

86 jobs · Systems analysts · remote

see the best paid →

Training for this job

Published on (1)

First seen on 16 Nov, 22:54