Intelligence Analyst
In short AI
Remote contractor role as an intelligence analyst to evaluate and improve AI systems: building analytic cases, writing test cases and grading AI-generated responses. No prior AI experience is required; all-source intelligence analysis experience is valued.
Description
- Role
- Develop and map analytic cases to defined taxonomies, write complex test cases on source handling, analytic rigor and expression of uncertainty, assess AI-generated responses, flag over-refusals by the model, document edge cases and work with project leads to calibrate analytic standards.
- Requirements
- All-source analytic experience or deep expertise in a specific analytic discipline
- Strong proficiency in source evaluation, confidence-level assessment and established analytic standards
- Experience teaching, mentoring or leading analytic reviews
- Exceptional written and verbal communication, with clarity and precision
- Ability to identify and articulate model limitations and over-refusal scenarios
- Preferred
- structured analytic techniques, intelligence tradecraft standards, prior participation in AI evaluation initiatives
- Conditions
- Contract: independent contractor
- Salary: not stated
- Hours: not stated
- Location: remote
- Benefits
- none stated
Role Title: Intelligence Analyst
Role Type: Contractor
Location: Remote
micro1 is engaging Intelligence Analysts to contribute to a customer-facing project focused on evaluating and advancing artificial intelligence capabilities. In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.
Scope of Work
- Develop and map analytic cases to defined taxonomies across multiple evaluation areas, ensuring comprehensive coverage of relevant intelligence scenarios.
- Author complex test cases designed to evaluate model performance on source handling, analytic rigor, and nuanced uncertainty expression.
- Assess and grade AI-generated responses based on criteria such as reasoning, source use, recognition of limitations, and practical utility.
- Identify and flag instances where legitimate analytic exploration is over-refused by the model, providing clear documentation and feedback.
- Document edge cases and analytic disagreements, supporting continuous improvement of evaluation methodologies.
- Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment.
- Contribute insights to help calibrate analytic standards and enhance the overall quality of AI system benchmarks.
Preferred Qualifications
- Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline, ideally with a civilian-IC background.
- Strong proficiency in source evaluation, confidence-level assessment, and established analytic standards.
- Experience in teaching, mentoring, or leading analytic reviews, with a record of fostering analytic rigor.
- Exceptional written and verbal communication skills, with a focus on clarity and precision.
- Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
- Familiarity with structured analytic techniques and intelligence tradecraft standards.
- Previous participation in AI evaluation or model assessment initiatives is a plus.
Originally posted on Himalayas
How to apply
Prepare your application
A cover letter and a short pitch for this job, written from your profile, and what to learn next.
Sign in and create your profile from your CV. It is private: we share it with no one.
Something wrong or unlawful in this ad? Report it
Similar jobs
Salary compared ⓘ
83 jobs · Systems analysts · remote
see the best paid →Training for this job
- a Inteligência Artificial no QuotidianoServiço de Formação Profissional de Viana do Casteloonline · evenings · 8 h · free · starts 03/12/2026Asked for in this ad: AI / machine learning
- Utiliz. Ferramentas de Inteligência Artificial - Pós-laboralServiço de Emprego de Portimãoonline · evenings · 25 h · free · starts 23/11/2026Asked for in this ad: AI / machine learning
- Utilizar ferramentas da inteligência artificial na organizaçãoAcademia CITEVEblended · 25 h · free · starts 09/12/2026 · Vila Nova de FamalicãoAsked for in this ad: AI / machine learning
Languages AI
Published on (1)
- Himalayas seen 16 Nov
First seen on 16 Nov, 23:19