Mercor Is Hiring A Bilingual Italian Generalist Evaluator Expert

🚀 Mercor is looking for talented native Italian speakers with exceptional writing skills to collaborate on a high-impact AI research project alongside a leading laboratory. This short-term, flexible opportunity is perfect for professionals who excel in language mastery, critical thinking, and clear instructional communication, especially those capable of translating complex ideas into culturally nuanced Italian while maintaining technical precision in English.

About the Role

Multilingual Prompt Design & Optimization

• Create detailed prompts in Italian and/or English with multiple constraints and instructions.
• Ensure prompts are naturally phrased and relevant for Italian-speaking users in real-world contexts.

Define and Document Evaluation Standards

• Establish high-level standards for correct responses within Italian consumer scenarios.
• Develop comprehensive rubrics that consider linguistic nuances, tone, and cultural conventions.

Model Testing and Grading (Bilingual)

• Run prompts through language models and assess initial outputs for accuracy, fluency, and cultural appropriateness in Italian.
• Compare results against English benchmarks when applicable.

Benchmarking & Quality Assurance

• Participate in QA review processes to ensure prompt tasks and rubrics meet high standards.
• Maintain consistency and reliability across Italian-language benchmarks prior to integration into official evaluations.

Minimum Qualifications

• Native-level fluency in Italian (written) with strong reading and writing skills in English.
• BS or BA from a reputable institution (completed or in progress).
• Excellent writing and critical thinking abilities.
• Ability to work independently and adhere to deadlines.
• Familiarity with ChatGPT or similar AI tools for personal decision-making, hobbies, or interests.
• Based in Italy or able to reliably produce culturally accurate Italian content specific to Italy.

Preferred Qualifications

• Experience in teaching, research, editing, or academic writing.
• Background in creating evaluation criteria, rubrics, or grading guidelines.
• Familiarity with Large Language Models (LLMs), prompting, or model evaluation (helpful but not mandatory).

Application & Onboarding Process

1. Complete a brief AI-led interview (~15 minutes).
2. Undertake a 45-minute written assessment focusing on writing skills and rubric creation.
3. If selected, receive an invitation to join the project team.

Additional Role Details

• Expect to contribute a minimum of 20 hours per week.
• Project duration of approximately 2+ months.
• Work within a structured environment with clear goals and supporting tools.
• Mercor values diversity and is committed to providing reasonable accommodations upon request, welcoming all qualified applicants regardless of legally protected characteristics.