OpenTrain AI 1 час назад
ML Engineer, LLM and Agentic Evaluations
- Latin America
- Удалённо
- Частичная занятость
PythonAILLMMLremoteLATAMEvaluationcontractpart-time
Описание
Responsibilities
- Design, build, and run evaluation suites for large language models across reasoning, coding, mathematics, tool use, and long-context tasks.
- Develop benchmarks and grading rubrics for multi-step planning, function calling, code execution, and browser or computer-use agents.
- Curate, clean, and annotate evaluation datasets, then write clear task specifications and grading criteria.
- Compare open-source and frontier models, analyze results, and produce structured reports with practical recommendations.
- Build programmatic and LLM-based graders, validate them against human reference judgments, and investigate recurring error patterns.
- Integrate evaluations into engineering harnesses, continuous integration pipelines, and dashboards.
- Contribute to documentation and tooling improvements.
Requirements
- Plan to work 20+ hours per week in a remote environment.
- The assignment lasts 3 months and may be adjusted based on the engagement.
- Pursuing or recently completed a Bachelor's, Master's, or PhD in computer science, machine learning, data science, mathematics, or a related quantitative field.
- Strong Python skills with familiarity with NumPy, Pandas, Jupyter, and standard machine learning workflows.
- Understanding of how modern LLMs are trained, prompted, and evaluated, plus experience using hosted or open-source LLM APIs.
- Fluent in English, with strong analytical and technical writing skills and the ability to work independently.
- Open to applicants in Argentina, Belize, Bolivia, Brazil, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Guatemala, Guyana, Haiti, Honduras, Mexico, Nicaragua, Panama, Paraguay, Peru, Puerto Rico, Suriname, Uruguay, Venezuela.
Nice to have
- Helpful background includes LLM benchmark design or use, agent frameworks, tool-using agents, LLM-as-a-judge methods, inter-annotator agreement, statistical analysis, prompt engineering, structured outputs, RLHF, RLAIF, and post-training data pipelines.
We offer
- Contract, part-time engagement.
- Remote work mode.
Hiring process
- 1 Press Apply now. You create a free OpenTrain account and add your resume.
- 2 Finish your application on the hiring company's site. The company decides who it hires, so it runs its own form.