LLM Evaluation Specialist

Job title:

LLM Evaluation Specialist

Company

Oura

Job description

Job Description:Our mission at Oura is to empower every person to own their inner potential. Our award-winning products help our global community gain a deeper knowledge of their readiness, activity, and sleep quality by using their Oura Ring and its connected app. By providing daily insights and practical steps to inspire healthy lifestyles, we’ve helped over one million people understand and improve their health.Join Oura in Shaping the Future of AI in Health Tech!At Oura, we’re building the future of personal health, powered by cutting-edge technology and backed by science. We’re expanding our efforts to bring more thoughtful, human-centered AI features into our mobile app, and we’re looking for a practitioner with a deep understanding of AI systems, decision logic, and user impact to help us ensure these features are accurate, trustworthy, and aligned with Oura’s unique tone of voice.This is a unique opportunity to directly influence how Oura supports members on their health journey – one conversation and one insight at a time.About the RoleAs an LLM Evaluation Specialist, you’ll play a central role in defining and maintaining the quality standards for our AI-generated content and insights. Traditional testing doesn’t fully apply to language models, so instead, we use purpose-built evaluations to determine what great AI responses look like across accuracy, tone, and user experience.You’ll be responsible for writing and organizing these evals, collaborating with cross-functional teams to apply them, and helping shape internal tooling that makes evaluation scalable and efficient. Your work will directly contribute to the trustworthiness and helpfulness of our AI features.What You’ll Do

• Define and write LLM evaluations: shape what high-quality outputs look like in terms of content, tone, and structure.
• Maintain and own evaluation datasets: manage and evolve the core set of evaluations that support LLM-powered features.
• Run evaluations: test outputs, flag issues, and ensure quality standards are met.
• Train others: teach best practices and offer consultation to other teams.
• Collaborate on tooling: partner with product and engineering to improve the evaluation infrastructure and define future tool features.

What We’re Looking For

• A strong interest in large language models and their applications.
• Exceptional English writing and editing skills, with a sharp eye for clarity and nuance.
• Curiosity about prompt design and how to influence AI behavior.
• Strong communication and collaboration skills across time zones and disciplines.
• Ability to teach and empower others in emerging spaces.
• Comfort working in ambiguous and evolving problem spaces.

Bonus Points For

• Familiarity with Oura’s product or a general interest in health, wellbeing, and human physiology.
• Experience working independently and driving projects forward.
• An understanding of product development or software workflows.
• Experience working with LLMs

What we offer

• Competitive salary
• Lunch benefit
• Wellness benefit
• Flexible working hours
• An Oura ring of your own
• Personal learning & development program
• Wellness Time Off

Oura is proud to be an equal opportunity workplace. We celebrate diversity and are committed to creating an inclusive environment for all employees. Individuals seeking employment at Oura are considered without regards to age, ancestry, color, gender (including pregnancy, childbirth, or related medical conditions), gender identity or expression, genetic information, marital status, medical condition, mental or physical disability, national origin, protected family care or medical leave status, race, religion (including beliefs and practices or the absence thereof), sexual orientation, military or veteran status, or any other characteristic protected by federal, state, or local laws. We will not tolerate discrimination or harassment based on any of these characteristics.

Expected salary

Location

Oulu, Pohjois-Pohjanmaa – Pohjanmaa

Job date

Wed, 18 Jun 2025 03:35:54 GMT

To help us track our recruitment effort, please indicate in your email/cover letter where (jobsineu.net) you saw this job posting.