Job description for AI Quality / Evaluation Engineer at PT Surya Digital Teknologi
Surya Digital Teknologi is a digital agency based in Indonesia. We specialize in creating mobile applications, websites, software solutions, and recruitment services for businesses of all sizes. Our mission is to help organizations streamline operations through technology, enabling them to focus on growth and innovation. We pride ourselves on delivering reliable solutions and building long-term partnerships with our clients.
Currently, we are looking for an AI Quality / Evaluation Engineer who is passionate about ensuring the quality, reliability, and safety of AI-powered products. This role offers an opportunity to work on innovative AI solutions, including AI assistants, conversational AI, and agent-based systems. If you enjoy analyzing AI behavior, identifying quality gaps, and improving model performance through structured evaluation, we'd love to hear from you.
As an AI Quality / Evaluation Engineer, you will be responsible for designing and executing evaluation frameworks to assess AI system performance across accuracy, relevance, safety, instruction-following, and workflow completion. You will develop test cases, evaluation datasets, and quality benchmarks, identify model failures such as hallucinations and reasoning errors, and collaborate closely with AI Engineers, Prompt Engineers, and Product teams to continuously improve AI performance. You will also contribute to prompt optimization, regression testing, and the establishment of quality standards prior to production releases.
Requirements
- Bachelor's degree in Computer Science, Data Science, Artificial Intelligence, Linguistics, Mathematics, or a related field.
- 4+ years of experience in Software Quality Assurance (QA).
- 2+ years of experience testing or evaluating AI/LLM-based applications.
- Strong understanding of Large Language Models (LLMs) such as GPT, Claude, Gemini, or Llama.
- Experience creating test cases and evaluation frameworks for AI output quality, including accuracy, hallucination detection, safety, and workflow completion.
- Strong analytical and problem-solving skills, with good written and verbal English communication.
We are looking for someone with strong attention to detail, a data-driven mindset, and a passion for AI quality and innovation. If you are excited about shaping the next generation of AI-powered products and helping deliver reliable AI experiences, we look forward to receiving your application.
