Overview

We are seeking an experienced Databricks AI Engineer to support and enhance a productionised AI information extraction solution. The individual will be responsible for the ongoing development, optimisation, evaluation and support of the solution, working directly with client stakeholders and operating with a high degree of autonomy.

Key Responsibilities

• Deep hands-on experience with Databricks, including Lakehouse architecture, Delta Lake, Unity Catalog, Workflows and production support.
• Strong Python, SQL and Spark development skills.
• Experience building and supporting AI-driven information extraction and document processing solutions in production environments.
• Expertise in AI evaluation frameworks, including:
• Human-in-the-loop evaluation approaches
• LLM-as-a-Judge methodologies
• Gold dataset creation and benchmarking
• Accuracy, precision, recall and business outcome measurement
• Strong understanding of model selection and evaluation, including assessing multiple foundation models and designing architectures that avoid dependency on a specific LLM provider or model.
• Experience implementing robust AI workflows that can adapt to changing model capabilities and market developments.
• Strong troubleshooting, optimisation and root-cause analysis skills.

Required Skills

• Exceptional communication and stakeholder management skills.
• Highly self-directed and comfortable working independently within a client environment.
• Able to translate business requirements into technical solutions and provide clear recommendations.
• Takes ownership of delivery outcomes and proactively identifies opportunities for improvement.

Other Details

• Years of Experience: 8+ Years
• Start date: 15th Oct
• Duration: 6 Months
• Work Timings: 1:30pm to 10:30 pm
• Remote Opportunity and Contract Role
• Note: Experience building CI/CD pipelines a bonus, but don't want to make the request unrealistic.