2023 to 2024 In-house LLM translation pipeline at Beekeeper
Replaced a vendor translation contract with a GitHub Actions and OpenAI pipeline in 6 weeks.
Python OpenAI GitHub Actions Evals
Problem
Translation was a recurring vendor line item with multi-week turnaround that slowed every release. Cost scaled with the number of locales and strings. Product waited on an external gatekeeper for every new string.
What I built
I designed and shipped a pipeline on GitHub Actions and the OpenAI API: locale-specific prompts, glossary enforcement, and automated evaluation against the old vendor output. It hooks into the PR flow, so new strings translate on merge. Shipped in 6 weeks. Over 12,000 key-value pairs translated since January 2024 with negligible regression.
What I learned
The eval harness sold the project, not the model. I compared LLM output with vendor output on real strings before I asked anyone to switch. Quality held, turnaround dropped from weeks to a PR, and the pipeline paid for itself many times over.