Two Projects, One Platform
How we fine-tuned an embedding model to deflect support tickets and a small LLM to power conversational search for a large online travel booking platform, and what each phase taught us.
Two problems landed in the same quarter
The client is a large online travel booking platform (name withheld) offering hotels, vacation rentals and activities. It lists hundreds of thousands of properties and serves millions of travelers a month. It runs entirely on AWS and has a small in-house ML team.
Support costs rose every holiday season
About 40,000 tickets a month, with sharp peaks before holidays. Roughly a third asked questions the help center already answered: changing dates, cancellation policies, refund status, a missing confirmation email. Travelers never found those answers because help search was keyword-based.
Goal: show the 3 most useful articles or past resolutions before a ticket is opened. Deflect 20% of tickets.
Search failed on how travelers actually ask
Queries like "quiet beach hotel in Crete for a family of four in late June, under €200 a night" or "romantic weekend in Lisbon with a view" got poor results. Nearly one in five free-text searches ended without a click, mostly on mobile.
Goal: a trip-planning assistant inside search that turns conversation into a structured search. +5% relative booking conversion on assistant sessions.
We deliberately started with the lower-risk embedding project, so it would build the shared platform (data pipelines, experiment tracking, model registry, evaluation practice) that the LLM project would later reuse.
Support deflection
Fine-tuned embedding model (~570M parameters), plus a distilled reranker, served from an OpenSearch vector index.
We pulled two years of resolved tickets and linked each one to the article or macro that closed it. We sampled 500 pairs and read them by hand.
Agents closed tickets with generic macros ("Thanks, closing this") or workflow notes that solved nothing. Training on all of them would have taught the model to retrieve generic closing messages. We filtered and kept about 70% of the pairs.
Travel tickets are full of personal data: names, passport numbers, phone numbers, booking references, card details. Our first masking dictionary was far too aggressive and blanked out thousands of passages, including hotel and city names that looked like surnames. We added a frequency cap: a "name" that shows up in more than a few percent of tickets is treated as ordinary text.
Before training anything, we built a fixed evaluation set: 1,000 held-out real tickets with known correct answers. We tested four off-the-shelf embedding models, BM25, a hybrid of both, and an off-the-shelf reranker.
A mid-sized multilingual model beat newer, higher-ranked models on our data, which mattered because travelers write in many languages. BM25 alone beat pure embeddings on short tickets and was essential for booking references and payment error codes. The hybrid helped. The off-the-shelf reranker made results slightly worse. Since then we choose models on our own eval set and use public benchmark rankings only to build the shortlist.
We fine-tuned the embedding model on about 6,000 cleaned ticket→resolution pairs with mined hard negatives. One epoch on a single L40S GPU took about three hours.
Ticket recall@10 rose by about 18 points, but help-article retrieval fell below the base model. The model had learned the language of tickets and forgotten the language of policy documentation. Generating synthetic questions for every article and mixing them in fixed both.
A second epoch and larger batches gave no gain or a small drop. We kept runs short and watched validation loss.
We distilled a small reranker from an LLM's relevance scores using only about 500 query groups. Top-1 accuracy rose sharply, with no reindexing. It was especially good at telling apart look-alike policies, such as a free cancellation versus a non-refundable rate. If we did it again, we would start here.
We ran an A/B test with suggestions shown on 20% of ticket forms.
Early results looked great until we checked how often travelers came back within 48 hours with the same problem. Some apparent deflections were travelers giving up. We made a reopen rate below a set threshold a release condition. Net deflection settled around 17%: below the 20% target, but well above break-even. The suggestions also cut handling time on tickets that did go through, which we had not planned for.
A traveler standing at a hotel desk with no reservation doesn't need an article. We added an urgency check (check-in today, "at the hotel now", overbooking) that skips suggestions and routes straight to an agent.
We set up monthly retraining from "this solved it" clicks, with an extra retrain before each peak season when policies change.
OpenSearch can't change a vector field in place, so each version gets its own index, filled in the background, checked against the evaluation set and switched in through an alias. We keep the old index for a week as a rollback. We should have designed this in Phase 0. We built it in a rush before the first retrain.
Conversational search
Fine-tuned 4B LLM (LoRA) that turns a trip-planning conversation into a structured search call, served with vLLM on a SageMaker endpoint.
The first request was "a travel agent chatbot." We split it into three parts:
- Conversation → structured search call (JSON): destination, dates, guests, budget, amenities, location. This is the core, and where we would fine-tune.
- Running the search against live availability, which the existing engine already did well.
- Writing a short reply, done with templates or a frontier model.
In travel, prices and availability change by the minute. The fine-tuned model never states a price or says a room is available. It only produces search calls, and the live engine supplies every fact, so it cannot invent rooms or rates.
We prompted a frontier model on Bedrock with the full search schema (destinations, amenity codes, property types) in the prompt and measured it on 1,500 labeled conversations.
A 2,000+ token schema in every request cost seconds of latency. Quality was strong on popular destinations and weak on our own amenity codes, property types and regional place names.
Two frontier models rated the same conversations very differently. We had experts label a sample, measured agreement, and rewrote the rubric until the judge matched them. Before calibration, the judge scores correlated poorly with expert ratings.
Given "somewhere warm in March," it picked a destination instead of asking about budget or flight time. Prompting couldn't fix this.
We built about 12,000 training examples: roughly half from real search logs (query → filters the traveler ended up applying → property they viewed or booked), half synthetic from the search schema. We first tried an 8B model, then a 4B model, using LoRA at rank 32.
When one model wrote both sides, the "travelers" wrote long, polite paragraphs. Real users type six words and change their dates halfway through. We switched to a two-agent simulation with personas ("impatient", "budget-focused", "undecided about dates") and checked it against real logs.
Our first version asked a clarifying question on almost every turn, because the training data had many clarification examples and few cases where it should just search. Balancing the two labels, and adding "don't apply a filter" examples, fixed it.
Small differences in tool names, field order, date formats and prompt wording between training data and the production prompt cost us several points. We made a single source of truth for the prompt and schema.
After fine-tuning, the 4B model matched the 8B model on structured-call accuracy, with about 99% valid JSON, and beat the frontier baseline on our own amenity codes and place names at about half the 8B serving cost. The frontier model stayed better on multi-stop trips ("3 nights in Rome, then 2 in Florence, with a car"), so we route those to it.
One run looked excellent, and two reruns with different random seeds came out noticeably worse. We now report the average and the spread.
We ran the model in shadow mode for two weeks, then served it to 1% of mobile search, then 5%.
At 1% traffic, the fine-tuned model converted worse than the prompted baseline even though offline scores were equal. Tagging every failure showed two slices causing most of the gap: flexible dates ("a long weekend in May") and landmark or neighborhood queries ("walking distance to the old town"). Targeted training data closed it within two retraining cycles.
Slow replies reduced conversion regardless of answer quality. We cached results for frequent queries so only the long tail reaches the model, added a 1-second timeout that falls back to plain keyword search, and kept at least one warm replica at all times. A cold start took several minutes.
No public data existed for our model on our instance type under real concurrency. A one-day load test, run at peak-season traffic levels, showed how many concurrent sessions each instance could handle before latency degraded.
The model is served with vLLM on a SageMaker endpoint and retrained every two weeks with new failure slices and seasonal queries.
A routine vLLM upgrade changed throughput noticeably, and a SageMaker SDK update broke a pipeline step. Both versions are now pinned and upgraded only after a load test.
Hard technical lessons
The phase stories above cover what we learned about the work itself. This section covers the platform: AWS capacity, SageMaker, Bedrock, model serving, the interfaces between components, and the models themselves. Each item is what broke and what we did about it.
| Layer | What we ended up with | What we tried and dropped |
|---|---|---|
| Embedding training | SageMaker Training Jobs on 1× g6e.xlarge (L40S 48GB), sentence-transformers | Larger multi-GPU instances: no quality gain |
| LLM training | SageMaker Training Jobs on g6e.12xlarge (4× L40S), LoRA r32, Spot with S3 checkpoints | p5 Capacity Blocks: not needed at 4–8B |
| Experiment tracking | SageMaker managed MLflow plus Model Registry | Notebook-only tracking in the first weeks |
| Vector search | OpenSearch, one index per model version, alias swap, BM25 in the same index | Separate keyword and vector clusters |
| LLM serving | vLLM on a SageMaker real-time endpoint, g6e, adapter merged, minimum 1 instance | Bedrock Custom Model Import for the live path; an EKS prototype |
| Frontier model | Bedrock Converse API: baseline, synthetic data, judge, fallback for multi-stop trips | Using the same model family as both teacher and judge |
AWS capacity and quotas
With approved quota, g6e launches still failed with InsufficientInstanceCapacity in our main region during a busy week.
FixLaunch logic that tries every availability zone, then a fallback instance type (g6e, then g6/g5 for embedding jobs), with a retry window of about 30 minutes.
EC2 on-demand, Spot, SageMaker training, SageMaker endpoints and HyperPod each have their own quota per instance type. New accounts start at zero for GPU families.
FixWe filed every quota request in Phase 0. P-family requests took the longest.
We had raised the training quota for an instance type, but the endpoint quota for the same ml.* type is a different line item.
FixCheck the specific endpoint usage quota before the first deployment, not on launch day.
Blocks are an EC2 feature; SageMaker has its own training plans. After the 2026 price increases, a p5 block was only about 13% cheaper than on-demand.
FixWe skipped them. Our 4–8B LoRA runs fit on g6e, which we could get on demand or as Spot.
SageMaker training and tooling
Each training job pulls a multi-GB container. For a 40-minute embedding run, provisioning and image pull were a large share of the wall clock.
FixWarm pools during hyperparameter sweeps (keep-alive is capped at 1 hour), and fewer, longer sweep jobs.
Training jobs default to a 1-day maximum runtime, and Spot interruptions restart from scratch unless you checkpoint.
FixCheckpoint to S3 every N steps and resume automatically. This made Spot safe for LoRA runs.
Moving to SageMaker Python SDK v3 produced an empty input config that the pipeline API rejected, and step caching stopped hitting when we used source_dir.
FixPin the SDK version in the lockfile and upgrade only in a dedicated change with a full pipeline run.
Our security team's SCPs used conditions, and the SDK's client-side role validation rejected roles that actually worked.
FixWorked through it with the platform team and documented the known false positive. Budget a day for IAM in any locked-down account.
For per-job training that premium was worth it: no idle machines and no cluster to run. For always-on research boxes it wasn't.
FixTraining on SageMaker jobs; dev boxes on EC2 with auto-stop after 1 hour idle.
Bedrock
We used Bedrock for the frontier model throughout and evaluated Custom Model Import (CMI) for serving our fine-tuned 4B model.
After an idle period, CMI evicts the model and calls return ModelNotReadyException. Restore time ranged from about a minute to tens of minutes, with no way to guarantee it.
FixKeep-warm calls every few minutes plus a long retry budget work for batch jobs. For the live search path we moved to a SageMaker endpoint.
CMI bills per active model copy in 5-minute windows. Keeping one copy warm around the clock priced out at roughly 3x a g6e.xlarge.
FixCMI only makes sense for traffic with real idle periods, such as nightly enrichment jobs.
We found no way to import a LoRA adapter on its own. Every retrain meant merging the adapter into the base weights and re-importing the full model.
FixMerge as a pipeline step, version the merged weights in S3, and account for 5–20 minutes of import time per release.
We had to render the chat template ourselves and call InvokeModel with a raw prompt. That created a second place where the prompt format could drift from training.
FixOne shared prompt-building library used by training data generation, evaluation and every serving path.
The same Converse calls served four jobs: baseline, data generation, judging and fallback. A silent model version change would have shifted all four.
FixPin exact model IDs in config, use temperature 0 for judging, and use a judge from a different model family than the teacher that generated the data.
Serving the LLM
Our first 8B endpoint took about 9 minutes from scale-out to first request. SageMaker container caching roughly halved it. On the EKS prototype, model streaming plus a persistent compile cache took it from about 5 minutes to under 3.
FixNever scale to zero. Keep a floor of one warm instance and scale out early on queue depth.
On a 24 GB GPU, latency was fine up to about 16–32 concurrent sessions and then collapsed, with timeouts not far beyond. Two 8B candidates behaved very differently under the same load.
FixMoved to L40S 48 GB and set the autoscaling target below the cliff measured in our own load test. Include load behavior in model selection.
Serving adapters on a shared base added latency, and vLLM reserves GPU memory for every --max-loras slot whether or not it's used.
FixProduction serves one merged model. Multi-LoRA is only for side-by-side experiments, with one standard rank for every adapter.
Quantizing to FP8 cut latency but reduced accuracy on rare amenity codes and small regional place names.
FixShipped BF16. Re-test quantization on per-slice metrics for every new model, not just the overall score.
The default 64 MiB /dev/shm crashed vLLM. The liveness probe killed pods while weights were still loading. The load balancer's 60-second idle timeout cut off streamed responses. CPU-based autoscaling meant nothing for GPU load.
Fix8 GiB memory-backed /dev/shm, a startup probe, a 300-second idle timeout, and autoscaling on vllm:num_requests_waiting.
A minor version upgrade changed throughput noticeably. LoRA-related crash bugs appeared in several 2026 releases of both vLLM and SGLang.
FixPin the engine version in the container image. Upgrade only after a load test and the full eval set.
Interfaces and formats
Training used the tokenizer's chat template; an early serving path built the prompt by hand. Quality dropped with no error anywhere.
FixAlways use the tokenizer's template, through the shared prompt library, and run a canary request with a known answer at deploy time.
Constrained decoding with a flat JSON schema added little latency. An early, deeply nested schema with many optional fields was slow and error-prone.
FixFlat schema, enums for amenity and property-type codes, ISO dates, and validation on every response with fallback to keyword search.
vLLM and SageMaker endpoints speak an OpenAI-compatible API; Bedrock Converse uses a different request and response shape.
FixA thin adapter layer so the app can switch between fine-tuned and frontier models with a config change. This made the A/B tests and the fallback routing simple.
The client, load balancer and endpoint each had different timeouts, so some requests were abandoned by one layer while another kept generating.
FixOne timeout budget (1 second for search calls), enforced at the client, with matching settings below it.
Models and training recipes
Many help articles answer the same question, so contrastive training punished the model for retrieving correct alternatives.
FixA no-duplicates batch sampler, and mined negatives filtered: skip the top 10 ranks and drop anything scoring above 95% of the positive.
Our first embedding run used a 512-token limit while serving embedded longer policy pages. It barely helped on those pages.
FixMatch the training max length to the serving chunk size. Larger chunks also retrieved better for policy pages.
A 0.6B reranker got worse after fine-tuning. A larger model distilled from LLM scores improved steadily.
FixDistillation from an LLM teacher rather than plain labels, on a reranker of at least base size.
Rank 16–32 was enough; 64 added nothing. General ability dropped sharply after fine-tuning, which was acceptable only because the model does one narrow job.
FixKeep the model's job narrow. If it ever has to chat, mix general instruction data back in.
Later checkpoints often scored lower on the eval set while the LLM judge scored them higher.
FixEvaluate every checkpoint on task metrics and pick by those, with the judge as a secondary signal.
Embeddings and the vector index
In OpenSearch, a knn_vector field's dimension, engine and distance metric are fixed when the index is created.
FixOne index per model version, background backfill, an eval gate, then an alias swap. Plan for 2x storage during overlap.
The online embedding server is built for requests, not millions of documents.
FixA separate batch job on a GPU instance. Tokenization on the CPU became the bottleneck, so we added CPU workers and grouped texts by length.
One ONNX build couldn't find the CUDA provider and quietly ran on CPU, about 100x slower, with no error.
FixAssert the execution provider at startup and fail fast if the GPU isn't in use.
A newer, smaller embedding model scored below the older multilingual model on our ticket set before any fine-tuning.
FixModel selection on our own eval set, across every language with real traffic.
What we'd tell the next team
It was the most valuable asset in both projects.
Hybrid search plus a reranker, or a prompted frontier model, got us part of the way with little effort. Fine-tuning had to earn its place against them.
Both successful models were small and did one job.
Data cleaning, evaluation and rollout took about 80% of the effort. GPU availability was a bigger constraint than price.
Reindexing, retraining, version pinning, warm replicas and seasonal peaks were all things we wished we had designed at the start.