The problem
You select an experience to book on Headout, pick the date and time, move to the checkout page, fill in your details and click Confirm, and suddenly, you’re shown the following error.
This happens hundreds of times every day on Headout. The customer reaches the final step and is ready to pay, when they’re told the slot is no longer available. The reason behind this is not that demand was high, but that the inventory data on our end had become stale.
Headout is connected to hundreds of different Supply Partners (SPs) through which we provide the different experiences to the user. The architecture using which we fetch inventory is straightforward. We have cron jobs set up for each SP. Each cron is set up for a particular date range. Every x minutes/hours, the system polls the inventory for all the tours under an SP from their API and updates it in our database. If the value of x is large enough, it is possible that the inventory data becomes stale by the time the user books the experience on Headout. To mitigate dirty bookings, we introduced a feature called Live Inventory Check (LIC) which checks in realtime while the user makes a booking if the slot is still available. If the slot is sold out, we show the user an error which you saw above.
While this helps us avoid dirty bookings, the drop off at this stage is massive. We’re talking about a stage at which the user was ready to pay. About 50% of the users end up booking a different slot but the remaining never end up finding their way back.
The hackathon
Every year, Headout organises an internal hackathon called Hackin. We team up and work on ideas over a sleepless weekend of coding, caffeine and partying. We’re allowed to work on any idea within Headout. The best projects win prize money and a trip if they’re taken live on production.
The idea
This year, we decided to tackle this problem of lost bookings due to LIC Failures. Here’s the idea we submitted.
Today’s inventory refresh strategy is time-based. We propose a demand-aware refresh engine that continuously prioritizes inventory updates based on factors such as user traffic, booking velocity, price volatility, and LIC failure rates. The system automatically allocates refresh capacity where stale inventory has the highest revenue impact.
Let me break that down for you. Each SP has multiple vendor tours. Each vendor tour is essentially a bookable experience that you see on Headout. We proposed a demand-aware engine which would constantly evaluate the LIC failures, frequency of bookings and other similar parameters of each vendor tour, assign them a score and accordingly set their refresh cron frequency.
Why make it so complicated? Why not just sync every vendor tour as frequently as possible? While that would solve the issue, every Supply Partner caps how many API calls we’re allowed to make. If we exceed the cap, we get throttled - not just for that tour, but for every single tour under that partner. So we can’t just sync everything every five minutes and call it a day. We need to be smart about where we spend a limited budget of API calls.
Thus, our idea was finalised. We named this project Clockwork and our team TVA (hi-5 Marvel fans) and set out to solve this problem the right way.
The iceberg
From the outside, this looked like a straightforward scoring problem. Score each tour by how much it needs fresh data, assign a frequency, done.
Then we started the brainstorming session, and the iceberg revealed itself.
Booking velocity is one signal, but a tour with high bookings and a high LIC failure rate needs a very different response than one with high bookings and zero LIC failures. The scoring formula had to balance competing signals without them canceling each other out. We needed a dead zone so the system wouldn’t thrash on noise. We needed log-space math so that a frequency change from 15 minutes to 30 felt proportionally the same as 6 hours to 12. We needed to enforce rate-limit budgets across groups of tours sharing the same partner account and distribute that budget fairly when demand exceeded supply.
What looked like a weekend project turned into a genuine systems design challenge. Getting every feature - scoring, rate-limit enforcement, zero-inventory detection, auto-pause, schedule diffusion - to coexist without contradicting each other was the actual hard problem. And honestly, the most fun one.
The score
Everything in Clockwork starts with a single number per tour. We call it the score. A high score means “this tour needs fresh data, poll it more often”. A low score means “this tour is fine, back off and save the budget”.
The score is built from two signals.
The first is the LIC failure rate. This is the most direct measure of the problem we set out to solve. If a tour keeps failing live inventory checks, its data is stale and customers are hitting that error at checkout. The more it fails, the more it needs attention.
The second is booking velocity. How fast is this tour selling compared to before? A tour that suddenly starts selling is a tour whose inventory is changing quickly, so we want to keep up with it even if it hasn’t started failing yet.
We combine the two into one number using a weighted average. The failure rate gets 70% of the weight and booking velocity gets 30%. The reasoning is simple. Failures are the actual pain. Bookings are an early warning. So we let failures do most of the talking, and let bookings nudge the score when something is heating up.
This weighting is also what quietly solves the competing-signals problem from earlier. A tour with lots of bookings and lots of failures scores high and gets polled aggressively. A tour with lots of bookings but zero failures scores much lower, because the signal that matters most is telling us everything is fine. The two signals don’t cancel each other out, because they were never equal to begin with.
Turning a score into a frequency
Once we have the score, we need to turn it into an actual cron frequency. The new interval is the current interval scaled by a factor derived from the score. A strong score shrinks the interval so the tour polls more often. A weak or negative score grows it.
We also cap how much any tour can move in a single run, at 50% in either direction. Gradual, proportional nudges are easier to trust and easier to debug.
Then there’s the dead zone. If a tour’s score is close to zero, we leave it completely alone. Without this, tiny fluctuations in bookings or a single stray failure would cause every tour to jiggle its frequency on every run, generating churn for no real reason. The dead zone means Clockwork only acts when it actually has something to say.
Finally, we don’t allow arbitrary intervals. The result gets snapped to the nearest value from a fixed list: 15 minutes, 20, 30, 60, and so on up to a few days. Not because odd intervals break anything technically, but because a clean set of frequencies makes the whole system far easier to reason about when you’re staring at it trying to figure out why a tour is behaving the way it is. It also makes it easier for us to fit everything in within rate limits by making the calculation easier.
Here’s what the scoring and frequency change looks like in our dashboard. For this particular vendor tour, the 0-30 day segment received a score of 0.53, so it’s frequency was bumped up from 60 minutes to 30 minutes.
The budget problem
Now the hard part. Scoring tells us what each tour wants. But wants are infinite and API calls are not.
After scoring has decided what everyone wants, we have to force the total back down to fit inside the budget. We do this in two layers.
The first layer is rebalancing. Think of it as an hourly budget for each partner account. We work out how many calls per hour the account can handle, keep a safety margin so we never sit right at the edge, and then add up what all the tours are asking for. If it fits, great, everyone gets what they want. If it doesn’t, we apply the slowdowns first, since those free up room, and then scale back the speedups proportionally so everyone shares the squeeze equally. Nobody gets singled out.
The second layer is diffusion, and this one is more subtle. Even if the hourly total fits, you can still get in trouble. Imagine twenty tours that all poll every hour, and all of them happen to fire at the top of the hour. For that one second you’ve sent twenty simultaneous calls, and the partner sees a spike even though the hourly average was fine.
So diffusion staggers the start times. If two tours poll every 30 minutes, one starts at minute 0 and the other at minute 15, so they never collide. To be sure this actually works, Clockwork then simulates the entire schedule down to the millisecond and finds the busiest moment, the single point in time where the most calls overlap. If that peak still breaks the limit, it stretches the intervals a little and simulates again, repeating until the busiest moment fits under the cap.
Knowing when to stop
Two situations don’t need scoring at all. They need a hard rule.
The first is zero inventory. If a date segment keeps coming back empty, every sync returning no available slots at all, there’s no point spending calls on it. Clockwork detects this and pushes that segment to the slowest possible frequency. There’s no point asking a question whose answer is always “nothing”.
The second is a tour where every single sync has failed over the past week. That’s not a tour that needs more frequent polling. That’s a tour that’s broken somewhere upstream, and hammering it harder won’t fix it. Clockwork pauses it outright and flags it for our operations team to look at, instead of quietly wasting budget on something that will never succeed.
These two rules sit above the scoring engine. No matter what the score says, a broken tour gets paused and an empty segment gets slowed down. This massively frees up our rate limiting budget.
Where it landed
What started as a weekend hackathon idea turned into a proper little engine: a scorer, two layers of rate-limit enforcement and hard rules for the broken cases. None of the individual pieces are exotic. The difficulty, and the fun, was in getting them to agree with each other. Scoring wants to speed things up. Rate limits want to slow things down. Diffusion wants to move things around. The whole game was making those forces settle into one schedule that made sense.
Clockwork is now projected to recover at least a million dollars a year in bookings which would have otherwise been lost to LIC failures.
A fun little graphic Claude made for our dashboard. Each gear is a Supply Partner: its size shows how many vendor tours it has, its colour shows how much of the rate limit we’re using, and its speed shows how often we’re polling it.
What we took away
The biggest surprise wasn’t technical. It was realizing that a problem we’d mentally filed as “medium-sized optimization task” was actually one of the deepest, most interconnected challenges we’d tackled, and that a small team with shared context could move through it faster than anyone expected.
It sparked a real conversation with our manager during the hackathon: what if we just worked like this more often? Small teams, one problem, full ownership and real speed instead of everyone in silos watching things get pushed quarter after quarter. Hackin didn’t just produce a project. It produced a proof of what three people can do when the problem is sharp and the runway is clear.