All field notes

Engineering3 min read

Virtual queues, bot walls, and why we stop at them

Some award pages refuse automated visitors outright; others admit everyone through a waiting room built for humans. Both are answers, and we treat them as answers.

Survey enough airline award pages with one polite request each and you meet two kinds of gatekeeper. This essay is about what they are and why, on meeting either, this product records a wall and moves on — a policy that costs us coverage and that we consider load-bearing. It is deliberately not a technical tour; describing gates in the detail that would help someone slip through them is a genre we decline to write in.

What a bot wall is

A bot wall is software that decides, before the airline’s own systems see a request, whether it appears to come from a person’s browser. When it decides not, the request is refused or challenged. A handful of vendors provide most of these walls, and our own coverage pages name the ones we met where we met them: Qantas’s reward finder returns 403 to any automated client, and American’s and United’s booking paths sit behind a commercial bot manager that refuses automated requests regardless of any permission we hold. Our survey method was one ordinary HTTPS request with a truthful user agent, no retries and no evasion — so a refusal is not ambiguous. The wall saw us coming precisely because we made no effort to be unseen.

What a virtual queue is

The other gatekeeper is not trying to detect machines at all. A virtual queue — a waiting room — admits visitors in turn when a page expects more demand than it can serve, granting each a timed session and requiring a return to the back of the line when it lapses. It is capacity control, built around human patience and human sessions. A search product cannot sit in a waiting room on your behalf without becoming, in effect, a machine occupying seats in a queue built for people — which is its own answer to whether we should.

A wall is not a puzzle the airline set for us to solve. It is an answer, and the answer is no.

Why we stop

The techniques that get past these gates exist and are not secrets: browsers dressed as people, addresses rented in residential blocks, challenges solved by services built to solve them, member logins borrowed to reach members-only inventory. We use none of them, and the reason is not that they fail. It is that each one obtains data whose owner has just finished refusing it — the wall is the refusal, delivered mechanically — and a product founded on provenance cannot start its supply chain with a routed-around no. We hold no member credentials, ask for none, and record a program that requires one as blocked rather than worked around.

The pattern underneath

Step back from the individual gates and they mostly say one thing: award inventory is a loyalty benefit, and airlines increasingly treat the search for it as members-only even when a revenue fare is shown to anyone. That reframes the entire problem. A members-only product does not become publicly searchable through cleverness; it becomes searchable through an agreement with the program that owns it. So a wall, for us, is a routing instruction: it moves a program off the engineering backlog and onto the commercial one. On the day an agreement exists, the programs page changes first — and until then the honest coverage number is the one we publish, with the walls counted as walls.