The question
JobHuntOS routes Claude, Gemini, and Groq with SSE streaming and provider failover. Neocortex pushes that further with a six-provider chain. The engineering is real. The research question is still open: what should “the request succeeded” mean when the model that answered is not the model you wanted?
Availability is the easy metric. Quality under substitution is the hard one. A fallback that returns in 400ms with a weaker model can be worse than a 2s wait for the primary — or better, if the user is mid-flow in a Chrome extension and the tab is about to background.
What I have observed shipping this
Timeouts and 429s are not the same failure. One is “try the next provider.” The other is “back off this key and maybe the whole vendor.” Treating them as one error class makes the chain look busy and still fail.
Streaming changes the contract. Once tokens have started, swapping providers mid-response is a different product than retrying before the first byte. Most router diagrams ignore that.
BYOK (bring your own key) makes failover a user-policy problem, not just an infra one. Some users have only one key. The honest UI is “this provider is down,” not a silent hop they did not consent to.
A sketch of the model I want
Score each attempt on four axes: time-to-first-token, cost, a cheap quality proxy (length, schema validity, or a tiny grader), and user-visible interruption. Failover is then a policy over that score, not a linked list of vendors.
I do not have a dataset or a paper here. I have production traces from my own products and a suspicion that “N providers” is the wrong unit. The unit is interruption.