How to choose an AI development agency: 10 questions that expose weak vendors
Pick the agency that can show you production systems, not case studies; that prices in fixed-fee phases; that demos working software weekly; that answers security questions with specifics; and that has told a client no. Yes, we're an AI agency telling you how to shop for AI agencies. Use these questions on us too. That's the point.
The 10 questions
"Show me something you built that is running in production today."
Good: a live system, real users, real volume, and they can explain the architecture. Bad: a slide titled "Case Study" with a logo and a percentage. Pilots that never shipped don't count. Demos in their own environment don't count.
"Who exactly will do the work?"
Good: named senior people, and the person pitching is one of them. Bad: the A-team sells, the B-team builds. Ask if the people in the room will be in the standups. Watch the pause.
"How do you price, and what's fixed?"
Good: fixed-fee phases with defined outputs. Diagnostics should cost $5k to $15k, not $150k. Bad: open-ended hourly. Hourly billing pays the vendor more the slower they go. Think about that incentive for one minute. Full market pricing: what AI systems cost in 2026.
"When do I see working software?"
Good: a demo in your environment within two to three weeks, then weekly. Bad: a "discovery phase" measured in months. If the first artifact is a deck, the last artifact is usually also a deck.
"Where does my data go?"
Good: a specific answer: which model providers, what's retained, no-training clauses, access controls, residency options. Bad: "we take security very seriously." That sentence has preceded every breach in history. Ask if anyone on the team holds a security certification and what audits they've survived.
"What will it cost to run after launch?"
Good: an inference, hosting, and maintenance estimate in the proposal, typically 15 to 25 percent of build cost per year. Bad: silence. A vendor who won't quote operating cost is planning to surprise you with it.
"How do you measure whether the AI is actually working?"
Good: evaluation suites, accuracy thresholds, fallback behavior, and monitoring after launch. Bad: "the model is state of the art." The model is not the system. The system is what you're buying.
"Tell me about a project you talked a client out of."
Good: a real story with a real number saved. Bad: they've never said no. An agency that agrees with every idea is selling hours, not outcomes. Most AI ideas should die in week one; ask how many of theirs do.
"Who owns the code, the prompts, and the data when we're done?"
Good: you do, in writing, including evaluation sets and documentation. Bad: "platform fees," proprietary lock-in, or IP terms that make leaving expensive. Lock-in is a pricing strategy disguised as architecture.
"Have you operated a business, or only advised them?"
Good: operators who have owned a P&L and lived with their own systems. Bad: career advisors who have never been paged at 2 a.m. The difference shows up exactly when something breaks.
The Canada question
If you're a Canadian company in a regulated industry, ask about data residency, PIPEDA, and PHIPA by name. A Canadian agency that already works inside these rules removes a quarter of your legal review. It also helps when the agency has shipped into healthcare specifically; here's how regulated AI builds actually work.
- Production references beat case studies. Demand to see running systems.
- Fixed-fee phases align incentives. Hourly rewards slowness.
- Security answers should be specific. "We take it seriously" is not an answer.
- The best predictor of a good vendor: they have told someone no.
Related questions
What should I look for in an AI development agency?
Shipped production systems you can see, fixed-fee phased pricing, weekly working demos, named senior people who do the work, real security credentials, and an honest run-cost estimate. Avoid agencies whose main output is strategy decks.
What are red flags when hiring an AI consulting firm?
No production references, hourly open-ended billing, demos only in their environment, vague data-handling answers, a strategy phase longer than four weeks, refusal to quote operating costs, and never telling you an idea is bad.
Should I hire a local AI agency in Toronto or work remotely?
For regulated Canadian industries, a Canadian agency simplifies residency, PIPEDA and PHIPA conversations, and contracting. Beyond compliance, shipping record beats geography. A Toronto agency that ships beats a famous one that decks.