The premise.
Almost every enterprise we work with has either run, is currently running, or is preparing to run an AI vendor selection. The conversations we hear sound similar across companies and across sectors. A short list of three to seven vendors is circulated. Each is invited to present a capabilities deck. The procurement function builds a scoring matrix. Demos are arranged. References are checked. A decision is made. Some months later, the program is reviewed. Roughly half the time the review is uncomfortable; some meaningful fraction of the time the program is paused or quietly wound down. The vendor is blamed. A second vendor is selected. The cycle repeats.
The reason this happens is not that the vendors are uniformly poor. The reason is that the evaluation criteria most enterprises apply select for the wrong things. Capabilities decks select for marketing investment. Demos select for engineering investment in the demo. Scoring matrices select for the vendor's ability to write to a scoring matrix. References select for the vendor's most flattering accounts. None of these criteria predict the operational outcome the buyer actually wants, which is a function that operates measurably better twelve to eighteen months after the engagement begins.
This piece lays out a buyer's framework for evaluating AI transformation vendors that selects for that operational outcome rather than for the procurement performance. It is written for a CIO, a CFO, a COO, a head of strategy, or a board member who is about to make a multi-million-dollar decision and wants to make it well. The framework is deliberately uncomfortable in places, because the criteria that actually predict success are the criteria that vendors find easiest to dodge.
Why most AI vendor selection processes fail.
There are five recurring failure modes in AI vendor selection. Each is a symptom of a deeper category error: treating a transformation as a software purchase.
RFP fetishism. The procurement function defaults to its standard tool, the request for proposal, which is designed for the purchase of well-specified, comparable goods or services. AI transformation is neither well-specified at the point of purchase nor straightforwardly comparable across vendors. Forcing the engagement into the RFP frame produces fifty-page documents that compete on completeness of response rather than on quality of judgment. The vendor with the best proposal-writing capability wins the procurement, which has limited correlation with which vendor will produce the best operational result.
Platform-capability confusion. A buyer learns that a vendor has a "platform," sees a polished interface, and assumes the platform represents the vendor's capability. In practice, the platform is often a packaging of generally available technology (foundation models, vector databases, orchestration frameworks) wrapped in the vendor's brand. The vendor's real capability is the human judgment that determines how to deploy these components against a specific operational problem. Buyers who select on platform select for marketing budget. Buyers who select on judgment select for an entirely different set of vendors.
Demo theater versus operational reality. Every serious vendor in this market can produce an impressive demo. The demo shows a curated scenario, a curated dataset, a curated outcome. The operational deployment will involve none of those curated conditions. The demo therefore tests the vendor's ability to demo, not the vendor's ability to deploy. Buyers who weight demos heavily are systematically misled in the direction of the most theatrical vendors.
Confused buyer accountability. In many organizations the AI transformation decision is owned by a committee, which means it is owned by no one. The CIO wants to know about technology and security. The CFO wants to know about cost and return. The business unit head wants to know about disruption to operations. Each of these is a legitimate concern, but in aggregate they produce a selection process that satisfies none of them clearly and leaves no individual accountable for the operational outcome. The vendor signs the engagement with a committee of buyers and then operates, in practice, without a single executive owner. When the program drifts, no one has the authority or the standing to correct it.
Treating the engagement as a software purchase rather than an operating shift. The most consequential category error is treating AI transformation procurement as analogous to buying enterprise software. Enterprise software is purchased, deployed, configured, and then operates. AI transformation, in contrast, is a redesign of how a function operates. The vendor is not delivering a tool to be installed; the vendor is delivering a change to the way work is done. Buyers who evaluate on the software-purchase frame miss the operational-redesign dimension entirely and end up surprised when the actual deployment requires twelve months of organizational adaptation that the software-purchase frame did not budget for.
The five categories of AI transformation vendor.
Before evaluating any specific vendor, the buyer needs a clean mental model of the categories of vendor in the market, because each category operates under different economic incentives and has different structural capabilities. Buyers who shortlist across categories without understanding the differences will compare vendors on dimensions where the differences do not matter and miss the dimensions where they do.
The first category is the large strategy and systems consultancy. These are the firms whose names appear in every CIO briefing and whose proposals run to several hundred pages: Deloitte, Accenture, PwC, KPMG, EY, IBM Consulting, McKinsey (and its QuantumBlack AI specialist arm), Bain (and its Vector arm), Boston Consulting Group (and its BCG X arm), and the major Indian-headquartered system integrators including Tata Consultancy Services, Infosys, Wipro, and HCL. The structural advantages of this category are scale of headcount, global delivery, brand recognition, and the ability to staff a program of any size. The structural disadvantages are the leverage model that pushes work down to junior consultants, the standardized methodology that produces standardized results, and the incentive to extend the engagement scope rather than complete it. The senior partner who sold the engagement is rarely the senior person who is delivering it.
The second category is the AI-native consultancy. These are firms founded in the last few years specifically to serve the AI transformation market. Names that recur in this category include the professional-services arms of the major model laboratories (OpenAI, Anthropic), specialist firms that branded themselves for the AI era, and the AI practices of the larger cloud-platform providers. The structural advantages are a focused practice, current technical knowledge, and proximity to the model laboratories that produce the underlying technology. The structural disadvantages are limited operating experience in the buyer's industry, a tendency to recommend more technology rather than less, and a frequent absence of senior operators who have run the kind of function the vendor is being asked to transform.
The third category is the boutique senior-led advisory firm. These are smaller firms structured deliberately around senior practitioners, with no junior consultants between the buyer and the work. The structural advantages are senior judgment at the point of delivery, alignment between the partner who sold the work and the partner who delivers it, and the ability to walk away from misaligned scope rather than expand into it. The structural disadvantages are smaller bench depth, slower scaling on larger programs, and the requirement that the buyer's leadership remain engaged because there is no junior layer absorbing the operational details. Asta operates in this category; this article is not the place to argue for the category over the others, but the buyer should know it exists and what it offers.
The fourth category is the platform-vendor professional-services arm. This category includes the professional services teams attached to major AI platforms (Microsoft, Google, Amazon Web Services), the systems-integration partners credentialed by those platforms, and the vendors whose primary product is an AI orchestration or governance platform. The structural advantage is deep technical depth on the specific platform the vendor sells. The structural disadvantage is the incentive to recommend solutions that route through the platform the vendor sells, which constrains the design space in ways the buyer may not be able to see.
The fifth category is the point-solution vendor. These are firms that sell a specific AI capability — a particular agent product, a particular vertical application, a particular workflow automation — rather than transformation services. The structural advantage is that what they sell is concrete and measurable on day one. The structural disadvantage is that the point solution rarely addresses the operational redesign that determines whether AI transformation actually compounds in the function. Point-solution vendors are often the right answer for a specific bounded problem; they are rarely the right answer for the transformation question.
Most buyers shortlist across at least three of these categories without naming the categorical differences. The first improvement the buyer can make to the evaluation process is to be explicit about which category each shortlisted vendor falls into and what the structural implications are for the engagement.
The five questions that actually predict success.
The substantive evaluation, after the categorical sort, comes down to five questions. Each question is designed to surface real capability rather than marketing capability. Each question is the kind of question that a well-prepared vendor will answer crisply and that an unprepared vendor will dodge, qualify, or change the subject around. The buyer's job during the evaluation is to ask these questions, listen to the answers, and observe whether the answers contain operational specificity or rhetorical fog.
Question one. What is your methodology for choosing what to augment first? The single most expensive AI transformation mistake is augmenting the wrong process. A vendor that begins with a model or a use case has already made this mistake; the buyer is just along for the ride. The question separates vendors who have a process-first methodology from vendors who have a technology-first methodology. A strong answer names a specific diagnostic approach (we map the function as it operates today, we identify where senior judgment is being spent on senior-adjacent work, we size value, we score feasibility, and we sequence the roadmap), takes the buyer through a worked example, and acknowledges what the vendor would not augment. A weak answer talks about the vendor's platform, the vendor's accelerators, or the vendor's library of use cases. The companion piece on this site, The AI Transformation Process Diagnostic, lays out one such methodology in full so the buyer can compare what the vendor describes against a documented reference.
Question two. How do you handle our data, our intellectual property, and our derived models? Every AI transformation engagement creates derived data assets: the prompt libraries developed during the work, the fine-tuned models trained on the buyer's data, the embeddings that encode the buyer's documents, the agent configurations specific to the buyer's processes. The question is who owns these assets and what the vendor's rights are. A strong vendor has a clear, written position: the buyer owns its data, its derived data, its fine-tuned weights, and its agent configurations; the vendor has the right to use de-identified learnings for its own methodology development; nothing the vendor builds for the buyer is reused on another buyer's engagement without explicit permission. A weak vendor either has no clear position, has terms that retain rights to the buyer's data for the vendor's training corpus, or has terms the buyer cannot fully understand. The data and intellectual-property terms are the single most important contractual section for an AI transformation engagement and the section that most buyers underweight at signature.
Question three. What happens when the model is wrong, the answer is hallucinated, or the system fails in production? Every AI system fails. The question is whether the vendor has thought through failure as a design constraint or has assumed away the possibility. A strong vendor talks about confidence thresholds, human-in-the-loop checkpoints, fallback behaviors, error escalation paths, monitoring, model evaluation cadences, and how the team will respond when the production system produces an output that is wrong in a costly way. A weak vendor talks about how the model has been trained on a large dataset and produces accurate results. The latter answer signals that the vendor either does not understand the failure surface of probabilistic systems or is deliberately not surfacing it during the sale. Either is disqualifying for a serious engagement.
Question four. Who specifically owns this engagement and how senior are they? Every AI transformation engagement has a senior partner who sold it and a delivery team who does the work. The question is whether those are the same person, or whether the senior partner exits after signature and the delivery is staffed by a team several levels below. A strong vendor names the specific senior person who will own the engagement, commits to that person's involvement in writing, and is candid about how the rest of the team is structured. A weak vendor speaks in terms of a delivery team without specifying who runs it, or names a senior person without committing to that person's hours or accountability. The senior-staffing question is the single most reliable predictor of engagement quality, because the senior judgment is the bottleneck on every consequential decision the engagement will make. A program staffed by junior consultants with senior oversight on a weekly call will produce a junior result, regardless of the vendor's brand.
Question five. When is the engagement designed to end, and what does the buyer own at the end? The healthiest AI transformation engagements are designed to step out cleanly when the function is operating at the new cadence. The vendor builds, integrates, change-manages, and operates alongside the buyer's team until the buyer's team is operating the augmented function independently, and then the vendor leaves. The question surfaces whether the vendor has built the engagement around an exit or around a continuation. A strong vendor describes the exit criteria, the knowledge transfer that occurs, the documentation handed over, and the conditions under which the engagement would extend versus end. A weak vendor describes the engagement as ongoing, recurring, or evolving — language that signals the vendor's interest in extending revenue rather than completing the work. The exit question also surfaces a deeper alignment issue: vendors whose business model depends on ongoing engagement length will design engagements that depend on ongoing engagement length, regardless of what the buyer's operational interest requires.
Red flags during the evaluation.
Some signals during the evaluation reliably indicate that the engagement will be more problematic than the procurement performance suggests. Buyers who learn to recognize these signals reject vendors more efficiently and improve the operational quality of their final selection.
The first red flag is the vendor that claims to be able to do anything with AI. The phrase usually appears in the opening five minutes of the first meeting and signals that the vendor is selling on aspiration rather than on capability. Real AI transformation is bounded and specific. A vendor that names what it will not do is more credible than a vendor that claims to do everything. The buyer should explicitly ask each shortlisted vendor what kind of engagement the vendor would decline and why; the quality of the answer is informative.
The second red flag is the vendor that resists sharing methodology. Methodology is the vendor's actual product. A vendor that treats methodology as proprietary and refuses to describe it concretely either does not have a methodology or has decided that revealing it would commoditize the sale. Either explanation suggests the engagement will be structured around the vendor's economic interest rather than the buyer's operational interest. Strong vendors share methodology in detail during the sales process, sometimes in writing, sometimes in published thinking, because they know that the implementation is the hard part and the methodology is what gets them invited to do the implementation.
The third red flag is the bait-and-switch staffing pattern. The senior partner presents the engagement during the sales process, the contract is signed, and the actual delivery team turns out to be several levels below the partner. The partner appears on a steering committee call weekly. The work is done by people the buyer did not meet during the sale. This pattern is so common in the larger consultancies that it is almost expected; serious buyers protect against it contractually by naming specific individuals in the engagement letter and reserving the right to approve any substitution.
The fourth red flag is excessive proprietary-framework branding. Every consultancy has frameworks; this is not the problem. The problem is when ninety percent of the methodology is repackaged industry knowledge under a proprietary brand name, designed to make the buyer feel that engaging the firm is the only way to access the framework. A buyer can usually detect this pattern by asking specific questions about what the framework actually does that other approaches do not; if the answer is vague, the framework is probably packaging rather than capability.
The fifth red flag is hesitation on data security. AI transformation engagements involve sending the buyer's data to the vendor or routing the buyer's data through the vendor's infrastructure. The vendor's posture on data security — encryption in transit and at rest, the SOC 2 or ISO 27001 certifications the vendor holds, the data residency commitments, the vendor's handling of subprocessors, the vendor's incident response process — should be answered confidently and in writing. A vendor that hesitates, that offers verbal assurances, or that asks the buyer to trust the vendor's reputation is signaling that the vendor either has not invested in the security posture or does not understand the question.
The sixth red flag, and the subtlest, is the absence of references in the buyer's specific sector or stage. Every vendor will provide three glowing references; the question is whether the references match the buyer's situation. A reference from a Fortune 500 manufacturing company tells a healthcare CIO very little about what the vendor will do for a healthcare company. The buyer should ask for references that match in sector and in company stage, and should be skeptical when the vendor cannot produce them.
How to run the evaluation.
The evaluation itself runs in four phases. Most buyers compress this into a single procurement process measured in weeks; the better discipline is to allow the four phases to play out across roughly four months, which sounds long but is short compared to the eighteen months of operational consequence the buyer is procuring.
Phase one is discovery and shortlist. Three to four weeks. The buyer's executive sponsor — and this should be a single named executive, not a committee — meets with eight to twelve candidate vendors for forty-five-minute introductory conversations. The objective is not to evaluate but to map the landscape, learn the vocabulary, and understand the categorical structure of the market. The sponsor leaves these conversations with a shortlist of three to five vendors who span the categories described above and who can articulate methodology in the first conversation. The shortlist is then briefed on the buyer's specific operational situation under a mutual non-disclosure agreement.
Phase two is the structured five-question evaluation. Three to four weeks. Each shortlisted vendor receives the five questions in writing and is invited to a ninety-minute working session with the executive sponsor and one or two named operators. The session is not a presentation; it is a working conversation in which the vendor's methodology is pressure-tested against the buyer's actual operational reality. The buyer leaves each session with a written assessment of the vendor's answer quality on each of the five dimensions and a clear sense of which vendors have substantive depth and which were performing.
Phase three is a proof of concept. Four to six weeks. The top two vendors from phase two are commissioned to run a paid, bounded, four-to-six-week proof of concept on a specific, named process. The proof of concept is not an experiment; it is a scoped engagement that produces a concrete deliverable — a process map, a value-and-feasibility ranking, a recommended augmentation, an architectural sketch, a risk assessment. The buyer pays both vendors for the work, because asking vendors to do real work for free distorts the engagement and selects for vendors with the budget to absorb pre-sales effort rather than vendors with the discipline to deliver it. At the end of the proof of concept, the buyer has two concrete deliverables to compare and a working sample of how each vendor operates under real conditions.
Phase four is decision and pilot. Two to three weeks for decision, then eight to twelve weeks for the pilot. The buyer selects one vendor based on the comparison of proof-of-concept deliverables and signs an engagement letter for a contained pilot — typically a single function, a single use case, a single team. The pilot is the first real engagement, with measurable success criteria, named owners on both sides, a defined timeline, and clear exit conditions. The pilot is sized so that if it fails, the buyer can absorb the loss and switch vendors without operational damage. If the pilot succeeds, the engagement scales from the pilot into the broader program. The discipline of running a pilot before committing to a full program is the most reliable single defense against vendor-selection mistakes.
Contract terms that matter.
The contractual structure of an AI transformation engagement is materially different from a traditional consulting contract. Several clauses are worth specific attention.
Data ownership and rights. The contract should explicitly state that the buyer owns its data, its derived data, its prompts, its embeddings, its fine-tuned model weights, and any agent configurations developed for it. The vendor's rights to use any of this material should be precisely scoped: de-identified learnings for the vendor's methodology development is reasonable; reuse of the buyer's data in another engagement is not; using the buyer's data to train commercial models is a meaningful concession the buyer should not grant casually.
Model ownership. If the engagement involves fine-tuning a foundation model on the buyer's data, the contract should clarify whether the fine-tuned weights are owned by the buyer or licensed to the buyer. If licensed, the term and termination conditions of the license matter. Buyers who do not negotiate this clause sometimes discover at the end of an engagement that the vendor retains the model that was trained on the buyer's data, which is a significant residual leverage point that should not exist by default.
Audit rights. The buyer should retain the right to audit the vendor's methodology, the vendor's model performance against agreed metrics, and the vendor's security posture. Audit rights are rarely exercised but their existence shifts the operating relationship; vendors who know they can be audited behave differently than vendors who know they cannot.
Performance commitments. The contract should name specific, measurable success metrics for each phase of the engagement, with clear consequences for missing them. A vendor that resists writing measurable metrics into the contract is signaling that the vendor does not intend to be held to them. The metrics should be leading indicators (cycle time, draft quality, senior-hour redeployment) rather than only lagging financial outcomes, because the leading indicators are what the vendor can actually move within the contract period.
Exit clauses. Every engagement should have explicit exit provisions: how the engagement winds down, what knowledge transfer is provided, what documentation is handed over, how the buyer's team takes operational ownership of the augmented function, and the conditions under which the engagement extends rather than ends. Buyers who do not negotiate exit clauses at signature often find at the end of the engagement that the vendor has structural leverage to extend simply because the buyer's team has not been prepared to operate without the vendor.
The first ninety days post-selection.
The decisions made in the first ninety days of the engagement determine whether the program produces an operational result or a series of well-documented meetings. Three operating disciplines matter most.
The first is named accountability on both sides. The vendor names a specific senior partner who owns the engagement. The buyer names a specific executive sponsor who owns the operational outcome. These two people meet weekly. Decisions made in the engagement go through these two people. Programs that operate without named accountability on both sides drift; programs that operate with named accountability tend to converge on the agreed outcome.
The second is measurable leading indicators reported on a weekly cadence. The metrics named in the contract are tracked weekly in a structured operating review. The review covers what moved, what did not, and why. Programs that report only on activities (we held five workshops, we built three prototypes) without reporting on outcomes (cycle time changed by X percent, draft quality scored Y, senior hours redeployed by Z) lose their connection to the operational goal. The leading indicators discipline is what keeps the program connected to the goal.
The third is a clean governance structure with a single steering committee that meets monthly, has a written charter, and has the authority to approve scope changes, escalations, and budget adjustments. Programs without clean governance accumulate scope drift; the work expands to meet the vendor's capacity to bill it, rather than contracting to the operational outcome. Clean governance is the structural defense against scope drift.
When to course-correct.
Most engagements that drift can be brought back on track through a candid conversation early in the program. The buyer who recognizes early signals and raises them in good faith, with the vendor's senior partner in the room, is doing the most useful thing available to the engagement. Wind-down is the option of last resort, not the first. The discipline worth cultivating is the discipline of surfacing concerns while they are still small enough to address collaboratively, rather than waiting for them to become large enough that the only remaining choice is to part ways.
The early signals that warrant a structured conversation, rather than alarm, include: the vendor is struggling to articulate methodology clearly by week six, when it should be operative; the senior staffing differs from what was named in the engagement letter; the pilot's leading indicators are not yet moving by week eight; questions about data security have surfaced that the vendor has not fully resolved; or the vendor's posture under stress has felt transactional in a moment when collaboration was needed. None of these signals, taken individually and addressed early, need to be terminal. A vendor with good faith and senior judgment will respond to the conversation, name what is happening, and adjust. The engagements that recover most successfully are usually the ones where the buyer raised the concern at week six rather than month six.
When the conversation does not produce a course-correction — when two or more of these signals persist after the buyer has surfaced them honestly, when the vendor cannot or will not address them, or when new signals continue to accumulate — the more difficult conversation about wind-down becomes the appropriate one. Even then, the goal is a clean transition rather than a confrontational exit: a written record of what was tried, a knowledge transfer to the buyer's team, and where useful, a recommendation from the vendor about who might be a better fit for the remaining work. Good vendors handle this conversation as well as they handle the engagement itself; that posture is one of the most reliable indicators that the firm was worth the engagement in the first place.
The opposite case — when an engagement is clearly on track and worth extending — has its own recognizable shape. The leading indicators are moving in the agreed direction. The senior partner is materially involved week to week. The buyer's team is learning operational disciplines that will persist after the engagement ends. The methodology is being adapted intelligently to the buyer's specific context. The vendor is willing to say no to scope creep that does not serve the buyer's stated goal. These signals are the operational shape of a working engagement, and they are usually visible by the end of the pilot.
A note on senior-led delivery.
One observation worth stating directly, and at risk of sounding self-interested given the firm publishing this article: the senior-led delivery model is structurally better suited to AI transformation engagements than the leveraged junior-team model that dominates the larger consultancies. The reason is not preference; it is that the underlying technology is changing faster than junior practitioners can be trained on it. A team whose senior members have been doing AI work for six years and whose junior members were trained on a methodology that is now twelve months stale will execute a methodology that is twelve months stale. A team led by senior practitioners who are themselves still learning the technology weekly will execute with current knowledge. The leverage model that worked for traditional consulting work — where the underlying methodology was stable across decades — does not transfer cleanly to a domain where the underlying methodology turns over every six to twelve months. Buyers should weigh this structural consideration when comparing the categories of vendor described earlier, even allowing for the obvious fact that the firm making this observation has a structural interest in the senior-led model.
Closing.
The right vendor evaluation, in the end, is not a procurement event. It is a four-month operating discipline that produces a decision the buyer can live with for the eighteen-month engagement that follows. Buyers who run this discipline well end up with vendors whose methodology fits the buyer's situation, whose senior staffing matches what was promised, and whose engagement is designed to end. Buyers who run a fast procurement process and skip the discipline tend to end up surprised eighteen months later.
The questions in this piece — the five questions that predict success, the red flags that predict failure, the contractual terms that matter, the first ninety days that determine the trajectory — are not new and are not proprietary. They are the questions a serious buyer asks a serious vendor in any procurement of professional services where the operational consequences are large. The AI transformation context just makes the stakes higher and the marketing louder, which raises the value of the discipline rather than lowering it.
If your firm is in the middle of an AI vendor evaluation, the single most useful action you can take this week is to write down your answers to the five questions for each vendor on your shortlist, before the vendors answer them. The exercise reveals what the buyer already believes, which is the baseline against which the vendor's answers should be measured.