3 Key Steps in Assessing AI Vendors

DigitalGenius is publishing a series of articles to empower you to navigate through the world of AI, helping you onboard the right AI company that practically solves real problems.

Phill Brougham
Posted by Phill Brougham

Subscribe to the DigitalGenius Blog

Get the latest content from the intersection of artificial intelligence and customer service delivered right to your inbox.

Keep up to date with the latest insights
in artificial intelligence and customer service



Last week’s blog focused on preparing your team to conduct market research, helping you to accelerate the research process while shortlisting the most relevant providers to meet your business needs. Let’s examine the next step.

Your team has now provided you a curated list of providers with snazzy names ending in .ai or .io, so now what? In this step, the Assessment Process, we’re going to build on the assignment you gave to your team, which was 1) Diagnose Pain 2) Define Decision-Making Process and 3) Assess Business Readiness. We begin with addressing the bigger picture and then get to the nitty-gritty by the end.

1) Define Objectives

During assessment processes things change. New inputs emerge, you’ll encounter unanticipated vendor capabilities, or conditions inside your business transform. So it’s important that there’s a fixed point of reference to the objectives.

Objectives is linked to Pain, but is more expansive. Ideally, any AI project is linked to your stated corporate or business goals. AI initiatives are transformation projects, demanding a change management program and executive support. Therefore this initiative must align to the business goals of the next 12 months and beyond. Articulating the project goals in the same vocabulary, and tethering them to exactly how they help the business in achieving its goals will help tremendously in focusing your assessment and getting a project approved.

Once objectives are set, decide how to measure them, and make sure you can do so independently through software or processes. Metrics and measurements are important, because the thing that you want to see improve may not be what the provider actually targets - use this as an evaluation criteria (see #2).

For example, if the success of the AI software is based on Customer Satisfaction (CSAT) improvement, ensure the vendor does it, measures it, and has case studies to support it. Doing one thing with a knock-on to CSAT should make you reconsider.

Finally, what happens if metrics improve? While everyone loves to see a graph heading top right, how do you quantify the value of improvements? In our CSAT example, if scores go up 5 points on average, how does that translate into business success? Are improvements linked to repeat customers, basket size, social promotion, and especially, revenue performance? If so, the more likely that the project will be 1) approved, and 2) successful.

With a clear project objectives framework, advancing or removing vendors in the process will be more decisive, efficient and anchored to your business needs.

2) Determine Evaluation Criteria

As the other side of the coin to the decision-making process, evaluation criteria are critical to weeding out irrelevant or unsuitable providers. Using variables important to project success and to the business, you’ll have a steady method of comparing the options.

Relevance to business objectives is an obvious starting point, you could consider variables like:

  • Ease of integration
  • Scalability to business or channels
  • Provider’s industry experience
  • ROI (see #3)

There are also some AI specific topics that could be considered as part of the criteria:

  • AI expertise - who works there and what is their background in data science?
  • Marketing & positioning - what customers do they have and how specific is language on the topic? I.e. how much generic AI phrases against practical results?
  • Company investors - what is their track record, especially in AI?

Armed with specific evaluation criteria, you should considerably accelerate the assessment process and focus on vendors that meet your needs.

3) Return on Investment

ROI is that wonderful divider between a nice-to-have pet project and a critical business initiative. Beware the AI company that cannot illustrate ROI. Ask providers for a pilot or a trial if they are unable to justifiably provide an ROI, so you can measure success (see above)

Unsurprisingly, there aren’t many technologies that get by without defensible ROI. Businesses stake resources, time and reputation to onboard new technology, even more so with emerging tech like AI - ROI is a good way to filter which vendors are real, and which are selling the dream. Dreams are good but we all have to wake up some time.

DigitalGenius’ method of measuring ROI has developed through time and a lot of learning: by projecting the number of customer journeys AI can automate total savings are calculated, from which ROI can be clearly extrapolated (learn more here).

This final step separates the wheat from the chaff, smoothes sign-off from procurement and protects the buyer and(!) imposing another success metric for the vendor.


Following these steps should be enough to:

  • Frame your assessment process with consistent objectives and goals
  • Ensure you are speaking to vendors that align to your focus
  • Separate real AI from the hype
  • Create a defensible framework for procurement and finance to sign-off

Subscribe to the DigitalGenius Blog

Get the latest content from the intersection of artificial intelligence and customer service delivered right to your inbox.

Keep up to date with the latest insights
in artificial intelligence and customer service