The ROI of investments in AI
Discover more from tech for humans
The ROI of investments in AI
Introduction
Building AI is hard. If you work for a company and are thinking about adding AI to some of your processes (literally everyone's concern in 2023), then there are two primary questions that you should be asking:
Who can I trust to build or buy this AI I need from?
Does it make monetary sense to pay for that AI if I can get it?
This post is about the second question. If you have questions about the first, ping me!
Artificial Intelligence is the only thing that most people can discuss in 2023. OpenAI introduced ChatGPT at the end of 2022, changing everything forever.
Rivers of digital ink have been spilled in writing about AI this year, and I don't need to add a lot of noise on top of what has been said. But I will do so for a bit because I can't start talking about AI without mentioning what is happening.
It's unbelievable out there.
What ChatGPT and OpenAI did was not a technological breakthrough; although it is hard to argue against their technology, it works very well. We have had foundational language models for quite a while now. Their stroke of genius was to enable the masses access to this technology. They transformed an odd, word-generating model into something that anyone could play a bit with. That sparked an interest in creativity, and the rest is history.
The funny thing is that one of the intrinsic problems of AI remains: is it worth it?
As a company, should I invest money in, say, adding AI to my photo editing process, my inbound email forecast, or a chat interface to my website so that users can converse with my data instead of browsing around?
Today, I want to discuss how you can measure if an investment in AI is sound. And for that, I want to use three different examples with their own complexities:
You have a repetitive process in place that you want to automate. Examples include adding information for personal cards into a CRM, editing images to add the company logo, or generating a text description for each video uploading to your video-sharing platform (think YouTube, Twitch, Vimeo, etc.)
You need to take action with incomplete information. There is an opportunity cost to making a wrong decision: you need to buy perishable goods before you know how much you are going to sell, you have to manufacture inventory before knowing how much demand there will be for your product, or you need to hire workers for your consulting service firm without knowing how much clients you will get.
You want to increase the usability/friendliness of your site, and you think that having a chatbot that allows customers to ask a robot for that information will result in a higher conversion rate.
The examples below are not a comprehensive list of all the scenarios for which you would want AI, but they probably cover most problems I see working in AI services. The million-dollar question is: how do we know if it’s worth the investment?
Let's see how we can measure ROI for the examples above.
Automating a repetitive process
This is the easiest of the three examples described above and has been the subject of extensive debate in the programming community: we even have jokes/memes about it.
If the task that you are performing is repetitive and remains almost the same every time it is completed, you can measure how often you do it and how long it takes.
In the case of typing into a computer the information in a personal card, depending on the card, more or less information is available (name, LinkedIn, phone number, email, etc). That might lead to varying times. In any case, it should be pretty easy to estimate the average time that the task takes. Let's say that for the personal card situation, it takes about 10 seconds.
The next question is how often you perform the task: do you process ten business cards per hour or one every week? Once you calculate that number, you can calculate the total amount of time spent on the task:
If the employee or the person performing the task earns 20 dollars per hour, you should not invest more than $8.3 monthly on a business card processing system.
The situation is never THAT SIMPLE. What if the processing system is not always accurate? What if I still need to pay for the time, even if it’s not doing anything? What if the delay negatively affects other parts of the business, even if indirectly?
We will not answer all those questions here, but if you are interested, let me know, and I'll consider writing about those topics in the future.
Acting on inaccurate information
The first time I learned about the Value of a Stochastic solution in the context of Stochastic Programming, I was fascinated. It’s one of those ideas that is very simple yet beautiful and makes a lot of sense when you see it.
I won't go into many details here, but let's say that a simple variation on it can be used to estimate the ROI of our uncertain projects.
Imagine that you run a manufacturing business and need to decide how much to produce for the upcoming week. The variable X denotes how much you choose to produce for the upcoming week, c is how much it costs to produce each unit, p is the price you sell your products, and D is the demand for your product.
If your goods are perishable, that means that whatever is not sold within a week needs to be discarded. Under such a model, the profit in a given week is given by:
The table below shows some examples of this for different values:
It’s easy to see that in the setting above, the optimal is to produce X = D. Unfortunately, we don't know D in advance, so that we will need a forecast for it. Depending on the business setting, predicting demand can be very easy or almost impossible.
Even without knowing how accurate our demand forecasting models will be, we can measure how their performance will impact the overall profitability of the company.
The following table shows how much the forecasted D (Df) differs from X and how that affects the price.
With this information, we can go back to our current demand prediction model and check for how much we have missed our estimates in the past. We can now quantify the Value of having more accurate demand forecasts.
It might be that the result of this analysis is that you can't get more profits by implementing a new AI system. It might also be the case that you are leaving millions of dollars on the table, and having access to a forecasting model that is 10% more accurate will be precisely what your company needs.
For completeness, let's assume that we have access to past forecasts (we can see how much product was manufactured and use that as a proxy for expected demand) and actual demand (as sales records).
Imagine that the table looks as follows:
In total, after five weeks, 17 units of profit were lost for having an inaccurate forecasting system. The total profit for those five weeks amounted to 91 units. That means it could have been 18% higher with a better forecasting system, not minor.
Will a chatbot make my site better?
With the rise of ChatGTP, most companies want a chatbot that can answer information about their data. For some companies, adding a chatbot will be a transformative experience that will revolutionize their businesses. For others, it might barely have an impact on their bottom line.
Is there a way to know what to expect before starting?
Well, the answer is yes and no. If what a chatbot will enable is a massive, unprecedented scale, then you are out of lack; it will be tough to measure that before getting started.
On the other side, if what you think will be the driving factor for your chatbot's success is the "personalization" and "human touch" that it offers, then you might have a cheaper way to get started.
My proposal for this last section is simple: fake it until you make it. Hire a human or two, and have them answer as fast as possible, as robot-like as possible, for a week, and see the results.
This works as follows:
You add a chat interface for your website.
You connect the interface with a chat platform so that the humans can interact with the chatbot and reply to many conversations simultaneously.
You can automatically set a starting message: "Hi, my name is XoXo, the friendly robot. How can I assist you today?" so that users think they are chatting with a chatbot.
You can configure it so that an automatic message is sent if the human is taking a long time to answer (because they might be searching their PCs for an answer)
This won't work perfectly, and many users will be annoyed about the long delays, so that's fine. What's important is that you will learn what type of questions your users are asking and how often they interact with your bot.
Along the way, you might learn things such as:
Most questions are related to information on the website that they could not find, or that was not present. Maybe the best solution is to start with an FAQ and take it from there.
Most questions are to perform a particular action not available or implemented on your website, which could be achieved when calling a call center. Think about adding that additional logic to your website.
Most users want a fast way to answer their issues, and they quit if it takes too long. This might indicate that your users DO expect a better customer experience, and developing the chatbot might be a good investment.
At the same time, you should try to measure KPIs for your website and see how they change during the intervention:
Are you getting more sales than average?
Are users staying longer on your website?
Are they clicking on more ads?
Are they filling out a contact form more often?
If you are not tracking the above metrics, you might learn something by doing this experiment, but it won't be as helpful as it could be.
Wrapping UP
Implementing an AI project is a complex decision that most organizations will face in the upcoming years.
Within the vast realms of what AI can do, there are several projects to choose from: some that will have a long-lasting impact on your business and others that will only drain your bank account.
I recommend always starting small, going for the lowest-hanging fruit, and getting a taste of what AI can do for you. After that, you will have time to go big and implement more complex projects.
When deciding which project should go first, consider the three approaches mentioned here to estimate the ROI of your investment.
Happy number crunching, and until the next time!