The 7 metrics to measure the success of your virtual agent

Every company wants to grow. But with thousands of issues to solve and employees working all over the world, supporting a growing workforce has never been a greater challenge. That’s why support teams are increasingly turning to virtual agents to get their people the help they need — anytime and anywhere.

Yet these virtual agents present a new challenge: knowing what good looks like. There’s simply no industry standard for tracking their progress or benchmarking their performance. And with many solutions on the market that all quantify performance in different ways, it’s hard to determine which metrics actually matter.

After working with more than 100 of the world's largest companies, we found that their leaders measure the impact of their virtual agent using these seven core metrics:

1. Resolution rate: the percentage of all service desk issues the virtual agent solves autonomously
2. Time to value (TTV): the amount of time it takes to get results from the virtual agent
3. Time to resolution (TTR): the amount of time it takes the virtual agent to solve employee issues
4. Resource utilization: the percentage of issues solved by existing knowledge base articles and forms
5. Bot preference: the percentage of employees who go to the virtual agent first when they need support
6. Issue prevention: the percentage of issues proactively prevented by the virtual agent
7. Satisfaction score (CSAT): the percentage of employees satisfied with the virtual agent’s support

Resolution rate

The primary goal for most virtual agents is to solve as many support issues as possible, automatically, with no intervention from a support team.

Tracking the resolution rate offers a high-level view of how much work your virtual agent is responsible for accomplishing on your behalf. The most important thing about calculating the resolution rate is deciding what should be in the numerator and the denominator. 

So how should you decide? One method is to divide the number of issues that your virtual agent solves end-to-end by the number of issues that employees directly submit to it. But while this method provides a sense of the virtual agent's quality, it fails to capture the impact it's making with respect to all support issues — including those submitted through other channels like email.

To get a broader picture of your virtual agent's success, you could use every employee-submitted support issue as the denominator for resolution rate. This method raises other important considerations, such as employee adoption, ticket interception, and proactive capabilities.

Neither calculation is wrong. The former is a narrower metric that focuses on the efficacy of your virtual agent, while the latter measures its total impact on your support team’s workload. Before deciding how to calculate the resolution rate, you need to know what you’re trying to achieve with your support organization.

With an in-depth understanding of the resolution rate, companies can incrementally improve the impact of their virtual agent. This means identifying the right limiting factors to address, be it adoption, deeper integrations, or resource creation. Real-time insight into the questions employees ask and the problems they have makes it easier to know which levers to pull.

Note that there is an additional metric — deflection rate — that's often confused with resolution rate. Deflection rate is defined as the rate that automated and self-help resources satisfy service demand that would otherwise be handled by assisted service staff. It’s a suitable metric if your goal is to reduce employee calls to the support hotline and instead rely on digital channels. However, it should not be confused with the resolution rate, which focuses instead on the percentage of employee support issues resolved end-to-end, without input from the support team.

Time to value (TTV)

Your virtual agent should make a rapid impact.

Traditional software solutions focus on go-live dates. But with virtual agents, you can’t just think about how long the implementation time will be. You have to think about how much value the bot will deliver on day one. You have to think about how long it will take for your company to get value from your investment. 

Value is measured by the benefit you — the customer — expect to receive, and time to value (TTV) is how long it takes to get to that benefit. To get a handle on TTV, you need to deeply understand your business objectives and your virtual agent’s role in getting you to the next benchmark. A meaningful result could be increasing your resolution rate, decreasing the amount of time it takes to incorporate a new use case, or keeping the support team on budget. The important thing is to pick a metric and measure how long it takes to get to a predefined milestone.

Implementing an AI solution is complicated. It can take months, years, or never get off the ground at all. That’s why it’s best to consult with vendors about their implementation process and, when possible, consider how long it will take for your system to deliver meaningful results for your employees, your support teams, and your company as a whole.

There are two main approaches to building and maintaining a virtual agent. There are toolkit solutions that require a dedicated team to build machine learning models, dialog flows, and interception methods — not to mention the additional burden of ongoing maintenance and refining scripts as new use cases and resources are created. And second, there are ready-made solutions that work right out of the box, using AI to learn your environment as it changes.

But remember — implementation time shouldn’t be your only consideration. At Moveworks, we prefer to measure time to value because it’s so hard to get right. That said, the shorter the TTV, the sooner you’ll be able to spend time on more pressing projects. Some of the key values or benefits provided by a virtual agent include the following, but this list is by no means inclusive and will depend on your objectives: 

• Increase in employee engagement
• Increase resolution rate
• Decrease operational expenses
• Improve the employee experience
• Increase the time the support team spends on transformation projects

loanDepot’s virtual agent delivered value on day one

In February 2022, loanDepot officially launched its Moveworks virtual agent — Elle-Dee — on Microsoft Teams. While evaluating solutions, Moveworks provided the loanDepot IT team with a comprehensive AI Readiness Assessment, which included insights into both its knowledge base and its employees’ most common IT issues. The team realized that Elle-Dee could do more than resolve tickets autonomously; it could also identify the gaps in existing knowledge articles and spot the inefficiencies with our workflows, opening a window into where to focus next.

Incremental improvements in knowledge creation and identity management increased loanDepot’s support efficiency, resulting in a rapid increase in IT efficiency. Elle-Dee was up and running in just weeks. Since then, the company has witnessed a gradual but profound transformation, resolving more than 2,000 issues and counting with AI every month. AI allows its team to approach dynamic support challenges in an agile way.

Time to resolution (TTR)

Your virtual agent should accelerate support, so employees never have to wait for help.

Time to resolution (TTR) is a support metric that measures the average elapsed time from when an issue is reported until it’s resolved. Many companies measure mean time to resolution (MTTR) instead, but this metric only tells half the story. MTTR measures how long it takes the support team to solve a problem in business hours. For example, an employee might be locked out of their account from 3:00 p.m. Friday until 9:00 a.m. the following Monday. MTTR would say the issue was resolved in two hours. But that’s almost three days for the employee. And that’s a lot of time to be locked out of an account, especially if you’re trying to close a deal or launch a new project.

TTR has a significant impact on employees’ overall satisfaction with support. This makes sense, as users may be forced to stop working entirely or use workarounds until their issues have been resolved. Especially with employees working remotely, some issues — like not properly connecting to the VPN or being locked out of an account — can completely derail productivity.

When it comes to evaluating your virtual agent, there are two ways to look at TTR. First, some issues — like unlocking an account or troubleshooting a VPN problem — can be solved end-to-end instantly. You should expect your TTR to drop from hours, or even days, to a couple of seconds for these issues.

But some issues cannot be immediately resolved. Think about accessing a certain software that requires approval from a manager or contacting the HR team about a highly personal situation. For these problems, a virtual agent should decrease TTR by accelerating issues through the approval process, rerouting requests to the correct assignment group, and allowing employees visibility into the status of their issue.

Equinix accelerates support with its virtual agent

Equinix used a virtual agent to decrease its TTR by almost a third. Based in the Philippines, Equinix’s help desk reads every ticket to decide which subject-matter experts were best suited for the job. Given that its international workforce is concentrated in the US and UK, Equinix wanted to avoid having IT issues pile up in a distant time zone. 

The Moveworks Triage Skill allows Equinix's virtual agent to assign high-touch issues to the correct subject-matter experts — with the same 96% accuracy rate achieved by help desk agents. Simply by getting the issues into the right hands, Equinix has transformed the support experience for its employees and support team.

Resource utilization

Your virtual agent should help employees find the resources they need to help themselves.

The effectiveness of your virtual agent depends on the coverage and accessibility of your resources. That’s why one of the most important metrics to think about is how employees are using — or not using — the knowledge base articles and forms you create. 

At a high level, resource utilization measures the percentage of issues your virtual agent solves with existing resources in your service catalog. If resources are easy to find and engage with, your virtual agent will be more likely to perform well. 

That said, the best virtual agents play an important role by offering visibility into how employees use the knowledge bases and forms at their disposal. Some solutions will show which resources are missing and automatically incorporate new ones as they are created. The greater the number of high-quality resources, the greater the number of employees’ questions answered in seconds.

We work directly with our customers to improve their resource utilization and drastically increase their resolution rate, satisfaction score, and other metrics by asking three questions:

1. Are your resources easy to find with the virtual agent?
2. Are your resources easy to engage with on the virtual agent?
3. Does your virtual agent reveal information gaps?

The best virtual agents expose inefficiencies, offering a comprehensive performance analysis of individual skills and shedding light on the most requested information. With these insights, support teams can see room for growth and adjust accordingly to boost their virtual agent’s resolution rate simply by improving the depth of their support catalog.

Broadcom uses AI to build a better knowledge base

Broadcom — a world-leading tech infrastructure company — dramatically increased the usage of its forms and knowledge base articles with its virtual agent. By making incremental improvements to its service catalog, Broadcom’s IT team systematically improved its knowledge base — filling in gaps, updating old articles, and retiring old information — so employees can self-service their own support. Today, their Moveworks bot, E-bot, automatically resolves 57% of all IT issues by making it quick and easy to find the exact snippet of information an employee needs to solve their problem.

Bot preference

Your virtual agent should be the first point of contact for all employees in search of help.

It doesn’t matter how many issues your virtual agent resolves autonomously or how quickly it helps employees if no one uses it. Employees might prefer asking questions at the walk-up help desk, sending an email, or filing a ticket through a portal. If you want everyone to go straight to the virtual agent when they need help, you’ll need a strategy informed by employees’ bot preference. 

Bot preference measures what percentage of employees use the virtual agent as their primary way to get support. This metric asks: Out of the total users within a company, how many go to the bot first when they have a problem? With greater insight into which channels employees prefer, you’ll be able to push people towards channels that best serve their needs and the needs of the IT team.

Since a virtual agent is designed to be more cost-effective and less time-consuming for employees and support teams — you want it to be a one-stop shop for support. But we all know changing habits is hard. Three factors determine whether or not employees go to their virtual agent first:

1. Awareness: do employees know about their virtual agent?
2. Accessibility: is the virtual agent easy to find and use? 
3. Performance: does the virtual agent offer fast and accurate help?

By focusing on each of these factors, your team can maximize all the metrics mentioned in this blog. For example, many virtual agents require the user to do some initial setup — an initial login — before using the bot. But with a virtual agent living in an enterprise chat platform, employees have somewhere to go for help 24/7. And if the bot instantly solves their problem, using it will become second nature. With fewer and fewer employees constantly messaging the support team with basic issues, the support team will be able to work on more complex and fulfilling projects.

Building one place for help at Vituity

For Vituity — an organization dedicated to transforming healthcare — a significant barrier to supporting medical professionals is that they don’t know where to go for help. The answer to their questions might live on the company website, FAQ, or IT portal. And each time they need help, they have to go somewhere else to find it. Meanwhile, support teams are flooded by requests that steal bandwidth from longer-term digital transformation projects.

Vituity realized that — to create a lasting employee habit — the bot would have to intercept issues no matter what channel employees used. Known internally as Otto, their Moveworks bot is now the single place to go for help. Whether users need to reset their passwords, access software, or ask HR about a policy, they can always ask Otto, reinforcing the habit of using it as the go-to place for support. Otto is constantly improving through machine learning — understanding company lingo, automatically syncing with backend systems and adapting to user feedback.

Issue prevention

Your virtual agent should engage proactively with employees to prevent issues.

Most virtual agents today are reactive, offering a place where employees can ask for help when they need it. But increasingly, leading virtual agents are helping support teams to be more proactive, alerting employees to system outages and policy changes. 

Issue prevention tracks the percentage of issues your virtual agent can prevent by sending proactive messages and targeted communications. A major goal of any virtual agent is to free up support agents to work on projects that require their expertise. Decreasing the average number of support issues an employee has over time can make a real difference in how support teams operate.

Many support issues are common. And support teams don’t want to spend half their day answering the same questions about when the healthcare enrollment period ends or unlocking a dozen or so accounts when employees forget to update their password every six months.  

Traditionally, most proactive communication takes the form of mass emails. But the more messages you send, the more you’re ignored. With a virtual agent — it’s just the opposite. Since every message is meaningful, every one reinforces the idea that the bot is the place to go for consistently high-quality information. In fact, by sending targeted campaigns, companies that use the Moveworks bot have seen engagement rates as high as 72%. 

A virtual agent can help manage change and get ahead of issues before they get out of control. A virtual agent can save HR, IT, Facilities, and Finance Teams days of extra work by sending quick, targeted reminders or intercepting issues in public support channels.

Verisk stops problems before they happen with proactive communication

Beyond just resolving IT issues with their Moveworks bot — Vic — Verisk is now predicting and preventing problems ahead of time. When an employee’s password is set to expire, Vic reminds him beforehand directly on Microsoft Teams to avoid getting locked out later with Moveworks for Employee Communications. 

Nowhere has proactive support made a bigger impact than in resolving account issues. Before Moveworks, the help desk would spend countless hours resetting passwords and addressing lockouts. But today, Vic addresses thousands of account issues proactively and autonomously without involving the help desk.

Satisfaction score (CSAT)

Ultimately, the best measurement of your virtual agent is whether employees are satisfied.

The satisfaction score (CSAT), typically measured on a 1-5 scale from “poor” to “excellent,” is the most essential service desk performance metric to gauge how your customers perceive your approach to support. You may think that you’re getting everything right, your systems are efficient, your agents are productive, and your knowledge base is comprehensive, but employees may disagree.

In fact, most people don't like chatbots. So it’s not enough to just think about resolution rate, TTV, resource utilization, or TTR if no one wants to use your virtual agent. You need to think critically about what makes a virtual agent a good experience. What is a good experience? Well, it will depend on employees, their preferences, and their pain points. That said, we’ve seen some overarching themes. Employees want a bot that:

• Solves as many problems as possible
• Isn’t overconfident
• Sends issues to the right expert, and 
• Doesn’t waste anyone’s time

A virtual agent that ticks each of these boxes is the key to happy employees and happy support teams.

In this blog, we've given you a lot of hard numbers to define your virtual agent’s performance, but what matters is do employees like using it. If they don't, your virtual agent will never get off the ground. 

And that’s why CSAT is so important. It drills down into the likelihood of your employees using and recommending the bot to others. The occasional survey asking questions about what’s going well and what’s not can reveal hidden opportunities and unlock the door to an even better solution.

Unity’s employees love their virtual agent

At Unity, 91% of employees are satisfied with NinjaUnicorn — their Moveworks bot. This means that of all employees surveyed, the bot received a majority of 4 or 5 stars out of 5. Getting to 91% satisfaction did involve effort. Support teams worked hand-in-hand with the Moveworks Customer Success Team to ensure that they could constantly improve their customer experience. By regularly surveying its employees, Unity knows how important fast resolutions times are and what knowledge bases could use some upkeep.

Metrics drive performance

Monitoring your virtual agent’s performance is the key to creating a seamless employee experience. This list of seven metrics is not exhaustive by any means, but it’s a great place to start.

Depending on your employee experience goals, you can add other metrics to this list or just focus on optimizing one or two for now. Maxing out these stats will transform your organization, from saving time and money to redefining the entire support experience.

Request a demo to see the Moveworks platform in action.