Feeling pressure to launch an AI chatbot soon?
Maybe your company’s accounting app just took off. Or maybe you’re an ecom brand ramping up for the next holiday season.
Whatever the situation, if your customer base and Zendesk ticket volume is growing quickly, launching an AI chatbot can feel like the magic key that will help you survive.
And while yes, chatbots bring a lot of benefits, you also know that launching a bot comes with the risk of providing wrong answers and frustrating customers, which can be more damaging to your brand than needing to wait for a few extra hours or days for an email reply.
This leads you to wonder, “How do I know when my chatbot is ready to go live?”
The aim of rigorous Zendesk AI chatbot testing is simple: you need to guarantee the bot is accurate, helpful, and feels on-brand. It’s about turning a complex piece of tech into a reliable member of your team.
This article is your straightforward guide to setting up a safe, step-by-step chatbot testing plan so you can confidently move your bot into production.
Note: we’re focusing on testing a new Zendesk chatbot here, but if you already have a chatbot live, you may want to check out these tips for optimizing your Zendesk chatbot.
Key Concerns to Remember When Testing Your Chatbot
There are two big areas where chatbots often go wrong:
- AI hallucinations
- Human biases
Before you start testing, it’s critical to understand how these both impact chatbot performance.
Mitigating AI hallucinations
Did you know that every pomegranate has the same number of seeds? And that blue light two hours before bedtime reduces melatonin production, making it harder to sleep?
False.
AI chatbots have a notorious problem of generating hallucinations: answers that are entirely believable, while being completely inaccurate.
These two examples—the pomegranate and blue light—are perfect “meme-versions” of this phenomenon. Widely known “factual” information has been seen and regurgitated on the internet so often that it’s just “known to be true” and sounds incredibly believable…
But later turns out to be either completely wrong (pomegranates actually have a wide range of seeds) or too nuanced to boil down to one short sentence blue light might not actually affect sleep-wake cycles as much as we first thought).
A Zendesk AI bot is trained on your own company data. But if your bot learns from old, messy, biased, or incomplete knowledge, it’s going to repeat those mistakes and potentially harm your customer experience.
To fight this issue, getting your knowledge base in shape is crucial before testing your chatbot.
Help Center Analytics is designed to help you quickly analyze the performance of your knowledge base.
And Help Center Manager makes managing large Zendesk knowledge bases easy, both helping you ensure your chatbot is learning from correct information.
Accounting for Human Bias
Your support team knows your product so well they can answer FAQs in their sleep (quite literally — I cannot count how many times I’ve dreamt about helping customers to login).
Because of this intimate knowledge with your system, your staff know exactly how to word questions in a way that will ensure your bot provides the answer they are looking for.
They also know how to interpret the answers that are given, even though they might not make sense to a third-party. They don’t mean to do this, but that’s the thing with bias, it’s typically subconscious.
It’s a bit like writing a knowledge base article. You need to have someone else edit your work, because you already know what you’re trying to say, so you might miss nonsense statements that a second set of eyes will quickly catch and correct.
To account for this, don’t have your support, sales, or QA team test your bot. Instead, pull in members of your marketing team, finance team, or even the family members of employees (if your company policy allows).
While there’s nothing wrong with having support agents test your bot too, only relying on subject matter experts in chatbot testing will introduce accidental bias in the results.
Core AI Chatbot Testing Methods and Goals
Effective chatbot testing requires more than just asking the bot random questions; it demands a segmented approach based on where and how the chatbot talks with customers.
Testing conversational flows
Testing conversational flows is arguably the most important step for customer-facing AI agents.
This is where you validate the end-to-end resolution of problems, ensure the tone is aligned with your brand, and ensure it can smoothly hand things off to a human when it gets stuck.
When developing Zendesk AI chatbot testing scenarios you’ll want to focus on three specifics:
- Use a variety of phrasing. Ask the same question in a multitude of different ways, such as “Where is my order?” vs “I haven’t gotten my shipment yet.” vs “item not delivered” vs “why isn’t my stuff here” (Yes, I’m serious on that last one. I’ve seen it.)
- Test the full chat dialogue. You need to test the entire conversational journey — the successful outcomes, the screw-ups, and points that should result in a hand off to a human. If the bot can’t help, does it correctly own its inability to aid the customer and hand it off to a human agent?
- Check for tone and brand alignment. Check if the language and personality align with your brand standards. This is deeper than just polite greetings. Brand tone is about the emotional vocabulary used by the bot. Is your brand professional and corporate, with a lot of legal speak? Then your bot should be too. On the other hand, if your brand uses loads of exclamation points and hyper friendly banter, then the bot should reflect that.
Zendesk AI Agents include a number of customization options, so you want to maximize those to get your bot as on-brand as possible.
For detailed steps on testing flows within the Zendesk environment, see Zendesk’s guide on Testing conversation flows in advanced AI agents.
Creating and segmenting testing goals by use case
It’s important to remember that all AI agents aren’t built for the same job.
A pre-sale bot is meant for capturing leads and deflecting those who don’t fit your product niche, while a support agent should be focused on things that keep paying customers happy.
This means you need to create and segment goals based on different use cases and test them accordingly.
Plot out all the use cases for your Zendesk chatbot to make sure your testing covers every base.
Zendesk AI Chatbot Testing Checklist
Before moving an AI agent to the pilot stage, every item on this Zendesk AI chatbot testing checklist should be signed off. Following these guidelines will dramatically reduce the risk of negative impacts from your chatbot launch:
Rate of recognized intent has an accuracy of >90%
Rate of first contact resolution is >60% without human transfer (this can vary based on industry)
Correctly transfers to a human when confused
Maintains brand voice across conversations
Answers are concise and easy to understand for your target audience
An internal test group gives high CSAT scores
Response time is under 2 seconds
All integrated systems are working
Bot handles sensitive data according to privacy policies and applicable laws
For a deeper dive into production readiness criteria, it’s always a good idea to check out other relevant industry resources like Is my chatbot ready for production?.
The 6-Step AI Chatbot Testing Readiness Procedure
Are you ready to move your bot out into the real world? Not so fast! A safe launch is one that is incremental.
This will help you effectively manage risks and gather real-world data under controlled conditions, so you can stop guessing and feel confident in knowing.
Step 1: Implement the chatbot in a staging environment
This should really go without saying, but unfortunately it doesn’t: never test in a live production environment. Mistakes happen. It’s a natural part of every process.
Testing in a dedicated sandbox environment allows those mistakes to freely happen without affecting real customers or real customer data.
Before you start any serious testing, you need to push your bot in a sandbox or staging environment of your website or help center.
This dedicated sandbox lets you test every integration, every trigger, and every response using live copies of your data without the risk of public exposure.
Step 2: Connect your CRM and other backend systems
It’s tempting to test your bot without bothering to hook up your real systems, but don’t do it. The AI agent must be tested with real data.
Why? Because real data connections allow you to test personalized, actionable interactions. A bot that correctly understands the query but fails to pull up the customer’s real order status is a failed bot.
Step 3: Do internal testing with real life scenarios
Next, you need to do a round of internal testing. I recommend copy/pasting verbatim questions from actual previous tickets for a lot of these test runs.
This exposes the AI to the complex language and jargon your customers actually use, rather than idealized perfect questions.
Use segmented, unbiased groups as we discussed in the above section Accounting for human bias and arm your testers with a list of Do’s and Don’ts.
Zendesk’s Golden Rules for testing AI Agents
Step 4: The customer-facing pilot phase
After you’ve done some internal testing and worked out your first round of kinks, you’re ready for a customer-facing pilot phase.
Since this is the first time your bot is exposed to real customers, you’ll want to keep this small and tightly controlled. This phase is about seeing the bot working in real life under close observation, so high volume is not the goal.
To manage the risk, limit the pilot to a three or four day window over a relatively low traffic period.
You could even consider only showing the bot to a specific segment of users, such as people located in one specific time zone, just to help further narrow things down.
Ensuring the group is small is crucial to ensure that there is minimal customer frustration and the ability to quickly roll back in the event that things go wrong.
While the pilot is running, implement hourly monitoring, with a dedicated team member doing some manual QA on all of the interactions the bot has with humans.
This person needs to be ready to jump in and take over a conversation immediately if the bot fails, as well as flag all questionable cases for post-pilot analysis and tuning.
Step 5: Post-pilot analysis, tuning, and retraining
“Have you tried turning it off and back on again?”
All jokes aside, turn off your bot after the pilot is complete! Don’t just leave it running while you conduct your review.
Do a full, detailed analysis of your performance data. Review all conversations, paying special attention to those that were transferred to a human agent, those that were abandoned, or anything that resulted in negative feedback.
Those insights will help you pinpoint issues so you can correct training data, update chatbot intentions, and retrain the model before proceeding to the next stage.
Depending on the outcomes, It’s a good idea to repeat steps 4 and 5 until you feel really comfortable with the way your bot is handling things.
Step 6: Incremental and full rollout strategy
Don’t go from a four day pilot to a full launch.
Start small (e.g., 10% of users, or users on a specific legacy channel) and roll out incrementally, comparing metrics between the legacy solution and the new chatbot. Use the baseline performance of your human agents to set the bar.
When fully live, assign a few team members to act as QA for a few months to ensure continued success.
Top Chatbot QA Troubleshooting Tips
It’s crucial to acknowledge that you shouldn’t expect perfection every time from an AI agent. Instead, you must have systems in place to catch failures quickly and consistently.
The human safety net
The best fail-safe is a rapid escalation process.
Empower your team to handle failures, specifically the mechanism of reopening tickets automatically if a customer abandons the chat following an interaction with the bot.
This allows a human agent to follow up proactively, apologize for the failed automation, and resolve the issue, turning a poor experience into a customer save.
Continuous evaluation
Your testing doesn’t stop after launch. Any AI model can degrade over time as new intents and products emerge.
Since your Zendesk AI Agent is connected to Zendesk Support and your Zendesk knowledge base, it should continually learn from how your agents are handling customer interactions.
That being said, it’s also a good idea to implement a mechanism for continuous evaluation, such as weekly “spot checks” on new topics to ensure the AI has adapted.
Some companies even treat their chatbot just like they would a human agent, including conducting regular QA reviews of the chatbot’s conversations
From Sandbox to Success: Chatbot Testing Makes It Possible
Rigorous Zendesk AI chatbot testing is the non-negotiable step between a promising AI agent and a successful customer service automation strategy.
By following this incremental procedure—from sandbox testing of flows and intents, through a small, monitored pilot, and into an incremental rollout—you gain the confidence needed to trust the agent with your customers.
A well-tested bot doesn’t just solve problems or deflect tickets; it makes life better for your customers and frees up your team to work on the most important issues.
If you’d like to learn more about how Swifteq’s Zendesk apps can help you get more out of your Zendesk instance, click here to book a demo today!
Written by Anne-Marie Traas
Anne-Marie is a Fractional Head of Customer Success focused on providing an optimal customer experience in every interaction. She specializes in driving process and product improvements, creating thorough and easy-to-understand product documentation, and teaching others how to communicate more effectively through the written word.
