Pre-COVID Challenges
Our client operates an AI-based customer engagement platform, the core of which comprises two elements – the AI-powered product and AI-powered customer engagement assistants. Prior to our involvement, neither of these elements underwent any form of quality assurance. Given their collaboration with highly regulated industries such as healthcare and insurance, the need for quality control was paramount, and the potential cost of failure was substantial.
The Solution
Mantis EVN undertook the task of integrating quality into every facet of the client’s operations, with the customer’s satisfaction as our foremost concern. A dedicated QA engineer was placed on-site with the client to ensure that quality assurance became an integral part of their business, from planning to release. Test plans and suites were implemented, and UAT tests were run for each AI assistant, significantly increasing the accuracy of AI assistant responses.
Post-COVID Challenge
Following the COVID-19 pandemic, our client began to scale dramatically. The number of AI assistants tripled, presenting challenges in maintaining all the assistants, updating them regularly, and ensuring quality wasn’t compromised.
Further Solutions
Mantis EVN quickly responded to these changes by onboarding two additional QA engineers who had been pre-trained on the product. This approach meant no time or financial resources were spent on a training period, and the contributions of the newly onboarded QA engineers were significant from day one. They managed not only to tackle the increased volume of QA work, but also introduced AI automation solutions to accelerate the delivery of each release.
Results
After three years of collaboration (and counting), we are proud to announce our key results as of June 2023:
- A fully integrated team of three QA engineers working full-time for the client.
- Zero time or resources spent by the client on hiring or training the QA team.
- Integrated AI automation solutions, reducing the time required for regression testing prior to release from ~14h to ~6h
- Boosted the QA process covering the entire spectrum of AI-based conversation model development
- Test Plans covering the whole functionality of VAs from both dialog design and NLP perspectives for more than 30 projects
- VA recognition rate of 98.8% and NLP recognition score of 0.94
Conclusion
This case study exemplifies the resilience and adaptability of Mantis EVN’s QA outstaffing solutions in the face of unprecedented challenges. By integrating quality assurance into every aspect of the client’s operations, we enhanced the reliability and accuracy of their AI-powered customer engagement platform. Our swift response to their post-COVID expansion demonstrates our commitment to meeting the evolving needs of our clients.
Through our collaborative efforts, the client could focus on scaling their business, secure in the knowledge that Mantis EVN was safeguarding their product’s quality. This not only saved significant time and resources on training and hiring but also ensured the seamless delivery of high-quality customer engagement.
