Back in school during statistics class, I remember thinking, “When will I ever use this stuff?” Population size, variance, and margin of error—all critical components of establishing a statistically valid sample size—seemed like distant concepts at the time. That convoluted formula we had to memorize felt more like a test of endurance than a practical tool.
Yet, here I am in the world of collections, realizing the importance of those statistical principles. Reviewing calls and accounts managed by a third-party vendor demands a solid grasp of sample size challenges. I am constantly considering how many individuals we need for a dependable sample and weighing the costs of acquiring the necessary tools.
As I contemplate how to maximize our oversight resources, I find myself exploring strategies for optimizing sample size. Here are some insights I have gathered on this crucial aspect of our work.
Sample Size ExampleÂ
When determining how many calls you need to review for call monitoring, the first question to address is the purpose of the monitoring: compliance, collection effectiveness, or both. Based on your purpose, different approaches can be applied.Â
For a population of 2000 calls, a common rule of thumb is to use a sample size that is at least 323 calls reviewed, assuming a 95% confidence level and a 5% margin of error. How do you review 323 calls?Â
- If you are only reviewing calls for compliance, for example the mini-Miranda or call recording disclosure, listening to the first 30 seconds of a call can be completed quickly (assuming no one joins the call after 30 seconds and needs to be advised the call is being recorded). If you need 30 seconds a call to review 323 calls, that will take you 9690 seconds to complete. This translates into 162 minutes or 2.7 hours. This can be completed by one full-time employee (FTE) in less than a day without interruptions or multi-tasking. Â
- If you are reviewing calls for collection effectiveness, including compliance, the time requirement is much higher. Let us use an average call handle time of 200 seconds per call. If you want to review 323 calls, that will take you 64,600 seconds to complete (assuming you do not have to listen to parts of calls multiple times). This translates into 1077 minutes or 18 hours. This would require over two days of work for one FTE. Using technology for collection effectiveness may not be as effective, as voice analytics might miss nuances that a trained human reviewer would catch.Â
Alternatively, technology can review all 2,000 calls for compliance, eliminating the need for FTEs, though you will need to consider the cost of the technology. Using technology to evaluate calls for collection effectiveness is not necessarily effective. As I stated in my Transforming Customer and Collector Interactions With Call Monitoring Blog:
“While voice analytics is invaluable for compliance-related aspects and identifying key triggers, I firmly believe that a trained eye is essential to    comprehensively understand the entire interaction, from start to finish. Teams often attempt to use voice analytics to score the overall call, but it may not capture the nuances of how effectively the collector navigated the customer’s financial situation.”Â
What happens if you want to review calls monthly for each collector that makes your calls. In this example, five collectors made 400 calls each. The required statistically valid sample size for each collector would be 197 calls. Why is the call sample so high for each collector?  Â
- Small Populations: For smaller populations, each additional sample significantly improves the estimate, hence a relatively larger sample size is needed.Â
- Large Populations: For larger populations, the improvement in estimate accuracy with each additional sample diminishes, leading to a slower increase in required sample size.Â
There is a considerable time element in using a statistically valid sample size when reviewing calls. Your overall call review strategy needs to include an evaluation of what is practical, effective, and what your organization is willing to spend for this type of oversight.Â
Sample Size ChallengesÂ
What are some real-world variables that need to be included in the decision of how many calls needed to be reviewed to have a good sample.Â
- Limited Time: There might not be enough time to thoroughly review all calls, so only a portion can be sampled within the available timeframe. When call monitoring is just one of many job responsibilities, it often becomes a chore and may not be completed effectively. Managers or compliance individuals frequently scramble to review calls, leading to rushed and incomplete monitoring. This often results in selecting shorter calls that do not accurately reflect typical interactions. In addition, having monitored thousands of calls, I have learned that it is necessary to listen to portions of calls multiple times to fully understand what happened. Limited time compromises the accuracy of call reviews.Â
- Restricted Access to Data: Limited access to call recordings or account details can make it difficult to select a comprehensive sample. Many organizations struggle to identify calls made by specific teams or clients. Searching a call recording database for team-specific calls adds unnecessary time to the review process. Ideally, you should have direct access to the call recordings related to your accounts, enabling you to review them at any time.  Without account details, you can only listen to the audio and cannot verify the accuracy of the information being discussed. Reviewing calls is crucial to ensure that accounts are handled accurately based on the call content. Therefore, I recommend ensuring that you have access to your accounts on your third party’s system of record.Â
- Technological Limitations: Technical issues or limitations with call monitoring systems can hinder the effective capture and review of calls. If calls are not recorded, obtaining a good sample of collector interactions is nearly impossible. Side-by-side call monitoring requires a considerable time investment to review enough calls, as a collector typically connects with a person only a few times an hour.   In some cases, collectors must manually activate call recording for each call. If your vendor uses this process, expect that not all calls will be recorded. Additionally, call escalations transferred to a manager often result in incomplete recordings. I have seen situations where managers use company-owned cell phones to ensure availability, but this often means the call recording ends at the point of transfer, and the portion handled on the cell phone is not recorded. This situation also can happen if calls come through a main receptionist and then transferred to a collector.Â
- Staffing Constraints: Limited workforce size or expertise in call monitoring can restrict the ability to analyze a large volume of calls, resulting in a smaller sample size. A dedicated team responsible for call reviews is considered best practice. This team typically handles reviewing, scoring, and reporting call review results. Often, they also use technology to evaluate all calls for specific words, phrases, and tone, which aids in additional reporting and coaching. Combining call listening with technological analysis is highly effective in assessing collector and department performance. This approach requires a properly staffed team with the right skills to execute the process effectively.  Â
What is the Right Answer?Â
The right answer is the one that aligns best with your organization’s budget and employee skill set. Any call reviews are better than none. Often, if errors are found in a small sample, it indicates a larger problem.Â
I have found that a sample of ten calls can effectively gauge how your third-party collection team is handling your accounts. If you use a ten-call sample, I suggest reviewing calls at least every other month. Additionally, require your third party to share call monitoring results with the collectors. Providing feedback to collectors should lead to improvements in future call monitoring sessions. Why do I think ten calls is a reasonable sample size? Let me explain.Â
To start, pick your sample size, allocate the necessary resources, and review the calls thoroughly. For example, if your budget allows for reviewing ten calls per collector instead of 197, focus on those ten calls in detail. Each call contains numerous variables for compliance and effectiveness. Reviewing ten calls with twenty variables each means you are actually assessing 200 variables, providing a more comprehensive view of the collector’s performance. In addition, make sure to review the account details for each call. This allows you to verify how the collection team worked the account based on the call. For example, if a call included a request for a debt verification, did the request get processed accurately. Not only can you verify the call handled, but you can verify account maintenance and work activity.  Â
If issues are found in the initial sample, you can increase the sample size for that specific collector. Depending on the identified issues, you can target additional samples or use technology to conduct a deeper analysis of the collector’s calls.Â
SummaryÂ
Effective call monitoring in collections is vital for ensuring compliance and optimizing collection effectiveness.Â
Whether a statistically valid sample size, or a smaller sized sample – both provide actionable intelligence for your collection oversight.Â
Balancing the purpose of monitoring, available time, access to data, and staffing constraints is crucial. Utilizing both manual reviews and technological tools can provide comprehensive insights while maximizing efficiency. Starting with a manageable sample size and adjusting based on findings ensures continuous improvement and effective oversight.Â
Ultimately, the right strategy aligns with your organization’s budget and skill set, providing a robust framework for accurate and reliable call monitoring. By prioritizing detailed and consistent reviews, organizations can enhance performance, identify issues early, and maintain high standards of compliance and effectiveness. Â
Regardless of sample size, implementing a call monitoring program is an important aspect of collections oversight.Â
Author:Â Â Ken Evancic
Ken.Evancic@ResourceManagement.com
As a consultant for Resource Management Services, Ken provides consulting, training and mentoring in all phases of collection and recovery, in addition to auditing third party vendors.
Ken Evancic is a Vice President at Resource Management Services, Inc.  Ken Evancic is a collections veteran with over 25 years experience. He has managed all phases of collection, including all levels of delinquency, automated dialer units, early out agency management, recovery, and skip tracing. In addition to collections operations management, he has lead initiatives in the areas of performance management, collections strategy development, collector and manager training, collector desktop design, collections reporting systems, and risk and compliance.
As a consultant for Resource Management Services, Inc., Ken has specialized in developing and completing third party compliance and performance audits for collections agencies and collection attorney firms for many top credit grantors and debt buyers. He has leveraged his 25 years of experience to develop multiple collector and collection management training classes designed to maximize collector performance. In addition to collection training, Ken helped develop and facilitates the RMS Third Party Vendor Auditing training. Â
Sign Up for the Twice Monthly Newsletter
Just enter your email address at the top orange bar at:
Collection Compliance Experts – “The Power of Expertise: Oversight Perfected”
It’s that easy! Twice a month – we provide blog updates and Resources for the Collection and Industry Professional.Â
Your email is just for this newsletter. We never sell your information. No fee. Opt-out at any time.
