Vendor Management: Audit Sampling and Intuitive Judgment 

Audit Sampling  

At first blush, it seems that pulling an objective and statistically valid sample is the key to success.  However, from experience, I’ve learned that there are many factors that affect the success of the audit. Developing a solid audit plan is the first step. Understand the goals for the audit, and then develop a work plan to reach those goals.  Part of the sampling question depends on your audit focus.   

Many years ago, a boss was vague when asking the vendor manager to conduct an audit of their agency.  After reviewing accounts for work performance, and checking for compliance with regulations, the vendor manager reported findings that included how well the vendor worked the accounts, timeliness of activation and current activity on the active accounts at the agency. Although the report was comprehensive and covered many areas – the boss then asked how many attempts were made prior to payments, settlement (if applicable) or closure.  So, the boss had some specific questions in mind – but didn’t share with the vendor manager conducting the review.  His concern was whether they had exhausted all efforts prior to closure, and the vendor manager was concentrating on understanding how quickly they initiated collection efforts.  Few accounts in his sample were even closed. 

The sampling methodology can greatly affect the results.  Once you understand the population you want to sample, there are a few next steps.  For example, if you wanted to better understand the accounts with settlements, you’d want to be sure to include a random sample of accounts that were identified as settled, and some accounts that were not identified as settlement accounts, but that have paid something on the account.   If you were to just do a complete random sample of accounts at the agency, you might not have enough accounts in the settlement sample to validate your findings. 

Don’t let “random” selection criteria deter you from using your judgment.  I remember one auditor that saw an unusual proportion of accounts with $10 payments.  This auditor delved deeper into that discovery, with a judgment selection sampling, rather than continuing only with the “random” accounts.  As it turned out, the accounts with the $10 payments were all within 10 days of a six-month required closure (with a contract that said accounts must be returned at 6 months if no payment.)  You can imagine the challenges this might have created. 

So, use random and statistics when you can, but trust your gut instincts to review other things or activities.  Rather than a pure statistical look at random sampling, the auditor needs to be sure there is an adequate amount of the high-risk areas in the sample, and that the sample provides the breadth and depth of coverage that they need to make the informed decisions about the overall compliance and effectiveness of the vendor.  Accounts in the sample can be rated for their overall compliance and work effort, or certain accounts can be sampled and rated for their activity around key areas.  For example, random selection is great for compliance tests, and a stratified sample, or a specific sample of paying accounts, would be good for a remittance check.  A simple random selection might be achieved with a computer program, or random number tables. Or some use systematic, or interval sampling, like choosing every tenth account.  Although both simple random and systematic sampling are generally free from bias, there is a potential hidden pattern in the systematic sample, for example if every tenth account also had been picked for additional work effort or a special campaign.     

Stratified sampling provides for a selection of sample items by breaking the population down into strata, or subgroups. A specific review of paying accounts or skip accounts might be good examples of strata to be included in a review.  

Block sampling is often used when evaluating remittance activity for example.  One might review all the remittances for a certain block of time, or month in this case.  Block selection should be used with caution, as valid references cannot be made beyond the period examined.  With block sampling, many blocks should be selected to minimize sampling risk.  

Personally, I like a combination of audit sampling techniques within the scope of an audit.  And, I think a judgment selection can be an important part of any audit process.  With judgment selection sampling, as illustrated above, there is a reliance on the expert judgment of the auditor. This method can be quicker and more cost effective than other sampling methods because it targets specific strata.  It also allows the auditor to adapt the sampling strategy as they learn more about the accounts.  It can also lead to in-depth insights that might not be captured through random sampling methods.  Of course, the downside to a judgmental audit sampling would be potential unconscious bias, so it’s important to to understand when the expert insight is more important than the random sampling.  It seems that by mixing random with judgmental when appropriate, we can have the best of both worlds.