You're Scoring the Questionnaire, Not the Vendor

Category: Operational Resilience Tags: third-party risk, TPRM, vendor onboarding, inherent risk, due diligence, vendor management

I once watched a vendor risk team celebrate a 94. It was the intake score for a new data processor, and 94 out of 100 felt like diligence done well. The vendor had answered all 312 questions, attached a current SOC 2, and written thoughtful paragraphs about their encryption and their access reviews. What nobody in the room could tell me was what the 94 measured. Not how much of our data the vendor would hold, not how deeply they would sit inside a critical process, not what would happen to us if they went dark on a Tuesday. The 94 measured one thing with real precision: how good this vendor was at answering questionnaires. That is a skill, and it correlates with sales maturity far more than with security.

This is the quiet failure at the center of most third-party risk programs. The questionnaire has stopped being an instrument for gathering evidence and has become the assessment itself. The vendor describes their own controls, the analyst scores the description, and the score inherits every bias in the source: the vendor with the biggest security team writes the best answers, the small shop that actually does the work by hand writes the worst ones, and the number rewards the wrong one. A program built this way is not measuring vendor risk. It is measuring vendor eloquence, and then filing it as assurance.

Tier the Vendor Before You Send Anything

The step that gets skipped is the one that should come first: deciding how much risk the vendor represents before they have said a word about their controls. That is inherent risk, and it is knowable from things you already own. What data will they touch, and how sensitive is it? What access will they hold in your environment? How central are they to a service your customers or regulators actually care about? How hard would they be to replace if the relationship ended badly? None of those questions are answered by the vendor. They are answered by the contract, the integration design, and your own criticality ratings, and together they tell you how much assurance you need to buy.

When you skip inherent-risk tiering and go straight to the questionnaire, you let the vendor's self-description define their risk. That is backwards. Inherent risk should set the bar the evidence has to clear, and a program that never computes it has no bar at all. It has a stack of completed questionnaires and a scoring rubric that treats a payroll processor holding two million records the same as the vendor that ships your office plants, because both filled in the same form and both got graded on how completely they filled it.

The Questionnaire Is Evidence, Not the Verdict

Once the inherent tier is set, the questionnaire has a proper job, and it is a narrow one. It is a source of evidence about whether the vendor's controls bring that inherent risk down to something you can live with. The residual score is the inherent risk moderated by verified control evidence, and the word that carries the weight is verified. A confident paragraph about least-privilege access is a claim. A screenshot of the access-review ticket, a named control owner, a SOC 2 whose scope actually covers the service you are buying rather than some adjacent product line, those are evidence. Scoring the paragraph the same as the ticket is how a 94 gets built out of sentences.

This is the same discipline I keep returning to across the whole program, from risk registers on down: a derived number stays honest because you can reconstruct it, and an asserted number drifts because no one can. A residual vendor score that you can trace back to an inherent tier and a specific piece of validated evidence will survive an examiner asking why. A score that is just the sum of the questionnaire's answer fields will not, because the honest answer to why is that the vendor said so.

One Questionnaire for Everyone Is the Tell

You can diagnose this problem in a program from across the room. If every vendor receives the same questionnaire, inherent risk was never computed, because a real inherent tier would change what you ask. The critical processor deserves the long, evidence-heavy assessment and probably a call with their security team. The low-risk, no-data, no-access vendor deserves a handful of questions and a quick check, and asking them 312 is not rigor, it is waste that steals hours from the vendors that could actually take you down. Assessment capacity is the scarcest thing in a TPRM program, and a uniform questionnaire spends it as if it were free.

Proportionality is not a nicety here, it is the whole point of tiering. The inherent tier should decide the depth of the intake, the seniority of who reviews it, the evidence you demand before onboarding, and, once the relationship is live, how often you come back to look again. That last one flows directly into cadence, which is why I argued separately that reassessment cadence should be earned, not calendared. Intake and cadence are two ends of the same decision about how much of your attention this vendor has earned.

What Onboarding Should Actually Produce

A vendor intake that works produces three things, and a completed questionnaire is not one of them. It produces an inherent-risk tier derived from data, access, and criticality. It produces a residual score that is the inherent tier adjusted by evidence you actually validated, with the evidence attached so the number can be reconstructed later. And it produces a reassessment cadence that falls out of the tier automatically, so the file knows when it will next be checked without anyone scheduling it by hand. The questionnaire is a tool used in the middle of that process, not the output filed at the end of it.

Reframed that way, intake stops being a gate the vendor has to talk their way through and becomes a decision you make with your eyes open: how much do we need to trust this relationship, and has the vendor shown us enough to justify it. Those are the same questions that make assessments worth running at all, which is the argument behind integrating risk assessments rather than running each one as an isolated form. The score is only useful if it feeds that decision. A 94 that measures fluency feeds nothing.

The processor I opened with was breached eleven months later, through a subprocessor its glossy questionnaire had mentioned in a single line nobody weighted. The 94 had been right about the vendor's ability to describe itself and silent about the risk it carried. Tier first, treat the questionnaire as evidence and not as the verdict, and size the whole exercise to what the vendor could actually cost you. Then the number on the file describes the vendor, instead of describing how well the vendor fills out forms.