Risk Intelligence You Cannot Audit Is Just an Opinion With a Logo

Category: Enterprise Risk Author: Cody Swidler Tags: risk intelligence, country risk, third-party data, auditability, traceability, due diligence

An internal auditor once asked a team I was working with why a particular country was rated as elevated risk in their supplier assessment. The analyst pulled up the subscription portal, pointed at an amber square, and said the vendor rated it a 3. The auditor asked what drove the 3. Nobody knew. There was no methodology document available at that tier of subscription, no indicator breakdown, no date on the underlying judgement — just a square that had been amber for as long as anyone could remember. The finding that came out of it was not about the country. It was about the fact that a control decision had been made on the basis of a number nobody in the organization could explain.

I have seen versions of that conversation at four different firms. It is the central, quiet problem with most commercial risk intelligence: the product is a conclusion, and the reasoning is the part you are not buying. That is a perfectly reasonable business model — the analysis is the value, and giving it away would be giving away the company. But it creates a specific and underappreciated problem for the buyer, which is that a rating you cannot reconstruct is a rating you cannot defend.

The Three Questions a Rating Has to Survive

Any risk number that ends up influencing a decision will eventually be asked three things, and it is worth knowing in advance which ones yours can answer.

The first is where did this come from. Not which vendor — which underlying observation. A country rating that traces to specific measurable things, whether that is a government advisory level, a governance index, or a count of recorded incidents, survives this question. A rating that traces to "our regional analysts" survives it only as far as the auditor's patience extends. The distinction matters most in regulated contexts, where DORA-style regimes increasingly expect you to evidence how third-party risk determinations were reached rather than simply that they were.

The second is when was this last true. Vendor ratings are often refreshed annually, which is entirely appropriate for structural political risk and entirely useless for anything operational. The trouble is that the interface rarely distinguishes between the two. An amber square looks identical whether it was set last week in response to something concrete or eighteen months ago and never revisited. If your register carries a date at all, it usually carries the date you copied the rating across, not the date the rating was formed.

The third is what would change it. This is the one that exposes the most. A rating you can act on has a threshold behind it — something that, if it moved, would move the rating. A rating without one cannot be monitored, only re-purchased. It is the same argument I have made about risk appetite statements that lack thresholds: without a stated trigger, you do not have a control, you have a sentiment.

Opacity Is Not Incompetence, It Is a Business Model

It would be easy and wrong to read this as vendors being careless. They are not. The major intelligence firms employ genuinely excellent analysts, and in a crisis their judgement is worth paying for. Their opacity is structural: if the methodology were fully public, the ratings would be reproducible, and reproducible ratings are not a subscription business. They are also selling into a market that mostly wants the conclusion, because the conclusion is what fits in a board pack.

The failure is on the buying side, and it is a failure of expectation-setting. Firms purchase a judgement product and then deploy it as if it were an evidence product. The rating goes into the vendor register, gets copied into the risk register, informs a control decision, and by the third hop the caveats have fallen off entirely. What began as "an experienced analyst's considered view as of last autumn" arrives at the risk committee as a fact.

This is the same failure mode I described in GRC as a knowledge management discipline in denial. The problem is rarely that the organization lacks information. It is that provenance is stripped at every handoff until nobody can say where anything came from.

What Traceable Actually Means

Traceability is not the same as being open source, and it is not a demand that every input be free. It means four specific properties, and they are all testable.

Every figure names its publisher. Not "market data" but the specific agency, index, or feed, such that a reader can go and look. Every figure carries the date it was published, distinct from the date you ingested it. Every derived score states its arithmetic — if a location scores 16, it should be visible that this is a four on likelihood times a four on impact, and what drove each. And where something is judgement rather than measurement, it says so in those words, rather than borrowing the visual language of data to look more certain than it is.

That last point is the one most often violated, and the violation is usually unintentional. A hand-curated watchlist rendered in the same colour scheme as a live seismic feed reads as equally empirical. It is not. Labelling the curated thing as curated costs nothing and preserves the credibility of everything next to it.

The Test I Would Apply to Any Provider

If you are evaluating a risk intelligence subscription, the useful questions are not about coverage or refresh rate. They are these. Can you show me the indicator breakdown behind a single country's rating? What date was that rating formed, as opposed to last displayed? What specific change would move it up a band? Can I export the underlying inputs, not just the conclusion? And if my regulator asks me to evidence this determination, what do you give me?

A good provider will have answers, and some of the answers will reasonably be "that is in a higher tier." That is fine — it is a commercial conversation, and at least you know what you are buying. What you are testing for is whether the answers exist at all. If the honest response to all five is that the analysts simply know, you have bought a consulting relationship with a map attached, which may well be worth the money, but should not be sitting in your register as though it were measurement.

Why I Built the Map the Way I Did

The PivotRisk risk intelligence map exists partly as a working argument for this position. Every signal on it names the agency that published it and links back to the record. Your browser fetches each feed directly from its source — there is no PivotRisk server in the path, no stored copy, and no interpretation between the publisher and the pin. When a location scores, the arithmetic is on screen: likelihood, impact, and what drove each.

Where something is not live, it says so. The US State Department publishes travel advisories without cross-origin headers, so no browser can read them directly; that layer is therefore a dated snapshot, and it is labelled "Snapshot" rather than "Live". AWS and Azure do not expose browser-readable incident feeds at all, so their regions are plotted from published documentation and their status is a link you click. The geopolitical watchlist is hand-written by me, so it is tagged "Editorial". Those labels make the map look less impressive than it could. They are also the entire point: a dashboard that shows green because a feed failed silently is worse than no dashboard, because it converts an absence of information into an assurance.

None of this makes the map better analysis than a firm with two hundred regional specialists. It plainly is not. What it is, is checkable — and for anything that ends up in a register, in a board pack, or in front of a regulator, checkable beats authoritative. You can disagree with a number you can trace. You can only accept or reject one you cannot.