Model Risk Management

Lots of good answers here.

One tough learning from implementing this at scale is that unlike traditional ML, automated tests can only capture a small fraction of what can go wrong with GenAI. While the automated validation is necessary, it is not sufficient.

We have typically needed to also develop large manual testing protocols for releases, where humans (either developers or a set of test users), attempts a mixed of predefined and new prompts, and judge the quality of the answers. Often we will uncover “issues” that are very subjective, such as the answers technically being correct but pulling from different files that we wished, or answers being less/more detailed than the average user prefers, or an entirely new file format having issues (hence not covered by tests yet), or a million other things!

For one of our recent clients, we ran “hackathons” along with releases where both new and power users would try various prompts and score the output. It was incredibly helpful to identify things the tests had failed to see

There is certainly precedent for this in loss forecasting, given various companies that need to follow both IFRS9 and CECL at different legal entity levels, and/or to follow different stress testing guidance for different regulators.   I can’t think of a case where I’ve seen it for the primary credit risk rating models however (at least not for literally the same exposures receiving two different ratings)

I agree, but wonder if there is something more here.

I have also seen such “findings” and in the two cases I saw, they were politically/ career minded where once the MRM team was fighting a past battle (comments on how the bank uses resources, and ones they didn’t get) and once where the MRM team lead wanted the lead modeler’s job (and was trying to make a point about how they would use resources). In neither case were such “resource” use findings actually anchored in trying to do MRM work…

I think most have been said already, but just for the record I have also seen a PD master scale being validated by MRM (US).

I’d say as mentioned, the guiding principle should be how you derive the scale. If it involves mathematical and statistical methods, assumptions etc. then it most likely falls under the “model” definition and should be validated. I would tie it to how you define a model internally and what the MRM policy says about it

Hello RiskbOWl community,
Oliver Wyman recently conducted a Model Risk Management Survey with 10+ GSIBs participating (thank you to the onces who participated). One of the questions we asked in this survey was "what are your current priorities for MRM?". The top answers were:

100% said: Expand the scope of MRM (i.e. adding new model types such as AI but also more and more non-models entering MRM) 92% said: Increase resource productivity via simplification, streamlining, etc. 92% said: Increase usage of AI tools to support validators 66% said: Increase validation quality less frequent answers: reduce validation frequency, change to event-based validation, increase offshoring or outsourcing

What are your current strategic priorities? Where do you see current challenges?