Model Risk Management

More questions trickling in from our attendees: Questions on DQMs Progress and Identification How far along is your firm in identifying material and complex deterministic quantitative methods (DQMs), as outlined in PRA SS1/23, and onboarding them into your model inventory? Review and Validation Practices Are you currently conducting any form of review or light validation of these onboarded DQMs? If so, what level of depth are these reviews reaching, and how are you prioritising them? Resourcing and Execution Are these DQM reviews being handled by your internal model validation teams, or are you outsourcing this work? What factors influenced your approach? Capacity and Resource Allocation How are you managing internal resource capacity to accommodate DQM reviews alongside traditional model validations? Have you had to make trade-offs or adjustments? Compliance and Readiness On a scale from early-stage to fully compliant, how would you assess your firm’s current alignment with the DQM-related expectations under SS1/23? What challenges have you encountered in meeting these requirements? Questions on AI / ML Validation How are firms adapting their model validation frameworks to address the unique challenges of AI and ML models, particularly around explainability, stability, and governance? Follow-up to that is : Are traditional validation techniques sufficient, or are you developing new tools and metrics specifically for AI/ML? Join us for a morning full of insights!

Lots of good answers here. One tough learning from implementing this at scale is that unlike traditional ML, automated tests can only capture a small fraction of what can go wrong with GenAI. While the automated validation is necessary, it is not sufficient. We have typically needed to also develop large manual testing protocols for releases, where humans (either developers or a set of test users), attempts a mixed of predefined and new prompts, and judge the quality of the answers. Often we will uncover “issues” that are very subjective, such as the answers technically being correct but pulling from different files that we wished, or answers being less/more detailed than the average user prefers, or an entirely new file format having issues (hence not covered by tests yet), or a million other things! For one of our recent clients, we ran “hackathons” along with releases where both new and power users would try various prompts and score the output. It was incredibly helpful to identify things the tests had failed to see

There is certainly precedent for this in loss forecasting, given various companies that need to follow both IFRS9 and CECL at different legal entity levels, and/or to follow different stress testing guidance for different regulators.   I can’t think of a case where I’ve seen it for the primary credit risk rating models however (at least not for literally the same exposures receiving two different ratings)

I agree, but wonder if there is something more here. I have also seen such “findings” and in the two cases I saw, they were politically/ career minded where once the MRM team was fighting a past battle (comments on how the bank uses resources, and ones they didn’t get) and once where the MRM team lead wanted the lead modeler’s job (and was trying to make a point about how they would use resources). In neither case were such “resource” use findings actually anchored in trying to do MRM work…

I think most have been said already, but just for the record I have also seen a PD master scale being validated by MRM (US). I’d say as mentioned, the guiding principle should be how you derive the scale. If it involves mathematical and statistical methods, assumptions etc. then it most likely falls under the “model” definition and should be validated. I would tie it to how you define a model internally and what the MRM policy says about it

Hello RiskbOWl community, Oliver Wyman recently conducted a Model Risk Management Survey with 10+ GSIBs participating (thank you to the onces who participated). One of the questions we asked in this survey was "what are your current priorities for MRM?". The top answers were: 100% said: Expand the scope of MRM (i.e. adding new model types such as AI but also more and more non-models entering MRM) 92% said: Increase resource productivity via simplification, streamlining, etc. 92% said: Increase usage of AI tools to support validators 66% said: Increase validation quality less frequent answers: reduce validation frequency, change to event-based validation, increase offshoring or outsourcing What are your current strategic priorities? Where do you see current challenges?