The client requested Avantage Reply’s assistance with the review and validation of their monitoring tool (i.e. Monitoring Scorecard). This tool aimed to standardise, centralise and implement a common control mechanism to the monitoring activities related to the general mandate universe.
In the course of the project, Avantage Reply applied its expertise in model validation and worked in close liaison with the team who developed the monitoring tool in order to ensure the comprehensiveness of the tool validation and to analyse a possible extension of the application of this tool to other mandates.
The client was seeking support to carry out the initial validation of a newly developed monitoring tool, while ensuring its reliability and flexibility in application to other mandates.
The steps agreed with the client included a technical review, including the analysis of the performance of the developed scorecard in the tool, a detailed review of the supporting documentation and an assessment of the governance around the tool regarding the existing regulatory expectations (i.e. BCBS 239, EBA IT RISK guidelines).
The requested validation exercise had to evaluate the model from a fit-for-purpose perspective through a structured, comprehensive model validation methodology. Both a qualitative assessment of the proposed methodology (including adequacy and fitness of the underlying theory for the intended purposes, validity of the involved assumptions, etc.) and a quantitative assessment of the model (including accuracy of the implementation, sensitivity to model parameters, performance with respect to the relevant indicators, etc.) had to be performed.
The tool was used for one special mandate at the time of review. The analysis of the future application of this monitoring tool required an understanding of the specific characteristics of each mandate and an identification of the similarities between all mandates.
The tool was developed in excel, and thus subject to manual manipulations and lack of an audit trail.
The validation of the monitoring tool has been performed by leveraging the Avantage Reply expertise in the model risk management field and the design & development of End-User-Computing solutions. The monitoring tool was built around a specifically designed scorecard. The team decomposed each element of the scorecard to validate the underlying assumptions, to analyse the correlations between each of the components and to review the appropriateness of each component for the application for other mandates.
In addition, we proposed a set of improvements for the monitoring tool:
As a result of the exercise, the client was provided with a comprehensive validation report, comprising a concise description of the monitoring tool, an assessment of the tool’s design and its fitness for purpose, an examination of the underlying assumptions of the scorecard, analysis of the governance process around the tool as well as an overall validation conclusion and a list of identified gaps and recommended actions.