The Black Art of Reference Data Distribution
Cliff Cunningham and John Randles
The challenge of controlling Reference Data Distribution is often underestimated. It can be a highly complex equation to manage, depending on the number of asset classes and downstream systems in play. The difficulty rapidly escalates when you add the intricacies of the Golden Copy message, the various nuances of individual asset classes and the interdependencies of the downstream systems. This paper addresses the:
- Complexity associated with Reference Data Distribution
- Difficulties with EAI/ETL tools
- Driver to turn Reference Data Distribution into a competitive advantage.
It is no surprise organizations struggle to effectively utilize and leverage rules for multiple downstream systems, all with different requirements, coding schemes, structures, versions and technology interfaces. In fact, the holy grail of Reference Data Distribution is achieving this while maintaining a transparent rule repository that is easily searched, maintained, understood, changed and verified. Consider, for example, the number of integration-use cases a Security Master RDD project has to handle. There are potentially many asset classes to manage, variations within asset classes, multiple downstream systems each with interdependent components and several potential operations (e.g. insert, selective update, replace, delete), and possibly different rules across markets and geographies.
In a recent Securities Master distribution project implemented we were involved in there were over 360 use cases involving 10,000 rules. Capturing the business rules in a manageable manner in such situations is always the key challenge.
While point-to-point integrations are easily created they can quickly become un-maintainable. Business users tend to say: “Take that piece of information from there in all cases, except if it is this asset class and therefore not for these operations.” Or “It’s the same message structure as for Equities with these fields added and these other fields being populated from different sources”. This is very difficult to achieve with point-to-point integration.
The dimensions for the rules can be arbitrary, such as asset type, geography, market, application version, application interface, operation type. The real challenge is to enable rules to grow in an ordered and manageable way as the complexity of your distribution requirements grows.
One approach is to allow all downstream applications access a central repository in order to get data or to take data in a common format from a message bus and convert this to a local format and manage local dependencies. With scale this becomes a very invasive and risky approach as each application line ends up having to engage in integration procedures that are not core competencies.
The Black art of Reference Data Distribution has been read by over five hundred executives across buy side and sell side firms. To receive a copy of this paper, please email info@polarlake.com.
2010