What is a “master data management system” and why is it needed?
What are the data
Before going directly to the master data management systems, let's determine what kind of data there is at all.
Below are 5 key types:
')
1. Metadata (Metadata);
2. Reference data;
3. Master data;
4. Transactional data;
5. Historical data.
Metadata is data about data. They are needed to understand and determine what data the company operates. Metadata defines structures, data types, access to them, etc. There are various schemes for describing metadata. For example, an XSD schema may be used to describe the structure of an XML document, or a WSDL schema to describe a web service.
Reference data is a relatively rarely changing data that determines the values of specific entities used when performing operations throughout the enterprise. These entities most often include: currencies, countries, units of measurement, types of contracts / accounts, etc.
Master data is the baseline data that defines the business entities that an enterprise deals with. These business entities typically include (depending on the subject-specific sectoral focus of the enterprise) customers, suppliers, products, services, contracts, bills, patients, citizens, etc. In addition to information directly about a particular master entity, master data includes relationships between these entities and hierarchies. For example, in terms of finding additional sales opportunities, it may be very important to identify explicit and implicit relationships between individuals. Master data is distributed throughout the enterprise and is involved in all business processes. Usually, master data is perceived as a key intangible asset of an enterprise, since on their quality and completeness depends on the effectiveness of its work. In Russia, the term “regulatory reference information” is often used instead of the term “master data”.
Transactional data is data that is formed as a result of an enterprise performing any business transaction. For example, for a commercial enterprise: sales of products and services, purchases, cash receipts / withdrawals, receipts to a warehouse, etc. Typically, such data is based on an enterprise resource management system (ERP) or other industry-specific systems. Naturally, transactional systems make extensive use of master data when performing transactions.
Historical data is data that includes historical transactional and master data. Most often, such data is accumulated in ODS and DWH systems and serves to solve various analytical tasks and support management decision making.
Before moving on to the master data management system, we define what master data management is in general.
Master Data Management (MDM) - a discipline that works with master data in order to create a “golden record”, that is, a holistic and comprehensive understanding of the master entity and relationships, the master data standard that is used by the entire enterprise and sometimes between enterprises to facilitate the exchange of information.
Specialized master data management systems (MDM systems) automate all aspects of this process and are the “authoritative” source of enterprise-wide master data. Often, MDM systems also manage reference data.
The situation when the MDM system is the only source of master data, all changes are made to the MDM system and only then transmitted to the system-consumers, is called the “recording system”. This is the ideal situation for master data management. However, in real life, things are not so simple: an MDM system will not always be a “recording system”. Due to the nature of the business processes of a particular enterprise, the technical difficulties of specific systems, etc., it is necessary to create "copies" of master records. The system that contains a copy of the master data is called the “reference system”. In order not to lose controllability, the “reference system” must be controlled and synchronized with the “recording system”.
Consider the MDM – system in three dimensions:
Usually, MDM-systems are not implemented “with a swoop”, because their implementation is a complex process of consistent transformations across the entire enterprise, from maintaining scattered data to creating a holistic, comprehensive view of the master entity. Therefore, the implementation of MDM systems is performed sequentially with a gradual approach to the target result in the three specified dimensions.
Let us consider these measurements in more detail.
In the context of master data management, a domain is a specific area of master data. The most common master data domains are the customer domain and the product domain. Western literature has established established terms for managing master data within these domains: Customer Data Integration (CDI) for the customer domain and Product Information Management (PIM) for the product domain.
CDI traditionally includes not only customers, but also organizations or individuals, which may be called differently depending on the industry of the enterprise: customers, suppliers, banks, funds, patients, citizens, etc.
PIM traditionally includes: products, goods, materials, services, jobs, etc.
There are many similarities in approaches to managing master data for CDI and PIM, but there are also many differences. For example, when deduplicating client entities, in most cases, simple syntactic analysis of entity attributes and their comparison based on probabilistic algorithms is performed, while in the product domain a semantic / ontological analysis of attributes is performed with self-learning mechanisms connected. In addition, in the grocery domain, entities may have different attributes depending on the category selected (for example, laptops have their own set of attributes, and washing machines have their own). All these features of different domains must be supported by MDM systems.
Recently, there is a tendency to create multi-domain MDM¬-systems with the possibility of flexible adjustment of the structure of metadata. This flexibility gives the company the opportunity to describe the master data specifically for themselves, taking into account all the features and nuances, but it requires considerable time and knowledge in order to correctly design and configure such a system. Also on the market there are systems with a "rigid" structure of master entities that have already correctly configured mechanisms, but the use of such a system is possible only by those enterprises that can adapt to it. Typically, such systems are well applicable for solving the problem of master data management within a narrow industry. In my opinion, the most promising are systems with a flexible metadata model, but having models that are pre-configured for enterprises in different industries, which can be quickly reconfigured.
Methods of using MDM (Method of use) determine what the MDM system will be used for in the enterprise. In other words, who will be the master data consumer (of course, there may be several of them).
There are three main methods of use:
1. Analytical
2. Operational (Operational)
3. Collective (Collaborative)
The analytical method of use supports business processes and applications that use master data primarily for analyzing business performance, provide the necessary reports and perform analytical functions. Often this happens through the interaction of MDM with tools and BI products. Usually, an analytical MDM system works with data only in read mode, it does not change the data in the source systems, but deals with their cleaning and enrichment.
The operational use method allows you to collect, modify and use master data during the execution of business transactions (operations) and serves to support the semantic consistency of the master data within these operations across all operational applications. In fact, in this case, MDM functions as an OLTP system that processes requests from other operational applications or users. Work in this mode often requires the construction of a unified integration landscape using the principles of service-oriented architecture (SOA) and the use of enterprise service bus tools (ESB). Ideally, such tools are either directly included in the MDM system, or are its continuation (there are vendors that have both MDM and ESB solutions deeply integrated with each other in their lineup).
The collective method of use allows you to create master entities in cases where collective interaction is required between different groups of users in the process of this creation. Such coordination usually has complex “branching” business processes consisting of various automatic and manual tasks. Manual tasks are performed by various data specialists (data stewards) in the order determined by the business process. Most often, the collective method of use is used in the grocery domain. For example, when creating a new product, when there are several people responsible for entering different data, a lot of manual work and final coordination. It is important that the MDM system allows you to customize arbitrary business processes to quickly support the business processes of a particular enterprise.
Usually there are three basic styles of implementation (implementation style):
1. Registry (registry);
2. Coexistence;
3. Transactional (transactional).
The registry implementation style involves creating a master data source as a “reference system” to downstream data sources. Registry MDM contains only the key attributes needed to identify and match entities. Registry MDM operates in read-only mode, data is entered in source systems and transmitted to MDM to resolve entities. Also in the registry MDM can be stored links to non-key data sources, but the data itself is usually not transmitted to MDM. The registry implementation style is usually used when choosing the operational method of using MDM (see above).
A co-existing implementation style implies the existence of distributed data entry in several sources (business applications and an MDM system). An MDM system in this case can be a “record system” for only part of the attributes. However, a full-fledged master entity is formed in the MDM system, the changes of which are translated into other systems (perhaps not all). The co-existing implementation style is fairly simple and is often used as the first step to the next, transactional style, because does not require deep processing of systems that interact with the MDM system.
Transactional implementation style involves the creation of a complete “record system” in which all data on master entities is stored. The MDM system in this case is the “only source of truth” for all systems-consumers. All operations for the creation and processing of data is performed at the level of the MDM system. Data entry at the system-consumer level is prohibited. This approach is usually quite complicated to implement, because requires a significant change in business processes and subscriber systems.
In practice, the choice of a particular MDM implementation strategy is determined by many factors: the company's objectives in the field of master data management, the level of enterprise maturity, the degree of readiness of the IT infrastructure, the availability of investments for the project and many other parameters. In order to determine the implementation strategy, it is necessary to conduct a thorough analysis of all these factors and draw up a detailed feasibility study of the project and a detailed schedule with indication of the phases of project development. But this is another extensive topic that requires separate consideration.
One thing is for sure that the introduction of an MDM system must be approached very carefully and progressively. Most of the projects implementing MDM systems fail precisely because of the underestimation of the complexity and amount of changes encountered in MDM projects.
Maxim Vlasov
Before going directly to the master data management systems, let's determine what kind of data there is at all.
Below are 5 key types:
')
1. Metadata (Metadata);
2. Reference data;
3. Master data;
4. Transactional data;
5. Historical data.
Metadata is data about data. They are needed to understand and determine what data the company operates. Metadata defines structures, data types, access to them, etc. There are various schemes for describing metadata. For example, an XSD schema may be used to describe the structure of an XML document, or a WSDL schema to describe a web service.
Reference data is a relatively rarely changing data that determines the values of specific entities used when performing operations throughout the enterprise. These entities most often include: currencies, countries, units of measurement, types of contracts / accounts, etc.
Master data is the baseline data that defines the business entities that an enterprise deals with. These business entities typically include (depending on the subject-specific sectoral focus of the enterprise) customers, suppliers, products, services, contracts, bills, patients, citizens, etc. In addition to information directly about a particular master entity, master data includes relationships between these entities and hierarchies. For example, in terms of finding additional sales opportunities, it may be very important to identify explicit and implicit relationships between individuals. Master data is distributed throughout the enterprise and is involved in all business processes. Usually, master data is perceived as a key intangible asset of an enterprise, since on their quality and completeness depends on the effectiveness of its work. In Russia, the term “regulatory reference information” is often used instead of the term “master data”.
Transactional data is data that is formed as a result of an enterprise performing any business transaction. For example, for a commercial enterprise: sales of products and services, purchases, cash receipts / withdrawals, receipts to a warehouse, etc. Typically, such data is based on an enterprise resource management system (ERP) or other industry-specific systems. Naturally, transactional systems make extensive use of master data when performing transactions.
Historical data is data that includes historical transactional and master data. Most often, such data is accumulated in ODS and DWH systems and serves to solve various analytical tasks and support management decision making.
Master Data Management Systems
Before moving on to the master data management system, we define what master data management is in general.
Master Data Management (MDM) - a discipline that works with master data in order to create a “golden record”, that is, a holistic and comprehensive understanding of the master entity and relationships, the master data standard that is used by the entire enterprise and sometimes between enterprises to facilitate the exchange of information.
Specialized master data management systems (MDM systems) automate all aspects of this process and are the “authoritative” source of enterprise-wide master data. Often, MDM systems also manage reference data.
The situation when the MDM system is the only source of master data, all changes are made to the MDM system and only then transmitted to the system-consumers, is called the “recording system”. This is the ideal situation for master data management. However, in real life, things are not so simple: an MDM system will not always be a “recording system”. Due to the nature of the business processes of a particular enterprise, the technical difficulties of specific systems, etc., it is necessary to create "copies" of master records. The system that contains a copy of the master data is called the “reference system”. In order not to lose controllability, the “reference system” must be controlled and synchronized with the “recording system”.
Three dimensions of MDM systems
Consider the MDM – system in three dimensions:
Usually, MDM-systems are not implemented “with a swoop”, because their implementation is a complex process of consistent transformations across the entire enterprise, from maintaining scattered data to creating a holistic, comprehensive view of the master entity. Therefore, the implementation of MDM systems is performed sequentially with a gradual approach to the target result in the three specified dimensions.
Let us consider these measurements in more detail.
Domains
In the context of master data management, a domain is a specific area of master data. The most common master data domains are the customer domain and the product domain. Western literature has established established terms for managing master data within these domains: Customer Data Integration (CDI) for the customer domain and Product Information Management (PIM) for the product domain.
CDI traditionally includes not only customers, but also organizations or individuals, which may be called differently depending on the industry of the enterprise: customers, suppliers, banks, funds, patients, citizens, etc.
PIM traditionally includes: products, goods, materials, services, jobs, etc.
There are many similarities in approaches to managing master data for CDI and PIM, but there are also many differences. For example, when deduplicating client entities, in most cases, simple syntactic analysis of entity attributes and their comparison based on probabilistic algorithms is performed, while in the product domain a semantic / ontological analysis of attributes is performed with self-learning mechanisms connected. In addition, in the grocery domain, entities may have different attributes depending on the category selected (for example, laptops have their own set of attributes, and washing machines have their own). All these features of different domains must be supported by MDM systems.
Recently, there is a tendency to create multi-domain MDM¬-systems with the possibility of flexible adjustment of the structure of metadata. This flexibility gives the company the opportunity to describe the master data specifically for themselves, taking into account all the features and nuances, but it requires considerable time and knowledge in order to correctly design and configure such a system. Also on the market there are systems with a "rigid" structure of master entities that have already correctly configured mechanisms, but the use of such a system is possible only by those enterprises that can adapt to it. Typically, such systems are well applicable for solving the problem of master data management within a narrow industry. In my opinion, the most promising are systems with a flexible metadata model, but having models that are pre-configured for enterprises in different industries, which can be quickly reconfigured.
Methods of use
Methods of using MDM (Method of use) determine what the MDM system will be used for in the enterprise. In other words, who will be the master data consumer (of course, there may be several of them).
There are three main methods of use:
1. Analytical
2. Operational (Operational)
3. Collective (Collaborative)
The analytical method of use supports business processes and applications that use master data primarily for analyzing business performance, provide the necessary reports and perform analytical functions. Often this happens through the interaction of MDM with tools and BI products. Usually, an analytical MDM system works with data only in read mode, it does not change the data in the source systems, but deals with their cleaning and enrichment.
The operational use method allows you to collect, modify and use master data during the execution of business transactions (operations) and serves to support the semantic consistency of the master data within these operations across all operational applications. In fact, in this case, MDM functions as an OLTP system that processes requests from other operational applications or users. Work in this mode often requires the construction of a unified integration landscape using the principles of service-oriented architecture (SOA) and the use of enterprise service bus tools (ESB). Ideally, such tools are either directly included in the MDM system, or are its continuation (there are vendors that have both MDM and ESB solutions deeply integrated with each other in their lineup).
The collective method of use allows you to create master entities in cases where collective interaction is required between different groups of users in the process of this creation. Such coordination usually has complex “branching” business processes consisting of various automatic and manual tasks. Manual tasks are performed by various data specialists (data stewards) in the order determined by the business process. Most often, the collective method of use is used in the grocery domain. For example, when creating a new product, when there are several people responsible for entering different data, a lot of manual work and final coordination. It is important that the MDM system allows you to customize arbitrary business processes to quickly support the business processes of a particular enterprise.
Implementation styles
Usually there are three basic styles of implementation (implementation style):
1. Registry (registry);
2. Coexistence;
3. Transactional (transactional).
The registry implementation style involves creating a master data source as a “reference system” to downstream data sources. Registry MDM contains only the key attributes needed to identify and match entities. Registry MDM operates in read-only mode, data is entered in source systems and transmitted to MDM to resolve entities. Also in the registry MDM can be stored links to non-key data sources, but the data itself is usually not transmitted to MDM. The registry implementation style is usually used when choosing the operational method of using MDM (see above).
A co-existing implementation style implies the existence of distributed data entry in several sources (business applications and an MDM system). An MDM system in this case can be a “record system” for only part of the attributes. However, a full-fledged master entity is formed in the MDM system, the changes of which are translated into other systems (perhaps not all). The co-existing implementation style is fairly simple and is often used as the first step to the next, transactional style, because does not require deep processing of systems that interact with the MDM system.
Transactional implementation style involves the creation of a complete “record system” in which all data on master entities is stored. The MDM system in this case is the “only source of truth” for all systems-consumers. All operations for the creation and processing of data is performed at the level of the MDM system. Data entry at the system-consumer level is prohibited. This approach is usually quite complicated to implement, because requires a significant change in business processes and subscriber systems.
Conclusion
In practice, the choice of a particular MDM implementation strategy is determined by many factors: the company's objectives in the field of master data management, the level of enterprise maturity, the degree of readiness of the IT infrastructure, the availability of investments for the project and many other parameters. In order to determine the implementation strategy, it is necessary to conduct a thorough analysis of all these factors and draw up a detailed feasibility study of the project and a detailed schedule with indication of the phases of project development. But this is another extensive topic that requires separate consideration.
One thing is for sure that the introduction of an MDM system must be approached very carefully and progressively. Most of the projects implementing MDM systems fail precisely because of the underestimation of the complexity and amount of changes encountered in MDM projects.
Maxim Vlasov
Source: https://habr.com/ru/post/324148/
All Articles