Applying Artificial Intelligence to Data Governance

Since 2016, investment in Artificial Intelligence (AI) by organisations has grown exponentially, and this trend is expected to continue over the coming years. However, almost 75% of AI initiatives fail to achieve their objectives, and more than 90% of these types of projects report serious problems with the raw material that feeds them: data.

This becomes a vicious circle: I invest in AI, but I don't achieve my goals because I don't have effective and efficient data governance, but I can't invest everything I need in data governance because AI projects are allocated a larger budget than data governance projects.

And how can this vicious circle be broken?

We have three clear examples of organisations that are attempting to do this (and largely succeeding):

• Organisations that are digitally native or whose business is data, which have a data culture in their DNA and, therefore, data governance is part of their day-to-day operations and evolves as business and technological scenarios advance. The problem is that few organisations are in this situation and, in general, they tend to be small and medium-sized enterprises.
• Organisations that have been forced to implement data governance due to regulatory requirements (especially banks and financial institutions) and, in addition, have had sufficient budget to carry out mammoth initiatives, with a huge outlay of money and a surely negative ROI. It doesn't seem to be within everyone's reach, does it?
• Organisations that entrust one of their strategic assets, such as data, to a single provider that is “capable” of covering the entire data lifecycle with proprietary solutions. Not only does this result in high-value, long-term contracts and licences, but, as with all outsourcing, they are losing control of that part of the business by leaving it in the hands of a third party.

Our vision at Anjana Data

The above groups are all well and good, but the reality is that they cover a very small percentage of the current business fabric. What if we created a new group that included the vast majority of organisations not represented in the previous ones? Could we fill those gaps with distinctive, innovative and disruptive solutions?

In Anjana Data We believe so, which is why we are working on the application of AI, but with a focus entirely geared towards promoting a data culture within organisations and the effective and efficient implementation of data governance that guarantees the success of data exploitation initiatives. The impact is particularly high in initiatives as sensitive to data quality as AI.

Furthermore, if we think about it carefully, we would be covering several fronts at once, as we would be carrying out the different tasks in parallel:

• AI implementation proof of concept: Even if they do not have directly measurable business objectives, the learning, results and methodologies can still be useful.
• Incremental implementation of data governance, resulting in a direct improvement in data culture in order to have more and better comprehensible and reliable data that will subsequently feed into our AI initiatives.
• Dedication of the talent and knowledge of key people to a single goal, rather than returning to a vicious circle.

Applying AI... wouldn't you have the same problem with raw materials?

Yes, but to a lesser extent. The raw material I will need initially for these initiatives is metadata., which are divided into:

• Technical: Extracted from systems and databases. They are usually completely reliable as they define the characteristics of the data at a technical level. The biggest problem they pose is their abstraction, standardisation and understanding.
• Operational: Extracted from processes and their execution. Often, acquiring this information is not as straightforward as we would like, and sometimes we may encounter situations where we lack information in this regard. However, as a general rule, and increasingly so, its presentation is being standardised to facilitate its capture and understanding.
• Business: These exist in people's minds, and if they have ever been digitised, they will also exist in some repository from which they can be extracted. They are the most complex and also the most valuable, but they are also the easiest to obtain thanks to the continuous learning of AI algorithms.

AI at Anjana Data

I'm sure by now you're wondering... And what AI applications is Anjana Data beginning to implement to assist me with my data governance initiatives? We can't give away too many spoilers yet (because of the competition), but we can give you a few hints:

• Improved experience in searching for both business and technical objects with proposals based on user interaction. Similar to what the most powerful search engines do.
• Suggestions for objects and elements that may be of interest based on your use of the application, for example, to request access to data. Similar to the shopping experience on e-commerce platforms.
• Inference and suggestion of business terms and metadata thanks to Natural Language Processing, pattern detection and relationships between objects.
• Discovery of relationships between objects (trace, lineage) based on similarity detection algorithms and Euclidean distance measurement.
• Intelligent detection of Golden Sources and Golden Records based on metadata, data quality, and user data usage, as well as critical and/or hot data.
• Detection of duplicate or similar objects and/or elements and proposal for improvement of data structures (location, data model, performance, partitioning, etc.) based on the use of data by users and processes.
• Detection of bottlenecks in internal governance procedures and predictive recommendations for their improvement.
• Identification of misuse of data based on specified licence terms and actual use of the data.
• Identification of anomalies in data processing based on the usual behaviour of the monitored platforms.

As you can see, at Anjana Data we want to offer our customers truly distinctive, innovative and disruptive capabilities within our Data Governance solution, which is why we encourage you to learn about our solution, its current status, and where we are headed.

Do not just think about the present; bear in mind that the future is just around the corner.

Find out what the digital media are saying about we.

** You can also read this article in the print edition. No. 2 of Big Data Magazine