【Governance of Data Islands】

1.1 What is a “data island”？

Data silos are a common phenomenon in the informatization construction of governments or enterprises, often caused by factors such as technological limitations and outdated structures. They are mainly divided into two types: physical and logical. Physical data silos refer to the situation where data is stored and maintained independently in different departments, isolated from each other. Logical data silos mean that different departments understand and define data from their own perspectives, so that some identical data is given different meanings, which无形中 increases the communication costs of cross – departmental data cooperation.

The figure below shows a very typical data silo model. Each department manages its data independently. There are differences in permissions, data structures, security, backup mechanisms, and even different database software is used. Under this model, it is difficult to carry out unified and effective data management. For an organization, every piece of data scattered among various departments is like an orphan abandoned by God, losing the value that data should have. At the same time, for the organization, it is also a great waste of resources. It seriously hinders the work efficiency among different departments of the organization. Just imagine, if I have to manage so many different types of data and database software, how much effort does the organization’s operation and maintenance personnel need to put in?

1.2 How to address data silos

The simplest way for an organization to eliminate data silos is to integrate them into a data warehouse.

BigQuery on the Google Cloud platform is a very typical product for governing data silos.

1. Create scripts or code to integrate data

Write code using scripts written in SQL or Python to extract data and move it to a central location. The only drawback is that it is time – consuming and requires a great deal of expertise.

2. Use ETL toolsETL

Extract, Transform, Load (ETL) tools can automate the entire process, eliminating the hassle of moving data. These tools extract data from sources, perform necessary transformations, and then load the data into the receiving data warehouse. These tools are usually hosted within the organization. Commonly used ones include Kettle, Sqoop, etc.

Load
Loading is the process of inserting the transformed data into the target database, data store, data warehouse, or data lake.

Extract
Extraction is the process of retrieving data from one or more sources (online, local, legacy, SaaS, or others). After retrieval or extraction, the data is loaded into a staging area.

Transform
Transformation involves obtaining data, cleaning it, and converting it into a common format so that it can be stored in a target database, data store, data warehouse, or data lake. Cleaning usually includes removing duplicate, incomplete, or obviously incorrect records.

3. Cloud – based ETL tools

These ETL tools hosted in the cloud leverage the expertise and infrastructure provided by vendors. Google Cloud Platform offers us a range of ETL services, including Cloud Data Fusion, Dataflow, and Dataproc.

A simple flowchart is as follows:

1.3 How to ensure data security

Data security is the practice of protecting digital information from manipulation and unauthorized access throughout its entire life cycle. Data security is an important part of an organization’s security posture.

For enterprises, this includes encrypting and protecting data at rest and in transit, safeguarding applications from malicious malware or ransomware attacks, and formulating security policies to detect and prevent data breaches.

1. Encrypt

Static Encryption in Google Cloud (Default Google Encryption Method)
Google has implemented multiple layers of encryption measures to protect customer data stored at rest in Google Cloud products. Google encrypts data at rest to ensure that only authorized roles and services can access this data, and access to encryption keys is audited. Data is encrypted before it is written to disk.

The following figure shows the encryption methods at each level of Google Cloud:

Static encryption in BigQuery

BigQuery automatically encrypts all data before it is written to disk. When an authorized user reads the data, the system will automatically decrypt it. By default, Google is responsible for managing the key encryption keys used to protect your data.

2. Key management

Customer Managed Encryption Key (CMEK)

CMEK for BigQuery: Cloud KMS. If you want to control encryption yourself, you can use customer-managed encryption keys (CMEK) for BigQuery. This allows you to control and manage the KEK used to protect your data in Cloud KMS, instead of having Google manage it.

The figure below shows that a storage service calls Google KMS to retrieve the unpacked DEK of the data block and decrypt the data block.

【Conclusion】

The governance of data silos is a challenge. If there is an entrenched culture of data separation, it is indeed difficult to change employees’ mindset. At the same time, there is a series of permissions and hierarchies that are difficult to remove, making it hard to eliminate silos. The simplest way is to move data from different systems into a data warehouse, which serves as a repository for all the collected data. The data warehouse is optimized for easy access and analysis, rather than transaction processing. This also ensures a comprehensive understanding of the company’s data. Once this is achieved, the company can promote the integration process and gradually build an enterprise data integration platform. To achieve all this, cross – organizational collaboration is crucial, and the executive leadership also needs to provide full – fledged support. Of course, as data is gradually used in operational and strategic applications, the changes you need to see in the organizational structure will occur naturally.