In fact there is not a more suitable collection of problems than those found in IT. I’ll mention two and walk through the second – Data Quality.
Use Six Sigma to squash bugs.
Those are the pesky defects that litter software applications. Reducing defects is the very reason Sig Sigma exists. Look no further than Six Sigma for the tool-set that can set teams free from the plagues of buggy software.
But Six Sigma is not only a collection of tools. It is also a discipline. In fact, although there are software packages out there that help you to implement it, there is no magic to it. Managers must commit to a process and do the work that reduces bugs, incidents, and defects.
Use Six Sigma to achieve continuous improvement in Data Quality
This is the corner of IT where I sneak away to these days and this is the problem that I’ll walk through using Six Sigma. The nice thing about taking ‘Data Quality’ as a use case for Six Sigma is that Data Quality inherently asks the question: “What are you talking about?” In other words, “What does data quality mean to you?”
DMAIC
Like I said, Six Sigma includes a collection of tools. One of the most fundamental is DMAIC, that is, Define, Measure, Analyze, Improve, and Control. I’ll walk through DMAIC for data quality in the next few paragraphs.
Before you can improve Data Quality, you have to do some analysis. Before you do the analysis, you have to take measurements. And before you take measurements, you have to define what data quality means. Also, since you are not just fixing the data quality problem one time, you need an ongoing process of reassessment and improvement – that’s the control part.
Define
- First, we must understand the problem. Poor data quality affects a company’s ability to execute. Thus, in this first step we must…
- Identify Processes affected by poor data quality.
- Describe the data pipeline. What sources and what intermediary steps handle and affect data.
- Answer this major data quality question: Is there a single source of truth, or are there redundant data stores with possibly conflicting data?
- Identify Stakeholders. Who is affected by poor data quality?
- What are the boundaries between types of data. Can we define domains or subject areas that are independently score-able?
- Define data quality expectations. These are the rules that data must live by…
| Examples of Rules |
|---|
| * Fields that are phone numbers must adhere to a given regular expression that matches phone numbers; similarly for zip codes. * Category fields – what standard sets of possible values must fields be limited to. * For what fields are duplicates allowed? * What are the dependencies between fields? |
Measure
Having gathered the information above, you can set out to profile, that is to judge and to score, the data. This step may include employing automation tools. Perform the following…
- Run profiles against the data in a given domain to validate and score the quality of that data.
- Make sure to store the results with the date of the profile so that you can check quality changes across time.
- Expose the results via reports. You may choose to show the score for a given domain, a given attribute, a given table, or a given row. Consider these reporting requirements when designing the storage of the profile results.
Analyze
Don’t assume that because data came from an application that provides validation, data quality is guaranteed. Consider these scenarios in your analysis…
- There may be faulty back-end integrations that pollute the data.
- There may be legacy data that was never validated.
- There may be validation loopholes in apps, exploited by renegade or non-compliant employees.
Your choice of reporting tool from the last step could aid in this analysis step. An analyst that is able to drill-through, slice, and dice across domains, attributes (fields), and possibly data types can uncover data quality patterns and sources of problems. You might employ other tools within the six sigma tool-set such as Statistical Process Control (SPC) to identify the types of sources for bad data. You may choose to perform rigorous numerical or statistical analysis on the profile results data. This could expose data quality relationships between data points.
Finally, analysis must include the human element.
- Could poor data quality be due to a lack of training?
- Do employees face competing interests when entering data? For example, is performance measured only in speed of data entry?
Improve
Make changes that affect data quality based on your analysis. This could include any of the following.
- Incorporate data cleansing steps in the data pipeline.
- Auto-deliver “bad data” reports to data entry personnel for manual correction.
- Ensure data entry quality compliance by reviewing existing standard operating procedures.
- Adjust the rules defined previously to broaden the scope of valid data. This final improvement suggestion should not be misconstrued; it is just a natural part of data quality. This is not cheating; it is just realizing that data we thought was bad, is actually okay.
Control
This step ensures continuous improvement in data quality. The following actions may drive continuous improvement…
- Schedule and automate profiles so that data quality is tracked over time.
- Schedule delivery of data quality reports to key personnel to drive performance.
- Establish a data quality cycle. Begin with the expectation that data quality is not a project but an ongoing repeatable process. Schedule analysis and quality improvement brainstorm sessions along profile and reporting schedules.
- Demonstrate data quality improvement across time and, most importantly, demonstrate the value that that improvement brings to the business.
- Clarify who is responsible for each domain and attribute. Holding *them* accountable ensures continued interest from business personnel. Then *they* are the ones driving data quality initiatives and providing the necessary resources.
In conclusion, discipline is necessary for significant change to occur. Otherwise, risk mediocrity and obsolescence. Six Sigma can provide that discipline to Data Quality and other IT initiatives.
