Data

In this post, we will explore how to use SQL window functions within the Gold layer to solve two common needs: calculating granular aggregates and generating synthetic test data for healthcare scenarios. Modern data engineering has converged on the Medallion Architecture as the gold standard for managing scalable data lakes. By organizing data into Bronze […]

Executive Summary Master Data Management (MDM) is the discipline that enables organizations to maintain accurate, consistent, and trusted domain-specific master entities—customers, products, distributors, patients, and more—across all systems. It ensures enterprise operations and analytics are based on a single source of truth. Historically, MDM has been slow and resource-intensive, creating latency between data capture and

As a data engineer navigating cloud platforms and analytics ecosystems, I’ve worked extensively with both Databricks and Azure Synapse Analytics. While they often appear side-by-side in Azure environments, they come from different design philosophies and cater to slightly different needs—even when they seem to do many of the same things. Both platforms allow you to

When One Field Tries to Do Too Much Across many organizations, the same pattern repeats. A single data field—perhaps introduced during a system rollout or legacy data migration—is intended to serve a focused purpose. Over time, however, teams start using it to represent different ideas. Marketing redefines the field to support campaigns. Sales uses it

When building cloud data pipelines with Azure Data Factory (ADF), you’re often presented with two paths: use the tried-and-true Copy Activity, or lean into the newer, powerful Mapping Data flows. Both are native ADF features and technically serve the same purpose—moving and transforming data—but the similarities end there. In our case, we started with Data

I worked many years with the NPI dataset. Transforming the data brought actionable insight. Imagine a conversation between an enterprise IT strategist, a healthcare IT consultant, and a healthcare administrator. The three sit around a conference table reviewing a dataset pulled from the National Plan and Provider Enumeration System (NPPES), full of NPIs, taxonomy codes,

In fact there is not a more suitable collection of problems than those found in IT. I’ll mention two and walk through the second – Data Quality. Use Six Sigma to squash bugs. Those are the pesky defects that litter software applications. Reducing defects is the very reason Sig Sigma exists. Look no further than

so you can report off of it. Note: I actually profile my Databases completely different than what is presented here, but this information might be useful to someone else. If you have Microsoft SSIS, then you can take advantage of the Data Profiling Task. It profiles your tables and columns and outputs an XML file.

I started out considering MDM, but then… I truly believe that the process is going to be the big differentiator in the Data Management space in IT. The technology will just aid the process. Actually, the process can commence before technology is ready and then technology can catch up easily. As I dug into this

Many organizations struggle with duplicate customer sites and accounts scattered across operational systems. These duplicates introduce confusion in invoicing, shipping, reporting, and analytics. The immediate need to resolve this data fragmentation often outpaces the availability of enterprise-wide tooling. This post outlines how we built a pragmatic, SQL-based deduplication framework to identify, cluster, and prepare duplicate

In the information age, competitive advantage comes from making business sense of vast amounts of data. Departments demand Business Intelligence for specific local needs that may not be offered by a central enterprise BI team. How do we begin to meet these needs in an orderly way? This post addresses this need with a framework

You don’t have to live with the repeated pain of transforming xml to rowset values when you need data from xml data stores. You also don’t need to transform and copy the data to relational database tables in MS SQL Server to be able to query easily in the future. I like to leave the data