Customized cloud transformation for efficient data productization
About the Company
MCH Strategic Data offers B2B services, primarily selling lists of contacts (names and emails) to customers who use the data for direct marketing or research purposes. MCH sources data from its own research teams, web crawlers, and several other data vendors. It primarily serves the education, healthcare, and government sectors.
OPPORTUNITY
Data is MCH’s product for its clients. As the main product, MCH needs a highly efficient data transformation flow to turn raw data into something usable for its clients. It is imperative that the data is clean, accurate, quickly ingestible, clearly cataloged, and securely stored.
In addition to data transformation practices, MCH is transitioning away from its existing hybrid IT architecture, consisting of on-premises infrastructure communicating with Azure-based virtual machines, in favor of a completely Azure cloud-based architecture.
SOLUTION
27Global architected, designed, and worked with MCH to build a new cloud based-data infrastructure. The creation of a new data model, data pipelines, and data warehouse resulted in improvements in the data ingestion process, less manual effort, and the decommissioning of the existing, on-premise data platform.
27Global Provided:
Data Engineering
- Created a new data model and cloud data architecture
- Replaced existing Redpoint and Stored Procedure (PL-SQL) data transformations into Microsoft Azure Data Factory
- Ingested and transformed data from a variety of data sources
- Integrated TowerData (email address validation) and SmartyStreets (physical address verification) directly into transformation pipelines
- Provided project management for 27Global and MCH developers
Cloud Transformation
- Built new data platform in the Azure cloud, implementing infrastructure as code using Terraform and automating deployment with continuous integration and continuous delivery/continuous deployment (CI/CD) pipelines in Azure DevOps
- Built automations for data ingestion, transformation, output, and other integrations
- Transitioned existing on-premises Oracle server data into new data model in Hyperscale Azure SQL
- Refactored existing Scrapy web crawler to run on Kubernetes
- Assisted MCH development team with source control and deployment best practices
RESULTS
The new platform will decrease monthly maintenance costs on MCH’s existing platform while increasing the accuracy and throughput of data. This results in higher data volume and quality for their customers. The cloud-based platform allows for easier and faster management of their services. The new Azure-based platform has reduced maintenance windows to allow for new data to be published nightly instead of weekly, reducing off-hours work and weekend commitments from MCH staff.
Through 27Global’s process, MCH was exposed to and began building competencies with several new technologies (Terraform, Azure Data Factory) so they can continue to maximize their investment. Our work with MCH also uncovered an aging desktop app that allows for direct data entry, research, and error correction against the data warehouse. This app needed to be rebuilt to integrate with the new data warehouse. 27Global consultants assisted with aligning the two initiatives, incorporating the data entry application into a shared design and architecture with the data platform.
