Accelerating dbt Core for Enterprise Data Platform

Customer Introduction  

Orrum Clinical Analytics leverages technology to revolutionize patient care through the transformation of clinical data into actionable knowledge. It was born from a commitment to patient quality and safety by leaders in perfusion and extracorporeal life support services, these pioneers sought to deliver life-saving innovations and share critical lessons learned with everyone.

Executive Summary  

Orrum partnered with Datacoves to address their data management issues and elevate their services. Utilizing Datacoves' comprehensive managed dbt core enterprise data platform, along with their expertise in Data Governance and Data Security, Orrum improved the reliability, scalability, and performance of their data infrastructure. This collaboration led to higher customer satisfaction by allowing Orrum to deliver robust insights to the hospital partners they work with, and prepared Orrum for expansion into the European region.

The Challenge

Security and Data Management Needs:

Orrum faced challenges in managing growing data needs while ensuring security. They needed an innovative solution that could leverage dbt core and utilize Snowflake's capabilities to maintain optimal security. A significant hurdle was establishing appropriate roles and permissions in Snowflake to safeguard sensitive patient information.

Platform failure and tooling issues:

Existing tools adhered to the traditional ETL approach involving data movement between tools and this led to increased platform failure points and delays. For Orrum, visualizing the entire process and comprehending the logic employed in various jobs and transformations had proved challenging when using visual transformation tools like SSIS and SQL Server.

Implementing and Maintaining a Robust Data Management Solution:

Orrum wanted to do things the right way from the start; they recognized that configuring Snowflake and dbt core with best practices from the outset would be a daunting task, potentially resulting in a slow and arduous implementation process. "Navigating the wide range of tools compatible with Snowflake felt like trekking through a jungle. It was a lengthy process, and I invested a lot of time trying to understand everything," said Eugene Kim, Orrum's Data Architect. The team might also be unable to fully take advantage of Snowflake and dbt's advanced features, which could hinder Orrum's ability to achieve their long-term data objectives.

The Solution  

Streamlined Analytics Infrastructure with DataOps:

Orrum assessed various platforms before ultimately choosing Datacoves for their ability to assist in creating a secure data platform that is tailored to their specific needs. Datacoves not only supplied a ready-to-use infrastructure deployable within Orrum's Azure account, but also contributed their accelerators and best practices. This enabled Orrum to swiftly set up a secure enterprise data platform that fulfilled their requirements.

A key advantage Datacoves offered was their seamless integration with Snowflake, allowing Orrum to retain their data within the Snowflake ecosystem. Datacoves’ dbt core proficiency was essential for establishing the architecture and infrastructure to meet Orrum's data demands. Datacoves' guidance on role management and permissions in Snowflake proved invaluable. They delivered a clear template and a set of tools for handling Snowflake permissions as code, significantly simplifying the establishment of a secure platform to manage Protected Health Information (PHI) data with fully auditable changes. This was particularly advantageous for Orrum's team, who operate in a regulated industry with stringent requirements.  

"Datacoves was instrumental in helping us adhere to best practices, especially regarding permissions management and separating development and production environments. Without clear guidelines, it's easy to inadvertently develop in production, leading to errors and inconsistencies in crucial reports. With [the] Datacoves [dbt core platform], we could establish and follow best practices," Eugene explained. Consequently, Datacoves became the key accelerator that enabled Orrum's team to set up DataOps processes for Snowflake and dbt more cost-effectively than doing it alone or hiring consultants to build everything from scratch.

The Result  

Effortless Deployment of a Modern Data Platform:  

Utilizing Datacoves' expertise and technology, Orrum seamlessly deployed a modern data platform. This process eliminated the need for purchasing and integrating multiple solutions, as Orrum could fully harness the capabilities of Snowflake, dbt and other tools. Datacoves' accelerators were instrumental in enabling Orrum to adopt customized DataOps best practices. Additionally, Datacoves streamlined Orrum's development process by implementing a hosted VS Code editor and establishing a production deployment process on Airflow. This process integrated custom Extract and Load scripts with Azure CLI dependencies before running the dbt transformation step, significantly accelerating Orrum's time to value and enabling them to become fully operational within weeks. Eugene Kim, Orrum's Data Architect, remarked, "Without Datacoves, I would have needed an entire team to integrate all these services."  

Cost-effective and Reliable Data Management:  

Datacoves provided Orrum with a cost-effective and reliable method for storing and analyzing their data, markedly reducing the risk of data loss or exposure. This allowed Orrum to draw more reliable insights from their data, enhancing their ability to deliver better insights to their customers and improving patient outcomes. The Datacoves managed enterprise dbt core platform played a pivotal role in helping Orrum establish a sophisticated data foundation. It offered improved insights for their data management strategy, giving Orrum a competitive edge in the healthcare market.  

Setup for Success at Scale:  

Orrum's collaboration with Datacoves underscores the importance of choosing the right managed dbt core enterprise platform and investing in dependable, secure infrastructure for business growth and success. Through this partnership, Orrum was able to manage the end-to-end data stack effectively. By adopting DataOps processes and tools, Orrum successfully established a strong and mature data management platform, setting a solid foundation for their ongoing development and expansion in the healthcare sector.

Conclusion  

Orrum's collaboration with Datacoves enabled them to implement sophisticated DataOps processes efficiently, providing them a competitive edge in the healthcare market. The partnership exemplifies the importance of choosing the right managed dbt core enterprise platform and investing in a reliable, secure infrastructure for business growth and success.