Migrating from Talend to a dbt-core managed platform
Martin Ryan
Technology Engineering
We have a background in Data Architecture, but building a way of managing dbt and Airflow together would have been a project in itself.
Martin Ryan
Technology Engineering
Customer Introduction
Johnson & Johnson (J&J) MedTech transforms medical intervention and empowers everyone to live their best life by leveraging diverse healthcare expertise, purposeful technology, and a passion for people. They have been using technology and innovation to improve patient care for over a century and are reimagining health in a digital world to improve outcomes, access, cost, and value.
Executive Summary
The J&J MedTech Team encountered issues with Talend, including high licensing costs and data pipeline delays. Partnering with Datacoves, they transitioned to a managed dbt Core solution with enhanced security, easy interface, and faster integration. This collaboration significantly reduced costs and expedited their time to market.
The Challenge
Licensing and Accessibility Challenges:
While using Talend, the MedTech Team grappled with a number of challenges, the most pronounced of which was Talend's high cost. This expense meant licenses had to be rationed, complicating the management of user access and effective permission control. Such constraints limited team empowerment and made it difficult to involve more members in the development process.
Development and Operational Hurdles:
The team aimed to integrate continuous integration/continuous deployment (CI/CD) workflows, automated testing, and deployment. However, Talend's limited support for these methodologies made it a less-than-ideal fit for DataOps and CI/CD initiatives.
Onboarding and Setup Delays:
Setting up a new developer on Talend was far from swift. It required downloading and installing a bulky software package, additional libraries, and configuring licenses. This resulted in a prolonged and tedious onboarding process.
Operational Delays and Productivity Issues:
The MedTech Team also faced significant time-related constraints with Talend. Upgrades consumed a lot of time, sometimes rendering half a day unproductive for the entire team. This, in turn, affected the timeliness of data pipelines. Moreover, the launch of new pipelines was often slow, and diagnosing issues within the pipelines became a cumbersome task. As projects scaled, the time to load the entire project in the development environment also surged, negatively impacting productivity.
The Solution
Adopting a dbt-core managed platform
The MedTech Team transitioned from Talend to Datacoves, a strategic move influenced by the promise of better data management and integration capabilities. Datacoves, known for its expertise with large enterprises, expedited the deployment of dbt Core and seamlessly developed CI/CD pipelines using tools such as Jira, Jenkins, and BitBucket.
In partnering with Datacoves, J&J benefited from a series of impactful implementations. Datacoves showcased its proficiency by adeptly managing dbt Core, a crucial tool for the MedTech Team. They tailored solutions to J&J's requirements, including deploying on a private AWS account. Their implementation of a web-based Visual Studio Code interface modernized the MedTech Team's onboarding experience. By centralizing the management of dbt Core upgrades, they ensured uniform updates across the board, enhancing collaboration and streamlining data transformation processes. The integration of dbt not only bolstered data quality but also facilitated detailed documentation and stringent pre-production testing. Datacoves' handling of multiple Airflow instances further enhanced the team's ability to create and analyze sophisticated data pipelines.
The Result
Cost Efficiency and Team Empowerment:
Datacoves emerged as an affordable alternative to Talend, offering licenses at a reduced cost. This shift eliminated license rationing, expanding user access and bolstering process transparency. By democratizing access, team dynamics improved, allowing all members to actively contribute without constraints.
Operational Excellence with Enhanced Code Management:
By integrating Datacoves, dbt Core, and adopting DataOps, the MedTech team enhanced code traceability and understanding. They can meticulously review, test, and release changes to end-users, ensuring rapid identification and resolution of issues. The structured organization of transformations in their dbt project streamlined pipeline development, resulting in better-organized scripts and mitigating redundant efforts.
Swift Onboarding and Issue Responsiveness:
The drastic cut in onboarding time stands out, reduced from a week to just 10 minutes. This efficiency enables the MedTech team to induct new members swiftly, allowing them to contribute in a fraction of the previous time. Collaborating with Datacoves has enhanced the team's responsiveness to new requests and defects, with quality assurance issues now addressed within days, a marked improvement from the previous weeks-long timeframe.
Conclusion
By adopting Datacoves, J&J was able to realize the full potential of dbt while lowering the total cost of ownership and speeding up their time to market. The collaboration has enabled the team to optimize their data management practices, ensure data security, and embrace modern development best practices, resulting in delivering value with data.
Related Case Studies
Accelerating dbt Core for Enterprise Data Platform
Building Insightly's Data Architecture Without a Team and Budget
