Let’s Do This builds an ETL pipeline and uses Amazon Redshift to perform data analytics

Overview

Let’s Do This, a prominent technology company connecting people with endurance sports events, faced a challenge in effectively utilizing their event data for analytics. To address this, they partnered with Lambert Labs to implement an automated ETL pipeline using Amazon Redshift. This solution enabled Let’s Do This to integrate their event data, stored in MongoDB Atlas, with their existing analytics data, allowing them to gain valuable insights and drive business improvements.

Opportunity / Customer Challenge

Let’s Do This faced a challenge in effectively leveraging their event data for analytics. While their use of MongoDB provided the flexibility needed to handle the unstructured nature of event data and accommodate a rapidly evolving data schema, it created a disconnect with their existing SQL based analytics.

The company’s analytics team relied heavily on Redshift, a data warehouse optimized for SQL queries. This meant that the event data stored in MongoDB had to be combined with the data in Redshift to provide a comprehensive view for analysis. Let’s Do This needed to extract, transform, and load (ETL) the event data from MongoDB into Redshift in an automated, reliable, and scalable manner. This process had to be implemented efficiently and at a low cost. They required a solution that would bridge the gap between their event data and their analytics platform, allowing their team to efficiently analyze event data alongside other key business metrics.

Solution

To address Let’s Do This’s challenge of integrating event data from MongoDB with their analytics platform on Redshift, Lambert Labs implemented a solution that balanced efficiency and customization.

To expedite the process and respect budget constraints, a third-party data integration provider, Stitch, was utilized. Stitch provided an off-the-shelf solution for the general task of extracting and loading the NoSQL data from MongoDB into Redshift. This included handling some data schema conversion, which saved significant development time and resources that would have been required to build this functionality in-house.

However, while Stitch handled the core data movement, it had limitations in delivering the data in the precise format required by the analytics team. To tailor the results to the exact specifications of the client and optimize the data for analysis, custom SQL queries were necessary. These custom SQL queries were implemented within a serverless AWS Lambda function. This Lambda function was then scheduled to run at specific intervals using Amazon EventBridge, automating the execution of the custom SQL and ensuring the data in Redshift was precisely formatted and up-to-date.

We turned to Lambert Labs when we needed help with our data ingestion and ETL processes. Work was always completed professionally, and they helped make our data pipelines more efficient. Their team really go above and beyond to ensure targets are met on time and I can wholeheartedly recommend them to anyone. (Sam Browne, CEO, Let’s Do This)

Outcome

The implemented solution successfully provided the Let’s Do This data analytics team with the necessary data tables in Redshift, effectively bridging the gap between their event data and their analytics platform. This integration eliminated the previous siloing of event data and unlocked significant potential for data-driven improvements across the organization.

With access to comprehensive data in Redshift, Let’s Do This gained the ability to enhance various aspects of their business. This included improvements in marketing effectiveness, more accurate tracking of key performance indicators (KPIs), enhanced business intelligence, better understanding of market trends, and deeper insights into consumer behavior.

The project’s completion opened up new avenues for analysis and optimization, enabling Let’s Do This to use their data more effectively in order to drive business growth and improve their services.

While it’s possible to do everything in AWS, we took a practical approach and combined the most relevant parts of AWS with other third party cloud solutions. (George Lambert, Founder & CEO, Lambert Labs)

About Let’s Do This

Let’s Do This is a technology company that connects people with endurance sports events. They operate a platform that makes it easy for users to discover and sign up for a wide variety of events, including running races, triathlons, and cycling events. Let’s Do This is a graduate of the Y Combinator startup accelerator and has gained significant recognition and funding, establishing itself as a key player in the endurance sports industry.