Using AWS Batch and Amazon Elastic Container Service to implement an AI-powered data transformation pipeline for Bat Conservation Trust
Who are Bat Conservation Trust?
The Bat Conservation Trust (BCT) is a non-profit organisation based in the United Kingdom that is dedicated to the conservation of bats and the habitats they reside in. They run a range of monitoring and research programmes to increase our understanding of bats, and engage in education and advocacy to raise awareness and support conservation efforts.
What were their requirements?
The Bat Conservation Trust required the development of a scalable and cost-effective audio processing pipeline based on an existing proof of concept. The existing system ran on non-scaling EC2 instances and needed reimplementation to scale to handle over a hundred terabytes of input data. For each individual audio recording, an AI classification task had to be processed to produce data to inform future conservation efforts.
Hear from BCT
Watch our case study video to hear first hand from Bat Conservation Trust how Lambert Labs helped them realise their goals.
Why Lambert Labs?
Lambert Labs was chosen for our experience with using AWS services to deploy secure and scalable solutions. Our work on previous projects with ETL tasks using AWS Lambda and AWS Fargate gave us valuable experience that we were able to draw on to design a robust solution for Bat Conservation Trust. Additionally, our experience in Python application development meant that we were able to pick up the existing Python codebase and rapidly iterate application code suitable for deployment to the cloud.
What we did
- Assisted in an application for the AWS Imagine Grant, which was successfully awarded to Bat Conservation Trust.
- Produced a highly scalable data transformation pipeline, reducing data processing times by weeks.
- Leveraged Spot instances on Fargate to optimise computation costs.
- Deployed a serverless technology stack to handle large volumes of data without incurring high costs during idle periods.
- Stored pipeline results in Amazon RDS Aurora Serverless.
If you would like to find out more about Lambert Labs, please feel free to read about our Python development services, AWS consulting services or AWS Well-Architected Framework Review services.
APN Case Study
Read more about our work with Bat Conservation Trust on the Amazon Partner Network website.
Read more