This project explores the intersection of serverless computing and stream processing frameworks to handle large-scale data streams with minimal operational overhead. Students will experiment with various platforms and orchestration strategies to ensure efficient scaling, low latency, and robust fault tolerance without relying on dedicated servers.
Focus
Investigating how serverless computing models (e.g., AWS Lambda, Azure Functions) can be leveraged to build scalable, cost-efficient, and event-driven stream processing solutions. Emphasis on elasticity, resource management, and fault tolerance in serverless environments.
Tasks
- Serverless Integration: Evaluate and integrate stream processing frameworks (e.g., Apache Flink, Kafka Streams) with serverless platforms.
- Elastic Scaling: Design and test auto-scaling policies that adapt to fluctuating workloads in real time.
- Cost Optimization: Investigate cost-performance trade-offs in serverless deployments for continuous data processing.
- Fault Tolerance and State Management: Explore strategies for stateful stream processing in stateless serverless functions.
Required Skills
Theoretical Skills:
- Distributed Systems Concepts
- Cloud Computing and Serverless Paradigms
- Performance Modeling and Cost Analysis
Practical Skills:
- Experience with AWS Lambda, Azure Functions, or Google Cloud Functions
- Familiarity with Apache Flink/Kafka Streams
- Scripting and DevOps (CI/CD, Infrastructure as Code)
