Listen carefully, I shall say zis only once… 📡
For one of our international clients, we are building a modern cloud data platform processing many bytes of streaming, audience, advertising, and content metadata in near real-time.
Project Overview:
- Building a scalable data platform for a global company supporting streaming, analytics, and real-time audience insights.
- Developing cloud-native data platforms based on AWS, event-driven architecture, and distributed processing (other cloud is also highly desired)
- Implementing batch and streaming pipelines for advertising, telemetry, and content metadata workloads.
- Designing solutions using AWS (S3, MWAA, Sagemaker, MSK, DynamoDB, Kinesis, ECS, Datahub, CloudWatch, EMR; Python, Pyspark, Spark Streaming, Shell;
tools: Jenkins - Driving reliability, observability, and governance across a multi-region data platform.
- Collaborating with Data Engineers, ML teams, Platform Engineers, and people asking “is the dashboard ready yet?” every 17 minutes 😉
Your Responsibilities:
- Designing and developing end-to-end pipelines using Python, PySpark
- Building and optimizing AWS-based solutions leveraging S3, Sagemaker, EMR, ECS, Kinesis, CloudWatch, MWAA etc.
- Implementing data ingestion pipelines for both batch and streaming workloads with focus on throughput, latency, and fault tolerance.
- Managing the data lifecycle: schema evolution, partitioning, metadata, lineage, and retention policies.
- Monitoring, troubleshooting, and performance tuning the platform (aka classic “it works on my cluster” debugging).
Requirements:
- Commercial experience as a Data Engineer / Data Platform Engineer / Cloud Engineer etc.
- Strong hands-on expertise with AWS and modern cloud data ecosystems.
- Practical experience with PySpark, EMR, and Python.
- Solid understanding of distributed systems, event-driven architectures, and large-scale data processing.
- Experience with Jenkins.
- Ability to work with large-scale datasets and awareness that “temporary workaround” usually means “see you again in nearest future” 😉
