Kaustav Maity

Kolkata, India

• Architected and delivered the end-to-end system design for an on-premises to AWS migration, moving critical workloads from Oracle to Amazon Aurora PostgreSQL, improving scalability and long-term maintainability.
• Executed large-scale database migration using AWS DMS and AWS File Gateway, ensuring zero data loss while handling high-volume historical and incremental data loads.
• Designed and implemented optimized ETL/ELT pipelines using AWS Glue and Amazon S3, reducing batch load times and improving pipeline reliability.
• Developed a custom observability dashboard using Python Flask, providing real-time visibility into data load metrics, failures, latency, and SLA adherence.
• Implemented parallel and asynchronous data loading using PostgreSQL dblink and aws_s3/aws_lambda extensions, increasing ingestion throughput and reducing processing latency.

Kolkata, India

• Owned and operated 30+ batch, streaming, and near real-time data pipelines processing millions of transactional records daily for a UK-based insurance client.
• Designed and implemented real-time streaming pipelines using Apache Kafka (via Amazon Kinesis) to ingest data into an Apache Iceberg–based data lake, enabling real-time dashboard updates and faster business insights.
• Optimized large-scale Spark workloads using caching, multithreading, and broadcast joins, achieving a 20% reduction in processing time for high-volume datasets.
• Led AWS migration initiatives and mentored three junior data engineers, improving delivery timelines, code quality, and production stability.

Remote (Freelance)

• Delivered expert-level academic and technical solutions in Python, Java, Statistics, and Data Analytics to a global learner base.