Big Data Engineer

Build and maintain ETL pipelines using Python and PySpark on AWS Glue and related platforms. 

Orchestrate workflows using AWS Step Functions and Lambda. - Implement messaging and event-driven integrations using SNS and SQS. Design and optimize storage and querying solutions in Amazon Redshift, RDS, Oracle and S3-based architectures. 

Write efficient SQL for transformations, validation, and reporting. - Integrate data from APIs and process structured and semi-structured JSON data. - Implement data quality checks, monitoring, and operational support processes. 

Participate in CI/CD and version control practices for deployment and release management. 

Collaborate with cross-functional teams to translate business requirements into technical solutions. 

8 years of software development experience across the appropriate platform. 

Strong hands-on experience with Python, PySpark, API’s and SQL. 

Experience with ETL/data pipeline development and Orchestration using Step functions / AirFlow .

 Working knowledge of AWS services including Glue, Lambda, Step Functions, Redshift, S3, SNS, and SQS. 

Experience with Athena, EMR, Kinesis, DynamoDB, or RDS. - Good Knowledge on CloudWatch, logging, and production support. 

Understanding of data warehousing, data lakes, Lake House and query optimization. 

Experience with GitLab/Terraform or similar and CI/CD workflows. 

Good understanding of using AI tools like Github Copilot or similar for code productivity .

Exposure to enterprise data lake or cloud migration initiatives. 

Have an eye to solving complex problems, great communication with stakeholders .

Have a good understanding of performance engineering of code pipelines and near real time systems - Good understanding on Agents and MCP".

A Bachelor’s Degree in Information Technology, Computer Science, Business Administration, or a related field is required.