Senior Data Engineer
Experience: 6+ Years
Location: WFH / Remote (India)
Open Positions: Multiple
Qualification: B.Tech / M.Tech / MCA orequivalent
Work Timings: 1:30 PM IST – 10:30 PM IST
Functional Area: Data Engineering
About the Role
Weare seeking an experienced Senior Data Engineer to design, develop, and maintain scalable, reliable, andhigh-performance data platforms and pipelines that support businessintelligence, analytics, and data-driven decision-making.
Theideal candidate will have strong hands-on expertise in Python,Apache Spark, Apache Airflow, Amazon Redshift, SQL, and modern data engineeringpractices, along with mandatory experience in the Healthcare domain. The candidate should haveexperience building end-to-end ETL/ELT pipelines, processing large-scaledatasets, implementing data quality frameworks, and supporting enterpriseanalytics and reporting needs.
This rolerequires a technically strong professional who can collaborate with businessstakeholders, data analysts, architects, and application teams to deliverrobust and scalable data solutions.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows for structured and unstructured data.
- Build and optimize data ingestion, transformation, and data integration processes using Python and Apache Spark.
- Develop, schedule, and monitor workflows using Apache Airflow.
- Design and manage enterprise data warehouse solutions using Amazon Redshift.
- Implement data models, data marts, and reporting datasets to support analytics and business intelligence requirements.
- Optimize data processing jobs and database performance for scalability and efficiency.
- Work with large healthcare datasets, ensuring data accuracy, consistency, and compliance with industry standards.
- Collaborate with business stakeholders to understand reporting and data requirements.
- Implement data quality checks, validation frameworks, and monitoring processes.
- Troubleshoot data pipeline issues and perform root cause analysis.
- Ensure data security, governance, and compliance requirements are adhered to.
- Participate in code reviews, development best practices, and continuous improvement initiatives.
- Create and maintain technical documentation, data flow diagrams, and data mapping documents.
- Work closely with cross-functional teams including Data Science, Analytics, Product, and Engineering teams.
Skills, Knowledge & Experience
Mandatory Skills
- 6+ years of experience in Data Engineering, Big Data Engineering, or related roles.
- Mandatory experience working in the Healthcare domain.
- Strong hands-on experience in Python development for data engineering solutions.
- Strong hands-on experience with Apache Spark (PySpark) for large-scale data processing.
- Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
- Strong experience designing, developing, and optimizing solutions on Amazon Redshift.
- Advanced SQL skills with experience in query tuning and performance optimization.
- Experience building ETL/ELT pipelines and modern data integration frameworks.
- Strong understanding of data warehousing concepts, dimensional modelling, and data architecture.
- Experience handling large-scale datasets and complex data transformation requirements.
- Strong debugging, troubleshooting, and performance tuning skills.
- Experience working with cloud-based data platforms and services (AWS or Azure) is preferred.
- Excellent analytical and problem-solving skills.
- Strong communication and stakeholder management capabilities.
Preferred Skills
- Experience with AWS data services such as S3, Glue, EMR, Athena, Lambda, or Kinesis
- Experience with Azure Data Factory, Azure Databricks, ADLS, Synapse Analytics, or other Azure data services
- Exposure to Databricks, Snowflake, or modern Lakehouse architectures
- Experience with CI/CD for Data Engineering pipelines
- Knowledge of Healthcare data standards such as HL7, FHIR, EHR/EMR, Claims, Provider, Member, or Clinical Data
- Experience with data governance, data quality, and compliance frameworks
- Experience with Git and DevOps best practices for data engineering
This jobdescription is not intended to cover or contain a comprehensive listing of allresponsibilities, duties, or activities that are required. Responsibilities,duties, and/or activities may change, or new ones may be added at any time withor without notice.
If you are amotivated professional with a passion for delivering impactful solutions, we'dlove to hear from you. Apply today to be part of a dynamic and forward-thinkingteam at NextGen Invent.
Originally posted on Himalayas
