ADF Data Engineer

@ Neshent Technologies
Neshent Technologiesneshenttechnologies.com

ADF Data Engineer

Pasadena, Texas
Posted 1 day ago

About the job

Our company focuses on data engineering solutions. This role involves designing and optimizing data pipelines using Azure Data Factory and Databricks for large-scale data processing.

Requirements

  • Azure Data Factory experience
  • Azure Databricks expertise
  • Advanced Python or Scala
  • Expert SQL skills
  • Knowledge of data storage technologies

Qualifications

  • 8-10 years professional experience
  • 3+ years Azure experience
  • Strong coding standards
  • Experience with CI/CD

Full job description

Experience : 8–10 years overall, with 3+ years of Azure experience.

Required Skills
  • Strong hands-on experience with Azure Data Factory (ADF) — pipelines, data flows, triggers, integration runtimes, parameterization, error handling, and retries.
  • Strong experience with Azure Databricks — Notebooks, Jobs/Workflows, PySpark/Scala, Delta Lake, cluster configuration, and optimization.
  • Advanced Python or Scala with strong coding standards, modularization, and unit testing (pytest).
  • Expert SQL — CTEs, window functions, query tuning, and large datasets.
  • Experience with ADLS Gen2, Delta Lake, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.
  • Experience with Git, branching strategies, and CI/CD for ADF/Databricks.
  • Exposure to Infrastructure as Code (IaC) is a plus.
Roles and Responsibilities
  • Coding & Development: Design, develop, and optimize ADF/Databricks data pipelines and solutions.
  • UI Testing: Perform functional/UI testing, data validation, and troubleshooting.
  • Stakeholder Management: Collaborate with stakeholders, understand requirements, and communicate project status/issues effectively.
Show full description