Python Data Engineering

Job Description

• Data Pipeline Development
  • Build and maintain ETL/ELT pipelines using Python
  • Ingest data from multiple sources (APIs, databases, files, streaming systems)
  • Optimize pipelines for performance and scalability
  • Clean, transform, and validate raw datasets
  • Handle structured and unstructured dataUse frameworks like:
    • Pandas
    • PySpark
    • Dask
    • Database & Data Warehousing
    • SQL (PostgreSQL, MySQL, SQL Server)
    • NoSQL (MongoDB, Cassandra)
    • Design schemas and optimize queries
    • Build data warehouses using:
      • Snowflake
      • Redshift
      • BigQuery
      • Big Data Technologies
      • Apache Spark
        • Hadoop
        • Process large-scale datasets efficiently
        • Workflow Orchestration
        • Apache Airflow
        • Cloud Platforms
        • Work on cloud environments:
        • AWS (S3, Glue, Lambda, EMR)
        • Azure (Data Factory, Synapse)
        • GCP (Dataflow, BigQuery)
        • Data Quality & Monitoring
        • Implement data validation checks
        • Monitor pipeline failures and fix bugs
        • Ensure data reliability and integrity