About Hassan Ammar

  • Academic Level Degree Bachelor
  • Industry development
  • Viewed 76



  • 2019 - Present
    S&P Global

    Lead software development engineer

    S&P Global Marketplace
    Designed and Implemented end to end data delivery pipelines for various third party and S&P propriety datasets which includes the following tools and processes:
    o Acquiring data files from s3 buckets and processing using NIFI.
    o Developing Data Ingestion framework using Apache Spark.
    o Implementation of Monitoring and alerting framework using ELK Stack.
    o Implemented Snowflake (SnowPipe) with AWS S3 (IAM User, IAM Role) for continuous integration.
    o Implemented near Real-Time CDC using Snowflake Streams and Tasks.
    o Implemented end-to-end ETL pipeline in NiFi, Databricks & Snowflake (S3, Delta Lake).
    o Developing REST API in java Spring boot.
    o Designed and Implemented a generic Data Ingestion framework using Apache Spark.
    o Designed and Implemented Data Profiling framework using Amazon Deequ to ensure accurate Data Models
    Implementation Data ingestion pipeline
    • Implemented end-to-end Apache NIFI flow to process files. Starting from acquiring files, applying various validations, and then storing them in S3 buckets.
    • Implementation of NIFI registry for version controlling the flows.
    • Invoking Databricks jobs from NIFI using Databricks APIs.
    • Implementation of spark jobs to Transform data, saving data into delta lake and delivering to Snowflake.
    • Setting up Spark clusters in Databricks.
    • Transformation of PL-SQL scripts to Spark.
    • Writing Scala code in IntelliJ for maintaining proper code structures and implemented Azure CI/CD for deployment.

Leave Your Review

  • Overall Rating 0