BE in Information technology

Lead software development engineer

S&P Global Marketplace
Designed and Implemented end to end data delivery pipelines for various third party and S&P propriety datasets which includes the following tools and processes:
o Acquiring data files from s3 buckets and processing using NIFI.
o Developing Data Ingestion framework using Apache Spark.
o Implementation of Monitoring and alerting framework using ELK Stack.
o Implemented Snowflake (SnowPipe) with AWS S3 (IAM User, IAM Role) for continuous integration.
o Implemented near Real-Time CDC using Snowflake Streams and Tasks.
o Implemented end-to-end ETL pipeline in NiFi, Databricks & Snowflake (S3, Delta Lake).
o Developing REST API in java Spring boot.
o Designed and Implemented a generic Data Ingestion framework using Apache Spark.
o Designed and Implemented Data Profiling framework using Amazon Deequ to ensure accurate Data Models
Projects
Implementation Data ingestion pipeline
• Implemented end-to-end Apache NIFI flow to process files. Starting from acquiring files, applying various validations, and then storing them in S3 buckets.
• Implementation of NIFI registry for version controlling the flows.
• Invoking Databricks jobs from NIFI using Databricks APIs.
• Implementation of spark jobs to Transform data, saving data into delta lake and delivering to Snowflake.
• Setting up Spark clusters in Databricks.
• Transformation of PL-SQL scripts to Spark.
• Writing Scala code in IntelliJ for maintaining proper code structures and implemented Azure CI/CD for deployment.