Data Engineer

Responsibilities:

• Design, manage, and optimize ETL processes using Azure Data Factory for batch data processing
• Use Databricks and PySpark to process and analyze large datasets across distributed environments
• Create automation workflows and perform complex data manipulations using Python
• Write and optimize SQL queries to efficiently retrieve and process data from various sources
• Implement real-time data ingestion solutions using Azure Event Hub and Spark Streaming
• Manage and interact with NoSQL databases, particularly CosmosDB, to support scalable data storage and retrieval
• Prioritize and enhance batch data processing workflows, ensuring high performance and reliability
• Collaborate with cross-functional teams, provide insights, and maintain thorough documentation of all data processes 

Requirements:

• Minimum 3 years of experience in similar position
• English level: B2, C1
• Strong experience with Azure Data Factory for orchestrating ETL processes
• Proficiency in Databricks and PySpark for distributed data processing
• Advanced skills in Python for data manipulation and automation
• Solid knowledge of SQL for writing complex queries and optimizing data retrieval
• Experience with Azure Event Hub for real-time data ingestion
• Knowledge of CosmosDB for working with NoSQL databases
• Understanding of Spark Streaming for processing real-time data streams
• Batch processing experience is more critical than streaming processing for this role
• Familiarity with Azure Synapse is a plus, but not mandatory
• Good to Have: Knowledge of Azure DevOps for CI/CD pipeline automation, experience in the Aviation Industry, and familiarity with Copilot as a tool for AI-driven development assistance

Offer:

• Private medical care
• Co-financing for the sport card
• Training & learning opportunities
• Constant support of dedicated consultant
• Team-building events organized by DCG
• Employee referral program