Data Ops Migration Engineer
TapTalent.ai
Date: 12 hours ago
City: Petaling Jaya
Contract type: Full time
Role Overview
Seeking a skilled Data Migration Engineer with 4-8 years of experience in data migration, data setup, and data systems development. The ideal candidate should have expertise in Spark, SQL, and Java(Scala) for data processing, reporting, and development. Strong knowledge of data architecture and semantic layer development is required, along with experience in regression testing and cutover activities in large-scale migrations.
Key Responsibilities:Spark:
Seeking a skilled Data Migration Engineer with 4-8 years of experience in data migration, data setup, and data systems development. The ideal candidate should have expertise in Spark, SQL, and Java(Scala) for data processing, reporting, and development. Strong knowledge of data architecture and semantic layer development is required, along with experience in regression testing and cutover activities in large-scale migrations.
Key Responsibilities:Spark:
- Develop and optimize Spark-based data processing for large-scale ETL and analytics.
- Implement Spark SQL, DataFrames, RDDs, and Streaming for efficient data transformations.
- Optimize Spark job performance, tuning memory, partitioning, and execution plans.
- Handle real-time and batch data processing using Spark Streaming or structured streaming.
- Write and optimize complex SQL queries for data extraction, transformation, and aggregation.
- Work on query performance tuning, indexing, and partitioning for optimized execution.
- Develop and manage stored procedures, functions, and views for data operations.
- Ensure data consistency, integrity, and security in relational database systems.
- Java is essential as Scala runs on the JVM, making JVM tuning critical for Spark-based workloads.
- Develop data processing applications using Scala (running on JVM) and Java-based backend services.
- Optimize JVM performance, memory management, and garbage collection to enhance Spark job execution.
- Utilize Scala’s functional programming features for efficient data transformations in Spark.
- Implement multithreading, concurrency, and parallel processing in Java for high-performance applications.
How to apply
To apply for this job you need to authorize on our website. If you don't have an account yet, please register.
Post a resumeSimilar jobs
Senior Manager, Growth Strategy and Planning
Grab,
Petaling Jaya
15 hours ago
Company DescriptionAbout Grab and Our WorkplaceGrab is Southeast Asia's leading superapp. From getting your favourite meals delivered to helping you manage your finances and getting around town hassle-free, we've got your back with everything. In Grab, purpose gives us joy and habits build excellence, while harnessing the power of Technology and AI to deliver the mission of driving Southeast Asia...
Senior Executive, Customer Experience Training & Quality APAC
Warner Bros. Discovery,
Petaling Jaya
15 hours ago
Welcome to Warner Bros. Discovery… the stuff dreams are made of.Who We Are…When we say, “the stuff dreams are made of,” we’re not just referring to the world of wizards, dragons and superheroes, or even to the wonders of Planet Earth. Behind WBD’s vast portfolio of iconic content and beloved brands, are the storytellers bringing our characters to life, the...
One Accounting Analyst
SLB,
Petaling Jaya
18 hours ago
About UsWe are a global technology company, driving energy innovation for a balanced planet. Together, we create amazing technology that unlocks access to energy for the benefit of all.Our inclusive culture is the key to our success. We collaborate with our internal community of colleagues, alumni, and our valued external partners to support each other and achieve our goals. We...
