Data Ops Migration Engineer
TapTalent.ai
Date: 2 weeks ago
City: Petaling Jaya
Contract type: Full time
Role Overview
Seeking a skilled Data Migration Engineer with 4-8 years of experience in data migration, data setup, and data systems development. The ideal candidate should have expertise in Spark, SQL, and Java(Scala) for data processing, reporting, and development. Strong knowledge of data architecture and semantic layer development is required, along with experience in regression testing and cutover activities in large-scale migrations.
Key Responsibilities:Spark:
Seeking a skilled Data Migration Engineer with 4-8 years of experience in data migration, data setup, and data systems development. The ideal candidate should have expertise in Spark, SQL, and Java(Scala) for data processing, reporting, and development. Strong knowledge of data architecture and semantic layer development is required, along with experience in regression testing and cutover activities in large-scale migrations.
Key Responsibilities:Spark:
- Develop and optimize Spark-based data processing for large-scale ETL and analytics.
- Implement Spark SQL, DataFrames, RDDs, and Streaming for efficient data transformations.
- Optimize Spark job performance, tuning memory, partitioning, and execution plans.
- Handle real-time and batch data processing using Spark Streaming or structured streaming.
- Write and optimize complex SQL queries for data extraction, transformation, and aggregation.
- Work on query performance tuning, indexing, and partitioning for optimized execution.
- Develop and manage stored procedures, functions, and views for data operations.
- Ensure data consistency, integrity, and security in relational database systems.
- Java is essential as Scala runs on the JVM, making JVM tuning critical for Spark-based workloads.
- Develop data processing applications using Scala (running on JVM) and Java-based backend services.
- Optimize JVM performance, memory management, and garbage collection to enhance Spark job execution.
- Utilize Scala’s functional programming features for efficient data transformations in Spark.
- Implement multithreading, concurrency, and parallel processing in Java for high-performance applications.
How to apply
To apply for this job you need to authorize on our website. If you don't have an account yet, please register.
Post a resumeSimilar jobs
QC Engineer
SNS Network,
Petaling Jaya
MYR 3,800
-
MYR 4,500
per month
22 hours ago
Job ResponsibilityQuality Assurance & ControlPerform inspections and quality checks on ICT components, including servers, switches, racks, cabling, wireless APs, and related infrastructure.Validate network and system configurations (e.g., IP addressing, VLAN, cabling, rack layouts) against approved designs.Conduct and document FAT (Factory Acceptance Test), SAT (Site Acceptance Test), and UAT (User Acceptance Test).Review installation workmanship according to standards (e.g., TIA/EIA, ISO/IEC, local...
Account Executive (Luxury and Premium Frame) - Central
EssilorLuxottica,
Petaling Jaya
1 day ago
If you’ve worn a pair of glasses, we’ve already met.We are a global leader in the design, manufacture, and distribution of ophthalmic lenses, frames, and sunglasses. We offer our industry stakeholders in over 150 countries access to a global platform of high-quality vision care products (such as the Essilor brand, with Varilux, Crizal, Eyezen, Stellest and Transitions), iconic brands that...
Senior Associate - Purchase to Pay (VMD/Payment)
DHL Express,
Petaling Jaya
1 day ago
Key ActivitiesTo provide accurate and timely processing execution of accounts payable related activities as per SLA.Account Payable ProcessesProcessing of payment for internal and 3rd party vendors via the automatic / manual method and cheque issuance.The payment activities are carried out based on schedules / ad-hoc.Process inter-company payments.Answering vendor / business partners queriesFollow-up with business partners for insufficient information / clarification...
