What does A Big Data Engineer (Hadoop/Spark) do?
A Big Data Engineer (Hadoop/Spark) is responsible for building and operating large-scale batch and streaming data platforms on Hadoop, Spark, and related distributed systems for organizations processing petabyte-scale datasets.
Day-to-day responsibilities
- Writing and tuning Spark jobs
- Managing Hive/HDFS or cloud-native equivalents
- Optimizing cluster performance and job costs
- Supporting downstream analytics and ML teams that depend on the pipelines
Low demand+16% Apache Spark (PySpark/Scala)+10% Hadoop ecosystem (HDFS, Hive, YARN)+14% Kafka for streaming pipelines