9 Apache Spark Interview Questions and Answers

Apache Spark interviews often test more than API knowledge. Interviewers want to know whether you understand how Spark turns code into distributed work, where performance bottlenecks come from, and how to diagnose them systematically.

A framework for answering Spark questions

Explain the concept, describe what Spark does internally, identify the trade-off, and finish with a practical example or diagnostic step. For performance questions, use the Spark UI and execution plan to support your reasoning instead of guessing.

1. Explain Spark's execution model from an action to task execution.

Short answer: An action triggers execution. Spark analyzes and optimizes the logical plan, creates a physical plan, divides work into stages at shuffle boundaries, and schedules tasks across executors.

For DataFrame and SQL workloads, the process typically looks like this:

1. Transformations build a plan. Operations such as filter(), select(), and join() are generally lazy; they describe what should happen without immediately processing all the data.
2. An action triggers a job. Examples include count(), collect(), show(), and writing output.
3. Catalyst optimizes the query. Spark analyzes the logical plan, applies optimizations such as predicate and projection pushdown where applicable, and chooses a physical execution strategy.
4. The DAG scheduler creates stages. Operations that require data redistribution, commonly shuffles, usually create stage boundaries.
5. The task scheduler launches tasks. A stage is split into tasks, commonly one per partition, and tasks are run on executors.
6. Executors process partitions. They read data, apply operators, exchange shuffle data when required, and report status and metrics.

Example

result = (
    spark.read.parquet("/data/orders")
    .filter("status = 'COMPLETE'")
    .groupBy("customer_id")
    .sum("amount")
)

result.write.parquet("/output/customer_totals")

The filter and aggregation are transformations. The write triggers execution. The aggregation commonly requires a shuffle so records with the same customer key can be combined.

Interview tip: Mention that exact physical operators and stage boundaries depend on the query, data source, statistics, adaptive query execution settings, and Spark version.

2. What is the difference between repartition() and coalesce()?

Both change the number of partitions, but they have different data-movement behavior.

Aspect	repartition(n)	coalesce(n)
Typical purpose	Increase or decrease partitions	Usually reduce partitions
Shuffle	Typically causes a full shuffle	Usually avoids a full shuffle when reducing partitions
Distribution	Can redistribute rows, optionally by key	Combines existing partitions without a full redistribution
Typical use	Balance data or set up parallelism before expensive work	Reduce small output files after filtering

# Redistribute data across 200 partitions
balanced = df.repartition(200)

# Reduce partition count before a small output
smaller_output = df.coalesce(20)

# Repartition by key to distribute matching keys together
by_customer = df.repartition(200, "customer_id")

Important nuance: coalesce() can produce uneven partition sizes, especially when reducing aggressively. If you need better balancing, a shuffle-based repartition may be worth the extra cost. Avoid reducing to one partition for large datasets unless a single output partition is genuinely required.

3. When would you use cache() vs. persist()?

Short answer: Use caching when an expensive result will be reused. cache() is a convenient default; persist() lets you choose a storage level.

• cache(): Uses the API's default storage level. For modern Spark DataFrames, the default is generally memory-and-disk storage; check the behavior for your Spark version and API.
• persist(level): Lets you specify a storage level, such as memory-only or memory-and-disk, depending on the API and language.
• Lazy materialization: Calling either method marks the dataset for storage; the cache is populated when an action computes it.
• Eviction and recomputation: Cached data is not a permanent backup. Partitions may be evicted or recomputed if lost.

from pyspark import StorageLevel

prepared = expensive_transform(source_df).persist(
    StorageLevel.MEMORY_AND_DISK
)

prepared.count()  # Materialize the persisted result

# Reuse prepared in multiple downstream actions
prepared.groupBy("region").count().show()
prepared.filter("amount > 1000").count()

prepared.unpersist()

Use it when: a DataFrame is expensive to rebuild and is reused multiple times. Avoid caching one-use intermediate data, very large datasets that compete with useful execution memory, or data that is cheap to recompute.

Common interview mistake: Caching does not automatically make a job faster. It consumes storage resources and can slow other work if the cached data is not reused enough.

4. How do you identify and resolve data skew?

Data skew occurs when some partitions receive substantially more data or work than others. A few slow tasks can hold up an entire stage even when most tasks have finished.

How to identify it:

• In the Spark UI, compare task duration, shuffle read/write size, input records, and spill metrics within the same stage.
• Look for a small number of tasks that run much longer or process far more data than the rest.
• Inspect key frequency and null/default keys; one highly frequent key can dominate a join or aggregation.
• Check whether skew appears in a join, group-by, window operation, or a partitioning step.

Ways to resolve it:

1. Enable Adaptive Query Execution (AQE). Spark can detect skewed shuffle partitions and split them for supported operations when the relevant settings are enabled.
2. Broadcast a genuinely small table. This can remove the large shuffle on the join side, if the small table safely fits the relevant memory budget.
3. Salt hot keys. Add a controlled salt value to distribute a hot key across multiple partitions, then remove the salt or aggregate the partial results correctly.
4. Pre-aggregate. Reduce rows before a join when the business logic allows it.
5. Handle exceptional keys separately. For example, process a known hot key through a separate path and union the results.

Key point: Adding more executors may not fix skew. If one task owns most of the data for a hot key, extra executors can remain underused while that task runs.

5. What causes a shuffle, and why can it be expensive?

A shuffle redistributes data across partitions, often across executors, so that records can be processed together according to a key or partitioning requirement.

Common shuffle-causing operations include:

• Joins when the chosen strategy requires redistributing data.
• groupBy(), groupByKey(), and many aggregations.
• distinct(), repartition(), and many global sorting operations.
• Window functions or other operations whose required partitioning differs from the current layout.

Shuffles are expensive because Spark may serialize and write intermediate data, transfer it over the network, read it back, sort or merge it, and spill to disk when memory is insufficient. They also introduce stage boundaries and can amplify the impact of skew.

How to reduce unnecessary shuffle cost:

• Filter early and select only the columns needed downstream.
• Use broadcast joins when one side is small enough.
• Reuse compatible partitioning where it helps, instead of repartitioning repeatedly.
• Use partial aggregation when supported and appropriate.
• Inspect shuffle read/write, spill, and task imbalance in the Spark UI.

Interview nuance: Not every join causes a shuffle. A broadcast hash join can avoid shuffling the large side, and Spark may use existing partitioning or other physical strategies.

6. How do broadcast joins work?

A broadcast join sends a small relation to executors so each executor can join its local partitions of the large relation against that smaller copy. For a broadcast hash join, this can avoid shuffling the large dataset by join key.

from pyspark.sql.functions import broadcast

orders = spark.read.parquet("/data/orders")
countries = spark.read.parquet("/data/countries")

result = orders.join(
    broadcast(countries),
    on="country_code",
    how="left"
)

Good candidates: small dimension tables joined with large fact tables, such as country codes, product categories, or a compact customer mapping.

Benefits:

• Can avoid shuffling the large side of the join.
• Often reduces network traffic and stage time.
• Can improve star-schema analytical workloads.

Risks and trade-offs: The broadcast relation must be small enough to distribute and hold safely in executor memory. Broadcasting a table that is too large can cause memory pressure, serialization overhead, or executor failures. Use table statistics and runtime measurements; a hint should not replace validation.

7. How would you optimize a Spark job that runs slowly despite sufficient cluster resources?

Start with evidence, not by adding more executors. Sufficient cluster capacity does not guarantee efficient partitioning, low shuffle cost, or balanced work.

1. Establish the bottleneck. Identify the slow stage and compare task durations, input size, shuffle metrics, spill, garbage collection, and executor utilization in the Spark UI.
2. Inspect the physical plan. Use explain("formatted") and look for unexpected exchanges, sort operations, repeated scans, or an unsuitable join strategy.
3. Check parallelism and partition sizes. Too few partitions can underuse the cluster; too many tiny partitions can create scheduling overhead. Partition count should reflect data size and workload.
4. Investigate skew. Compare the slowest tasks with the median. Check hot keys and uneven input sizes.
5. Review join strategy. Broadcast small dimensions when appropriate, verify statistics, and evaluate AQE settings.
6. Reduce data early. Push filters toward the source when possible and avoid reading unused columns.
7. Check I/O and file layout. Many tiny files, poor partition pruning, remote storage latency, and inefficient formats can dominate runtime.
8. Review memory and spill. More cluster memory may not help if the plan forces large sorts, skewed tasks, or unnecessary materialization.
9. Benchmark one change at a time. Compare runtime, cost, shuffle volume, and output correctness on representative data.

df.explain("formatted")

# Example: project and filter before an expensive join
filtered = (
    fact_df
    .filter("event_date >= '2026-01-01'")
    .select("customer_id", "event_date", "amount")
)

result = filtered.join(
    broadcast(dim_customer.select("customer_id", "segment")),
    on="customer_id",
    how="left"
)

Strong interview answer: Explain the evidence you would collect, the hypothesis it supports, the smallest change you would test, and how you would verify the result.

8. How does Spark recover from executor failures?

Spark can often recover by rescheduling failed tasks and recomputing lost intermediate data from lineage. Executors are workers; the driver coordinates the application and tracks job execution.

• Task retry: A failed task may be retried, subject to retry limits and the failure type.
• Lineage-based recomputation: If a cached partition or shuffle output is lost, Spark may recompute the missing data from earlier transformations or rerun relevant work.
• Shuffle recovery: Depending on deployment and configuration, lost shuffle files may require map tasks to run again. External shuffle services or shuffle tracking can affect behavior.
• Driver failure: Recovery depends on the cluster manager, deployment mode, and configured driver supervision; it is not guaranteed automatically in every setup.
• Application-level correctness: Retries do not make arbitrary external side effects exactly-once. Writes and external calls must be designed for safe retries, idempotency, or transactional semantics.

Example

If an executor holding cached partitions fails, Spark can recompute the lost partitions from lineage. If a task writes to an external system before failing, however, a retry could repeat the write unless the sink handles retries safely.

Interview tip: Distinguish task-level recovery from driver or application recovery, and distinguish recomputing data from guaranteeing exactly-once output.

9. How would you process a dataset larger than the available memory?

Spark is designed to process distributed datasets that do not fit in the memory of one machine. The goal is to partition the data and process it across executors while controlling the amount of data each task needs to hold at once.

1. Use distributed DataFrame or SQL operations. Avoid collecting the full dataset to the driver with collect() or converting it into a local in-memory structure.
2. Read incrementally or in partitions. Use columnar formats such as Parquet, partition pruning, predicate pushdown, and only the columns required.
3. Choose sensible partition sizes. Avoid oversized partitions that create memory pressure and tiny partitions that create excessive scheduling overhead.
4. Prefer spill-capable operations. Spark can spill certain shuffle, sort, and aggregation data to disk, though spilling can significantly slow execution.
5. Use storage levels thoughtfully. Persist only reusable intermediates and choose an appropriate memory/disk strategy.
6. Control expensive operations. Avoid unnecessary global sorts, huge broadcasts, unbounded groupings, and wide joins without a plan for their resource needs.
7. Write results incrementally. Use distributed writes to a suitable storage system instead of bringing all output to the driver.

large_df = (
    spark.read.parquet("gs://my-bucket/events/")
    .filter("event_date >= '2026-01-01'")
    .select("event_id", "customer_id", "event_date")
)

summary = large_df.groupBy("event_date").count()

summary.write.mode("overwrite").parquet(
    "gs://my-bucket/output/daily_counts/"
)

Important: Disk spilling is a safety mechanism, not a substitute for a good plan. If one key or partition is much larger than the others, address skew and partition sizing rather than assuming Spark will make the workload efficient automatically.

Quick Revision Cheat Sheet

Topic	Remember
Execution model	Action → optimized plan → stages → tasks → executors
repartition vs coalesce	Repartition shuffles for redistribution; coalesce usually reduces partitions without a full shuffle
cache vs persist	Cache is the convenience method; persist supports explicit storage levels
Data skew	Look for outlier tasks and hot keys; consider AQE, salting, pre-aggregation, or broadcast joins
Shuffle	Network transfer, serialization, disk I/O, sorting, and stage boundaries
Broadcast join	Useful for a small relation; validate memory size
Slow Spark job	Inspect Spark UI, physical plan, skew, shuffle, spills, I/O, and partition sizing
Executor failure	Task retries and lineage can recover work; external side effects still need safe retry design
Data larger than memory	Distributed processing, partitioning, pruning, spill-aware operations, and no full collect to driver

Scenario-Based Interview Practice

Try answering each scenario out loud before expanding the suggested approach.

Scenario 1: One task takes 25 minutes while most finish in 2 minutes. What do you check?

Compare task input and shuffle sizes, duration, spill, and GC in the Spark UI. Investigate skewed keys and uneven partition sizes. Then test a targeted remedy such as AQE skew handling, salting, pre-aggregation, or a different join strategy. Confirm correctness and compare the slowest-task duration after the change.

Scenario 2: A join takes most of the job runtime, but one table is small. What would you do?

Check the physical plan and table size, then test a broadcast join if the smaller table can safely fit in executor memory. Review the impact on shuffle volume and runtime. Do not broadcast solely because a table has few rows; row width and serialized size matter too.

Scenario 3: The job has many executors but low CPU utilization. Why?

Check whether there are too few tasks or partitions, whether the job is waiting on remote I/O or shuffle fetches, and whether a small number of skewed tasks are limiting progress. Also inspect scheduler delays, source throughput, and stage dependencies before increasing the cluster again.

Scenario 4: A Spark job fails after writing some output and then retries. How do you prevent duplicates?

Determine the output sink's commit and retry semantics. Use idempotent writes, deterministic keys, transactional or atomic commit mechanisms where supported, and a clear strategy for overwrites or incremental updates. Spark task retry alone does not guarantee exactly-once external side effects.

Chapter Summary

Strong Spark engineers understand how execution plans become distributed tasks and know how to diagnose the expensive parts of a job.

• Use the Spark UI and physical plan to locate bottlenecks.
• Understand shuffle boundaries, partitioning, joins, and skew.
• Cache only data that benefits from reuse.
• Design for executor retries without assuming all external side effects are exactly-once.
• Keep processing distributed; do not collect huge datasets to the driver.

Practice Challenge

Design a Spark pipeline that joins a multi-terabyte events table with a small customer dimension, aggregates events by customer, and writes partitioned Parquet output. Explain your join strategy, partition sizing, skew checks, shuffle reduction, failure handling, and how you would validate performance in the Spark UI.