Skip to content

Data Skew and Small Files: Why Your Spark Job Waits on One Task

19 May 2026·3 min read
pysparkdatabricksdata-engineeringperformancespark

A stage has 200 tasks. The median finishes in a few seconds and the summary metrics look boring. The stage has still been running for forty minutes because one task hasn't finished.

That's skew. The partitioning put a wildly disproportionate share of the rows on a handful of partitions, usually because a join or groupBy key is hot. Typical culprits are a null customer ID, a default country code or one giant account that owns half the events. Every row with that key hashes to the same partition, so one executor core grinds through it while the rest of the cluster sits idle. Averages don't show it because hundreds of normal tasks dominate them.

The usual reaction is to scale up the cluster. That does nothing here, since the straggler can only ever use one core. Look at the max column in the task summary instead of the median. If max is an order of magnitude above the median, count rows per key before you touch anything else.

Small files

This one is related but runs the other way. Skew is too much data in too few partitions. Small files come from a high-parallelism write where each of hundreds or thousands of tasks emits its own little file, often just a few kilobytes or megabytes.

In a lakehouse that compounds fast. Each file is an object in cloud storage that every read has to list and open. Delta Lake also tracks each one in the transaction log. Query planning, metadata handling and scan time end up growing with file count rather than data size. A table appended to every few minutes by a streaming job or a frequent batch can collect an absurd number of tiny files in a week. Downstream readers pay for that on every query.

The two problems feed each other. Someone runs repartition(2000) to spread out a skewed join and the write produces 2000 files. Someone else runs coalesce(1) to avoid small files and drags the whole upstream computation onto a single task with no parallelism left.

Fixing it by hand

The manual toolkit is well known.

  • Large table joined to a small one: use a broadcast join with F.broadcast(small_df). The small side ships to every executor and the shuffle disappears, so skew in the big table's keys stops mattering.
  • Big table joined to big table on a hot key: salt it. Append a random suffix (say 0 to N) to the key on the skewed side and explode the other side into N copies. The hot key now spreads across N partitions. It works, at the price of a bigger join and uglier code.
  • Writes: repartition() by a sensible column or count before writing. That beats both extremes from the previous section.

Check what AQE already did

Before writing any of that, check the platform. Modern Databricks runtimes (and Spark 3.2+) turn on Adaptive Query Execution by default. AQE uses real shuffle statistics at runtime to coalesce tiny shuffle partitions into reasonably sized ones. It converts a sort-merge join into a broadcast join when one side turns out to be small after filtering. It also splits skewed partitions in sort-merge and shuffled hash joins into smaller sub-partitions automatically, which amounts to salting that you didn't have to write.

It has limits. AQE won't fix skew in aggregations or window functions. Its skew detection only kicks in once a partition is both several times the median size and above an absolute threshold, so moderate skew can slip past it.

Storage has its own version of this. Delta's optimized writes and auto compaction bin-pack small files into larger ones without hand-tuning every writer. A scheduled OPTIMIZE does the same job, as does predictive optimization on Unity Catalog managed tables.

So open the SQL tab in the Spark UI before you salt a key and see whether AQE already split the skewed partition. If it did, skip the workaround. If the hot key sits in a groupBy or a window function, the salting is still your job.