[Paper Review] UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation

Note: This is a review of the paper “UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation” (IEEE TMI 2020 (online 2019), arXiv:1912.05074). The original nested-architecture version, “UNet++: A Nested U-Net Architecture for Medical Image Segmentation”, appeared at DLMIA 2018.

Code: official implementation.

For a Korean version of this review, please visit the OUTTA AI Tech Blog.

Why I Read This Paper

When studying medical image segmentation, U-Net is like a mountain you cannot avoid. But if you’ve ever wondered, “Why does it have to be U-Net?” or “Why do Skip Connections just simply concatenate?”, UNet++ might be the answer.

This paper solved the structural limitation of U-Net (the Semantic Gap) with a clever method called Nested Dense Skip Connections. I found it particularly interesting that Deep Supervision allows for model pruning, which is a very useful tip when model lightweighting is needed in actual clinical settings.

---

Introduction

U-Net has been the dominant architecture for medical image segmentation. However, it has two main limitations:

1. Unknown Optimal Depth: The optimal network depth varies depending on the task difficulty and data size.
2. Restrictive Skip Connections: The simple skip connections in U-Net only combine features at the same resolution, which may have a “semantic gap” between the encoder and decoder.

UNet++ addresses these issues by introducing Nested Dense Skip Connections. It effectively integrates U-Nets of varying depths into a single unified architecture, allowing the model to capture multiscale features more effectively.

Figure 1: The UNet++ architecture. It consists of an encoder and decoder connected by a series of nested, dense skip pathways (green and blue lines). Deep supervision (red lines) allows for model pruning (from Fig. 1 of the paper).

Context / Related Work

UNet++ extends U-Net (plain same-resolution skip connections) and its wider variant Wide U-Net (more filters). Its nested dense pathways borrow the dense-connectivity idea from DenseNet, and its auxiliary losses come from deeply-supervised networks. The contribution is combining these to bridge the encoder–decoder semantic gap within one prunable architecture.

---

Methods

1. Nested Dense Skip Connections

Unlike U-Net, which directly connects the encoder feature map to the corresponding decoder layer, UNet++ introduces intermediate convolution blocks on the skip pathways.

• Dense Connections: For a node $X^{i,j}$ with $j>0$, the inputs are all earlier nodes on the same skip pathway, $X^{i,0},\ldots,X^{i,j-1}$, plus the upsampled output $X^{i+1,j-1}$ from the next deeper level. The concatenated tensor is then processed by the convolution block $H$. The dense connectivity is therefore inside each redesigned skip pathway, not a claim that every node in the whole network connects to every other node.
• Semantic Gap Reduction: These intermediate blocks help bridge the semantic gap between the low-level encoder features and high-level decoder features, making the optimization easier.

2. Deep Supervision

UNet++ employs Deep Supervision, where auxiliary loss functions are attached to the output of each decoder branch (at different depths).

• Training: The model is trained to minimize the loss at all levels simultaneously ($L^1, L^2, L^3, L^4$).
• Benefits: This improves gradient flow and acts as a regularizer.

3. Model Pruning

Thanks to deep supervision, UNet++ can be pruned at inference time.

• Fast Mode: If a shallower sub-network yields sufficient accuracy, the deeper parts of the network can be removed during inference.
• Efficiency: This allows users to trade off between performance and inference speed without retraining the model.

Results

The expanded TMI paper evaluates six applications: five 2D tasks (EM, cell, nuclei, brain tumor, and liver) and one genuinely volumetric task. For the latter, the authors train VNet++ on 1,012 LIDC-IDRI lung-nodule crops of size $64\times64\times64$. Table IV reports IoU $71.17\pm4.53$ for V-Net and $77.05\pm2.42$ for VNet++ with deep supervision, so the redesigned connections were validated in a 3D setting as well as on 2D images.

Method	Lung Nodule (IoU)	Colon Polyp (IoU)	Liver (IoU)	Cell Nuclei (IoU)
U-Net	71.47	30.08	76.62	90.77
Wide U-Net	73.38	30.14	76.58	90.92
UNet++ (w/ DS)	77.21	32.12	82.90	92.52

Table 1: Segmentation performance comparison (IoU). UNet++ consistently outperforms U-Net and Wide U-Net. These four-dataset numbers come from the original DLMIA 2018 conference version (Table 3); the IEEE TMI 2020 redesign reviewed here expands the experiments, so treat this as the predecessor’s comparison for the same architecture.

Figure 2: Qualitative comparison. UNet++ produces segmentation masks that are closer to the Ground Truth compared to U-Net and Wide U-Net, especially for fine details (from Fig. 3 of the paper).

---

Conclusion & Insight

UNet++’s clearest contribution is reducing the semantic gap of skip connections with nested dense pathways, while deep supervision adds training stability and inference-time pruning.

Strengths

• Nested dense skip pathways measurably reduce the encoder–decoder semantic gap and consistently improve IoU over U-Net and Wide U-Net (Table 1).
• Deep supervision is dual-purpose: a regularizer during training and a knob to trade accuracy vs. speed via pruning at inference — handy for clinical deployment.

Limitations

• More parameters, memory, and compute than plain U-Net; the IoU gains are large on some datasets (Liver, ~+6) but modest on others (Cell Nuclei, ~+1.8).
• The 3D evidence is limited to one lung-nodule CT task using fixed $64^3$ crops and a V-Net backbone. That supports volumetric feasibility, but not broad generalization to full-volume segmentation, other 3D organs, or other backbones.
• Deep supervision adds loss-weighting choices ($L^1$–$L^4$) to tune, and pruning trades accuracy for speed without a principled stopping rule.

Open Questions / My Take

The most transferable idea is “one architecture you can prune to the right depth.” The paper includes one 3D VNet++ experiment; the next question is whether that result holds with modern backbones and across larger, full-volume 3D datasets.