What I Learned Building a WGAN-GP for Human Face Generation
Insights into training stability, Wasserstein loss, gradient penalty implementation in TensorFlow, and evaluating generative quality using FID and Inception Score.
Introduction & Technical Context
Standard Deep Convolutional GANs (DCGANs) are notoriously difficult to train, frequently suffering from mode collapse and vanishing gradients caused by Jensen-Shannon divergence saturation. While working on synthetic human face generation in TensorFlow, I implemented Wasserstein GAN with Gradient Penalty (WGAN-GP) to achieve stable adversarial training dynamics.
1. The Math Behind Wasserstein Distance & Gradient Penalty
Vanilla GANs optimize Jensen-Shannon (JS) divergence, which suffers from vanishing gradients when generator and discriminator distributions do not overlap. WGAN uses the Earth Mover's (Wasserstein-1) distance. WGAN-GP replaces unstable weight clipping with a gradient penalty term enforced along random interpolations between real and generated samples.
2. Custom TensorFlow Gradient Penalty Implementation
Implementing WGAN-GP in TensorFlow requires custom training loops using tf.GradientTape to record generator and critic passes separately, calculate sample interpolations, and compute gradients of the critic with respect to the interpolated input.
import tensorflow as tf
@tf.function
def gradient_penalty(critic, real_images, fake_images):
batch_size = tf.shape(real_images)[0]
alpha = tf.random.uniform([batch_size, 1, 1, 1], 0.0, 1.0)
interpolated = real_images + alpha * (fake_images - real_images)
with tf.GradientTape() as tape:
tape.watch(interpolated)
pred = critic(interpolated, training=True)
grads = tape.gradient(pred, [interpolated])[0]
norm = tf.sqrt(tf.reduce_sum(tf.square(grads), axis=[1, 2, 3]) + 1e-12)
gp = tf.reduce_mean((norm - 1.0) ** 2)
return gp3. Evaluating Generative Quality with FID & Inception Score
Visual inspection alone is insufficient for evaluating GAN performance. Fréchet Inception Distance (FID) measures the distance between feature representations extracted from a pre-trained Inception-v3 network for real versus synthetic faces. Achieving an FID of ~60.8 on custom-curated face samples provided quantitative feedback during hyperparameter tuning.
Key Engineering Takeaways
- Gradient Penalty eliminates mode collapse and stabilizes loss curves across long training runs.
- Critic loss in WGAN correlates directly with visual image quality, unlike vanilla GAN discriminator loss.
- Quantitative metrics like FID are essential for objective model checkpoint selection.
Sujan K S — AI/ML Engineer