Virtual Adversarial Training — The Boundary May Not Live Here
A worked example: the same eight labels, one new loss
The stage is unchanged — Part 1’s exact two-moons data and its five random 8-label draws — but the model is new: a small neural network (2 → 16 → 2), the track’s first since Part 1’s cautionary tale about softmax confidence.
DEMO 1 --- The baseline: a neural network and 8 labels
draw: 74.2% 72.8% 72.6% 85.0% 82.2% mean 77.4%
Trained on eight points alone, the network does what unregularised networks do: fits the labels perfectly, parks its boundary wherever those eight points permit — usually straight through both moons — and reports 99% confidence while doing it.
Add the attack
Same network, same labels, plus the VAT penalty over all 500 points:
DEMO 2 --- The scoreboard: add the attack, evict the boundary
draw supervised only + VAT penalty
0 74.2% 90.8%
1 72.8% 97.4%
2 72.6% 98.0%
3 85.0% 93.8%
4 82.2% 97.2%
mean 77.4% 95.4%Mean prediction confidence across the pool:
supervised only: 0.99 (99% sure, boundary through the moons)
with VAT : 0.95 (hedges exactly where the data thins)
A +18-point mean jump, with every draw at 90% or better — including draw 1, the one that broke self-training (59.6%) and resisted tri-training (74.6%). That draw’s problem was always coverage: its labels left whole stretches of moon closer to the wrong frontier, and every committee method inherited the blind spot through its error estimates. VAT never estimates errors. The unlabelled points in those uncovered stretches simply forbid the boundary from crossing them, whichever side the labels happen to sit on. This is the strongest 8-label scoreboard of any inductive method in this track — only Part 5’s transductive label propagation (98.3%) sits higher, and VAT hands you a trained network at the end rather than answers for one fixed dataset.
The dial: how hard to shove
ε — the attack radius — is VAT’s one important knob, and it is wonderfully interpretable: the distance within which predictions must not change.
DEMO 3 --- The eps dial: how hard to shove
eps per-draw accuracies mean
0.05 85% 74% 93% 87% 85% 85.0%
0.2 91% 97% 98% 94% 97% 95.4%
0.5 92% 95% 95% 88% 93% 92.6%
1.0 84% 94% 87% 90% 78% 86.7%
2.0 49% 74% 80% 71% 77% 70.5%
A clean inverted U, and both ends fail for stated, geometric reasons. At ε = 0.05 the forbidden zone around each point is smaller than the spacing between points, so the boundary can still thread between them. At ε = 0.2 — roughly the data’s own noise scale — the only region not claimed by some point’s bubble is the gap itself: 95.4%. And at ε = 2.0 the attack radius spans the gap: smoothness now demands the same prediction on both moons, and the classes weld together (one draw collapses to 49%). The dial is the cluster assumption made quantitative: ε must be larger than the within-cluster spacing and smaller than the between-cluster gap. When no such ε exists, VAT has nothing to work with — which is exactly the condition under which the assumption itself is false.
One honest footnote from this article’s experiments: the literature typically pairs VAT with entropy minimisation, because a large model can satisfy smoothness by simply refusing to commit away from the labels (uniformly uncertain predictions are perfectly smooth). On this problem the cross-entropy on eight labelled points supplied all the commitment needed — we measured the EntMin variant at 91.0%, slightly below pure VAT — so the script ships the pure form, and the flatness escape-hatch stands as a warning for bigger models rather than a demonstrated failure here.
