Medium
Model Distillation is back in the news!!!
Model Distillation is back in the news! And for all the wrong reasons…just to be clear, model distillation as such is not a nefarious technology. It is a specific way of fine-tuning especially smaller models based on the output from larger models.
In short, Model Distillation allows you to leverage the outputs of a large model to fine-tune a smaller model, enabling it to achieve similar performance on a specific task.
This process can significantly reduce both cost and latency, as smaller models are typically more efficient.
The natural way of doing this within an organisation is to capture all the good inference inputs/outputs and organise it as training data. So it is in most cases not general, open use but domain specific use.
But obviously it can be scaled to distill not a small model, but a comparable large model.
So model distillation can be used efficiently in organisations, I wrote a year about this approach and was amazing how little it was being used…from my view at least.
A few months ago Anthropic released a statement where they accused three Chinese AI labs of conducting industrial-scale distillation attacks on Claude, extracting its capabilities through millions of fraudulent API calls.
So the very technique that democratises AI is now being weaponised to steal it.
So with all the praise being sung about Chinese models, is worth noting how existing US tech were leveraged.
Below are two links, one explaining in simple terms how model distillation works…and a second explaining how it can be used to steal the “shape” or behaviour of a model.
