Lessons Learned After 8.5 Years of ML
than eight years that I have spent with machine learning. The more I get to know this field the more I am astounded by how diverse it is. When I started my studies of machine learning, the only thing I back then knew about it was the classic machine learning techniques, such as KNN or clustering algorithms. Now, with more than eight years of further experience, I think that the field is so broad that no single person can understand all of this.
However, I think that there are still some lessons that are applicable regardless of the field one actually is doing machine learning research or machine learning practices in. Frequently, I take the opportunity of another half year of progress in my machine learning journey to step back for a moment and look at the previous years. What have I learned? Which lessons seem to persist?
In this edition of my lessons learned articles I look back at the previous eight years and try to distill lessons that are persisting in coming up over and over again. This time I found that making progress in machine learning, or any field, really, often comes down to the following five things. They are patience, discipline, optimism, good projects, and good teams. I will give more details about these in the remainder of this article.
Patience
Nobody is born an expert, in no field. If one is not born with genius-grade capacity — and even then –, being patient will work wonders. I frequently like to compare the progress in machine learning with the progress one makes in sportive activities. Often times as a beginner you will fail very much, very often in the beginning; in the early days and months of the new activity.
Take learning the handstand, for example. Until you have mastered the balance to hold your weight overhead, and built the strength in the first place, it takes quite a while. And learning to do so you will also encounter a lot of failures. Probably you will see no progress at all in the early days.
With machine learning, it is the same. It will take a couple of iterations on a project until it is good enough. Along the way, your work will be rejected and criticed, until it is deemed good enough. This happened to me as well, of course. A paper of mine has been rejected four times over two years until I finally got it accepted at an A* conference.
During these times, I had to resubmit the paper over and over again, and redo the story, the experiment, almost everything. I would be lying would I say that I was always positive. No, from time to time I would have been more than happy to drop the paper and focus on other things. But — patience. I focused on doing what I could do at that moments, and that was redoing and resubmitting. And then waiting.
Now it is accepted, but had I not been patient enough, then it would certainly NOT be published now.
Optimism
This brings me to the next lesson. You need to cultivate healthy optimism (or, healthy ignorance). When working on a project, such as a research paper or a deployment, you will inevitably face obstacles. Persisting is necessary, but you need more than to just persist.
I think that you need to be silently optimistic about your work. Others can say their things, and you can listen, but in the end it is your work. As long as you trust the progress, you are generally fine. Or, to be more precise: as long as you trust the process for the majority of times, then you are fine.
Discipline
Again, this neatly leads to the next lesson underlying my past years: discipline. In our daily lives, there surely are things that are more engaging at the moment. Well, sitting down to read another paper is interesting, but checking Twitter or YouTube is more engaging. But, that will not bring you forward, or even actively pull you back.
To make progress, you need to ignore deflections and daily distractions. You will need the quality that has been praised in all disciplines across most of humanity: discipline. When learning for an exam, you do so, regardless of the circumstances. When writing a lab report, you do so, regarless of your peers going partying. When drafting a thesis, you do so regardless of others going on vacation.
Having nearly written a report does not count. Neither does a nearly written thesis. Only if you can focus and be disciplined about it, you can move forward. My “secret” to this is to put the important things first, day after day. I try to have an 80% adherence to this schedule to make it realistically workable. I found it to work quite well.
Projects
This is a lesson specific to people working towards a (doctoral) thesis: you need a sufficiently good project to work on. It might not matter too much if you are intelligent enough. What might matter more is that your topic is sufficiently niche/new AND stable enough so that others care about it, and that you can utilize your strengths on it.
Some years ago, I had a colleague who studied emergent capabilities of LLMs. This is quite an interesting topic, and I came across it at ICLR 2024 in Vienna, where I saw some posters on it. My colleague was quite skilled in his transformer model knowledge and programming skills. However, the field was TOO new, too unstable. The models were progressing too fast, and he could not get a grip on the topic. In the end, it burned him out and he switched to other things. (He’s doing fine now!)
Teams
I reserved the most important lesson for the end. It often matters more who you work with than what you work at. (If you tick both, meaning you have a good group AND your work is cool, then you are especially lucky). If you are good at what you do, but your environment does not let you put this to use, then you are in the wrong place. If you cannot make mistakes, then you are in the wrong place. If you fear showing up, then you are in the wrong place.
There is this terminology of a team’s effectiveness. You (and the team) first need to be in a comfort zone. This means being well and working together well. Only afterwards can you advance to the high-output zone, where you produce good work under pressure. Make sure that you are there as well.
Closing thoughts
Looking over the lessons, none of them is machine learning-unique. Rather, these are the same old things that have defined good progress in humanity over the last millenia. This is not bad, quite on the contrary: if a technology would overhaul everything that has accompanied us since back then, then we would be lost. Thus, it is comforting to see the classic lessons reappear:
- Be patient
- Be optimistic
- Be disciplined
and also the more passion-driven lessons
- Work on something where you can utilize your strengths
- Choose good and healthy teams
Nothing new, and all relevant.
