Could Machine Learning Practitioners Prove Deep Math Conjectures?

Many of us have solid foundations in math or have an interest in learning more, and are passionate about solving difficult problems during our free time. Of course, most of us are not professional mathematicians, but we may bring some value to help solve some of the most challenging mathematical conjectures, especially the ones that can be stated in rather simple words. In my opinion, the less math-trained you are (up to some extent), the more likely you could come up with original, creative solutions. Not that we could end up proving the Riemann hypothesis or other problems of the same caliber and popularity: the short answer is no. But we might think of a different path, a potential new approach to tackle these problems, and discover new theories, models and techniques along the way, some applicable to data analysis and real business problems. And sharing our ideas with professional mathematicians could have benefits for them and for us. Working on these problems during our leisure time could also benefit our machine learning career, if anything. In this article, I elaborate on these various points.

The less math you learned, the more creative you could be

Of course, this is true only up to some extent. You need to know much more than just high school math. When I started my PhD studies and asked my mentor if I should attend some classes or learn material that I knew was missing in my education, his answer was no: he said that the more you learn, the more you can get stuck in one particular way of thinking, and it can hurt creativity. That said, you still need to know a minimum, and these days it is very easy to self-learn advanced math by reading articles, using tools such as OEIS or Wolfram Alpha  (Mathematica) and posting questions on websites such as MathOverflow (see my profile and posted questions here), which are frequented by professional, research-level mathematicians. The drawback by not reading the classics (you should read them) is that you are bound to re-invent the wheel time and over, though in my case, that’s the best way I learn new things.

Professionals with a background in physics, computer science, probability theory, pure math, or quantitative finance, may have a competitive advantage. Most importantly, you need to be passionate about your own private research, have a lot of modesty, perseverance, and patience as you fill face many disappointments, and not expect fame or financial rewards – in short, not any different than starting a PhD program. Some companies like Google may allow you to work on pet projects, and experimental research in number theory geared towards applications, may fit the bill. After all, some of the people who computed trillions of digits of the number Pi (and analyzed them) did it during their tenure at Google, and in the process contributed to the development of high performance computing. Some of them also contributed to deepen the field of number theory.

In my case, it was never my goal to prove any big conjecture. I stumbled time and over upon them while working on otherwise un-related math projects. It peeked my interest, and over time, I spent a lot of energy trying to understand the depth of these conjectures and why they may be true. And I got more and more interested in trying to pierce their mystery. This is true for the Riemann hypothesis (RH), a tantalizing conjecture with many implications if true, and relatively easy to understand. Even quantum physicists have worked on it, and obtained promising results. I know I will never prove RH, but if I can find a new direction to prove it, that is all I am asking for. Then I will work with mathematicians who know much more than I do, if my scenario for a proof is worth exploring, and enroll them to work on my foundations (likely to involved brand new math). The hope is that they can finish a work that I started myself, but that I can not complete due to my somewhat limited mathematical knowledge.

In the end, many top mathematicians made stellar discoveries in their thirties, out-performing their peers that were 30 years older despite the fact that their knowledge was limited because of their young age. This is another example that if you know too much, it might not necessarily help you.

Note that to get a job, “the less you know, the better” does not work, as employers expect you to know everything that is needed to work properly in their company.  You can and should continue to learn a lot on the job, but you must master the basics just to be offered a job, and to be able to keep it. 

What I learned from working on these math projects: the benefits

To begin with, not being affiliated with a professional research lab or the academia has some benefits: you don’t have to publish, you choose your research project yourself, you work at your own pace (it better be much faster than in the academia), you don’t have to face politics, and you don’t have to teach. Yet you have access to similar resources (computing power, literature, and so on). You can even teach if you want to; in my case I don’t really teach, but I write a lot of tutorials to get more people interested in the subject, and probably self-published books in the future, which could become a source of revenue. My math questions on MathOverflow get a lot of criticism and some great answers too, which serves as peer-review, and they even point me to some literature that I should read, as well as new, state-of-the-art yet unpublished research results. On occasions, I correspond with well known university professors, which further helps me not going in the wrong direction. 

The top benefits I’ve found working on these problems is the incredible opportunities it offers to hone your machine learning skills. The biggest data sets I ever worked on come from these math projects. It allows you to test and benchmark various statistical models, discover new probability distributions with applications to real-world problems (see this example), new visualizations (see here), develop new statistical tests of randomness and new probabilistic games (see here), and even discover interesting math theory, sometimes truly original: for instance complex random variables with applications (see here), lattice points distribution in the infinite-dimensional simplex (yet unpublished), or advanced matrix algebra asymptotics (infinite matrices, yet unpublished)  and a new type of Dirichlet functions. Still, 90% of my research never gets published. I only share peer-reviewed, usually new results. The rest goes to garbage, which is always the case when you do research. For those interested, much of what I wrote and that I consider worth sharing, can be found in the math section, here.