"For many students who enter the pit of machine learning, mathematics may be a difficult part. The author introduces the mathematical background needed to build machine learning products or conduct machine learning research, as well as the valuable experience and suggestions from machine learning engineers, researchers and teachers, and provides many courses and book resources.
At present, it is not completely clear what level of mathematics is required to start learning machine learning, especially for those who have not studied mathematics or statistics in school.
The goal of this article is to propose the mathematical background needed to build a machine learning product or conduct academic research on machine learning. These suggestions come from the dialogue with machine learning engineers, researchers and educators, as well as the author's experience in machine learning research and industry.
In order to construct the prerequisite of the required mathematics level, this paper first puts forward different thinking modes and strategies, so that readers can contact mathematics education outside the traditional classroom. Then, the paper outlines the specific background required for different types of machine learning projects, as these disciplines range from high school level statistical and calculus knowledge to the latest development of probability graph model (PGM). In the end, I hope readers can feel about mathematics learning. You need to make effective use of this knowledge in your machine learning project, no matter what it is!
First of all, I recognize that the learning of learning styles / frameworks / resources, learners' personal needs and goals may be unique. Welcome to put forward your opinions on HN!
Some experiences on mathematics anxiety
Facts have proved that many people, including engineers, are afraid of mathematics. First of all, I want to talk about the topic of "being good at mathematics".
In fact, people who are good at mathematics will have a lot of practical experience in using mathematics. As a result, they will find it common to get into trouble in mathematics. Recent research shows that learners' mentality (rather than innate ability) is the main predictor of a person's ability to learn mathematics.
It should be clear that it takes time and energy to achieve this habitual good state, but this is certainly not a natural ability. The rest of this article will help readers determine the required level of mathematical foundation and outline strategies for how to establish this level.
Getting started: math and code
As a soft prerequisite, we need readers to master some basic knowledge of Linear Algebra / matrix operation (to avoid confusion in symbolic representation) and have a preliminary understanding of probability theory. In addition, we encourage you to master basic programming skills as a tool for learning mathematics in a given context. After that, you can adjust your focus according to the type of project you are interested in.
How to study mathematics outside school
I believe that the best way to learn mathematics is to study wholeheartedly (that is, as a student). Without this all day learning environment, you may not master the teaching structure in the academic classroom and obtain (positive) pressure from peers and available resources.
If you want to study mathematics outside school, I suggest organizing study groups or lunches and study seminars as an important resource to obtain learning motivation. In the research laboratory, reading groups can also be organized. In terms of learning structure, your reading group can browse the chapters of the textbook, hold regular discussion lectures, open up channels, and hold Q & A activities.
Here, culture plays an important role. This "extra" learning should be encouraged to avoid forgetting the knowledge in the complex affairs day after day. In fact, despite the short-term costs of this approach, building a peer driven learning environment can make you more efficient in the long term.
Math and code
Mathematics and code are highly integrated in the machine learning workflow. Code is usually constructed directly by mathematical intuition. Code even shares some symbols and syntax with mathematics. In fact, the framework of modern data Science (such as numpy) makes the readable code converted from mathematical operations (such as matrix / vector product) more intuitive and efficient.
The author encourages readers to use code as a way to consolidate learning. Both mathematics and code depend on the accuracy of understanding and representation. For example, practicing the manual implementation of loss functions or optimization algorithms can be a good way to truly understand the basic concepts.
Here, let's give a specific example of learning mathematics through code: back propagation of relu activation in neural network (yes, it can be done through tensorflow / pytorch! See the link for details https://medium.com/ @karpathy/yes-you-should-understand-backprop-e2f06eab496b）。 Back propagation (BP) is an efficient technique for computing gradients based on calculus chain rules. To use the chain rule under this setting, we multiply the upstream derivative by the gradient of relu.
First, we visualize the activation of relu, as shown in the figure:
In order to calculate the gradient (intuitively, the slope), a piecewise function can be visualized, expressed as an index function as follows:
Numpy provides us with useful and intuitive syntax. Our activation function (blue curve) can be explained in the code, where x is the input and relu is the output:
relu = np.maximum(x, 0)
Then there is the gradient (red curve in the figure), where grad represents the upstream gradient:
grad[x < 0] = 0
Without first deriving the gradient yourself, the meaning of this line of code may not be so obvious. In our code, all values in the upstream gradient (Grad) are set to 0 for all elements that meet the conditions, [H < 0]. Mathematically, this is actually equivalent to a piecewise representation of the relu gradient. When multiplied by the upstream gradient, it will return any value less than 0 to zero!
As can be seen above, through our basic understanding of calculus, we can clearly think about the code. Here is an example of the completion of the neural network implementation: https://pytorch.org/tutorials/beginner/pytorch_ with_ examples.html
Mathematical knowledge required to build machine learning products
In writing this section, I consulted machine learning engineers to determine how to use mathematics most effectively to debug the system. The following example of a question is an engineer's response from a mathematical perspective. If you have never seen it before, it doesn't matter. I hope this chapter can provide some appropriate background for the special issues you are interested in.
What clustering methods should I use to display high-dimensional customer data?
Methods: PCA and tsne
How should I calibrate the threshold for "blocking" fraudulent user transactions?
Method: probabilistic calibration
How to correctly describe the deviation of satellite data in different regions of the world（ Silicon Valley and Alaska)
Methods: the research questions were made public. For example, the equalization of population informatics?
Usually, statistical tools and linear algebra can be applied to every problem in some way. However, to get a satisfactory answer, we usually need specific methods in a specific field. In this case, how should you determine what math knowledge you need to learn?
Define system
The resources we can use now are very rich (for example, scikit learn for data analysis and keras for in-depth learning), which helps readers start writing code to model the system. When using these resources, you can try to answer the following questions:
1. What is the input / output of the system?
2. How should data be prepared to adapt to the system?
3. How to build functional or planning data to help summarize the model?
4. How to define reasonable objectives for the questions raised?
You may be surprised that it may be difficult to define the system! After that, the works required for pipeline construction are also very important. In other words, building machine learning products requires a lot of heavy work, which does not require an in-depth mathematical background.
More information: • best practices for ML engineering by Martin zinkevich, research scientist at Google
Learn math on demand
First, look at the workflow of machine learning. You may find that you get stuck in some steps, especially during debugging. Do you know what to look for when you are trapped? How reasonable is your weight? Why can't your model be integrated with a specific loss definition? What is the correct way to measure the success of the model? At this time, it may be helpful to make assumptions about the data, constrain optimization in different ways or try different algorithms.
Generally, you will find that there are mathematical intuitions (such as how to select loss function and evaluation index) in the process of modeling / debugging, which may help to make informed engineering decisions. These are your opportunities to learn!
Rachel Thomas from fast.ai is a supporter of this "learning on demand" approach. When educating students, she found that it was more important for students with deep learning to be excited about these materials. After establishing interest, mathematics education becomes to check and fill gaps on demand.
More information:
Course: computational linear algebra fast.ai
YouTube:3blue1brown: Essence ofLinear AlgebraandCalculus
Textbook: linear algebra done right by Axler
Textbook: elements of statistical learning by Tibshirani et al
Course: Stanford's cs229 (machine learning) course notes
Mathematical knowledge required for machine learning research
The author now wants to describe which mathematical ways of thinking are useful for research-oriented work in machine learning. The pessimistic view on machine learning research is reflected in plug and play systems, in which the model will invest more computing power to squeeze out higher performance. In some populations, researchers still doubt empirical methods (such as some deep learning methods) that lack mathematical rigor.
It is worrying that the future research system may be based on existing systems and assumptions, which will not expand our basic understanding of the field. Researchers need to provide original resources and build new infrastructure modules that can be used to obtain new ideas and methods to achieve practical goals. For example, it may be necessary to rethink the construction modules such as convolutional neural network for image classification, as Geoff Hinton, the "father of machine learning", said in his recent paper on capsule networks（ Paper link: https://arxiv.org/pdf/1710.09829v1.pdf ）
Next, we need to ask basic questions. This requires "deep understanding" in mathematics, which Michael Nelson, the author of deep learning, calls "interesting exploration". This process involves thousands of hours, often "stuck by problems", constantly asking questions and changing views when exploring new problems“ "Interesting exploration" enables scientists to ask profound and insightful questions, not just a combination of simple ideas / architectures.
Of course, it is obvious that you still can't learn everything when studying machine learning! To correctly carry out "interesting exploration" needs to be carried out appropriately and follow their own interests, rather than giving priority to the most popular new results.
Machine learning research is a very rich research field, and there are urgent problems in fairness, interpretability and accessibility. In fact, in all scientific disciplines, basic thinking is not an on-demand process, but requires patience to think with the breadth of advanced mathematical framework and solve key problems.
More information: • blog: do SWEs need mathematics? By Keith Devlin • reddit post: qualifications of an AI researcher • blog: how to read mathematics by Shai Simonson and Fernando gouvea • paper: nipsandicml and other conference papers
• article: a mathematician's lamentby Paul lockhart1
Democratize machine learning research
I hope I don't describe "studying mathematics" too deeply, because the viewpoint constructed by mathematics should be presented in an intuitive form! Unfortunately, many machine learning papers are still full of complex and inconsistent terms, which makes the key information difficult to distinguish. As a student, you can translate concentrated papers into digestible and understandable views through blogs, tweets, etc., which is also doing good for yourself and this field. You can even take distill.pub as an example, which is a publication dedicated to providing a clear explanation of machine learning research results. In other words, it will be very helpful if we can turn the mystery of technical thought into a means of "interesting exploration"!
Finally, some conclusions and suggestions are given
In short, I hope this article can provide a starting point for readers to consider the mathematics education problems related to machine learning.
Different questions require different degrees of intuitive views, and I encourage readers to first figure out what the goal is.
If you want to build machine learning products, please find peers and learning groups through questions, and stimulate your learning through in-depth study of the ultimate goal.
In the research field, having a deep mathematical foundation can provide you with rich tools to promote the development of machine learning by proposing new infrastructure modules.
Generally speaking, mathematics (especially the mathematics in research papers) may be daunting, but being stuck by difficult problems in learning is an important part of the learning process., Read the full text, the original title: [Hacker News hottest tutorial] necessary mathematical knowledge for machine learning
Source: [micro signal: AI]_ Era, WeChat official account: new smart yuan] welcome to add attention! Please indicate the source of the article“
Our other product:
