> there's only a 50% chance they can write down the chain rule
Why should I, though? I remember the concept from calculus. I know pytorch keeps track of the various stuff I do to a vector and calculates a gradient based on it. What more do I need to know when all I want to do is to play with applications, not implement backprop myself?
Certainly there's a lot you can do without understanding backprop - you can train pre-made architectures, you can put pre-made layers together to build your own architecture, you can tweak hyperparameters and improve your model's accuracy, and so on. But I also think you will eventually run into a problem that would be much easier to debug if you understand backprop. If your model isn't learning, and your tensorboard graphs show your gradient magnitude is through the roof, it'll be much easier to track that down if you have a strong conceptual model of how gradients are calculated and how they flow backwards through the network.
if you don't understand chain rule then you dont understand backprop, which means you do not really understand how deep learning works. at most you can follow recipes cook book style. it is kind of how one can make a website without a deep understanding of networking
This kind of argument can always go further by removing abstraction… you could argue that you don’t understand the chain rule without understanding the quotient and product rules which are used to prove the chain rule, and then once further with the epsilon-delta proofs required to get limits going for the derivative in the first place, and then continue until we reach some axioms.
I think you can accept that the chain rule is a thing, without understanding it, and then go further to understand its application to backprop.
Here I disagree with you pretty strongly. Once someone is comfortable with differentiable programming it's much more obvious how to build and optimize any type of model.
People should be more concerned about when to use derivatives, gradients, hessians, Laplace approximation etc rather than worry about the implementation details of these tools.
Abstraction can also aid depth of understanding. I know plenty of people who can implement backprop, but then don't understand how to estimate parameter uncertainty from the Hessian. The latter is much more important for general model building.
i am not sure what you are disagreeing with. chain rule is basic calculus that precedes understanding hessians. my argument is, if you can not understand what the chain rule is, you will not understand more complicated mathematics in ML. do you think i am wrong ?
EDIT: also uncertainty estimation is the stuff of probabalistic approach to ML. i would say that people who do probabalistic ML are quite mathematically capable (at least to my experience)
> chain rule is basic calculus that precedes understanding hessians.
It doesn't have to be that way. The hessian is an abstract idea and the chain rule and more specifically backpropagation are methods of computing the results for an abstract idea. When I want the hessian I want a matrix of second order partial derivatives, I'm not interested in how those are computed.
For a more concrete example, would you say that using the quantile function for the normal distribution requires you to be able to implement it from scratch?
There are many, very smart, very knowledgeable people that correctly use the normal quantile function (inverse CDF) every day for essential quantitative computation that have absolutely no idea how to implement the inverse error function (an essential part of the normal quantile). Would you say that you don't really know statistics if you can't do this? That a beginner must understand the implementation details of the inverse error function before making any claims about normal quantiles? I myself would absolutely need to pull up a copy of Numerical Recipes to do this. It would be, in my opinion, ludicrous to say that anyone wanting to write statistical code should understand and be able to implement the normal quantile function. Maybe in 1970 that was true, but we have software to abstract that out for us.
The same is becoming true of backprop. I can simply call jax.grad on my implementation of loss of the forward pass of the NN I'm interested in and get the gradient of that function, the same way I can call scipy.stats.norm.ppf to get that quantile for a normal. All that is important is that you understand what the quantile function of the normal distribution means for you to use it correctly, and again I suspect there are many practicing statisticians that don't know how to implement this.
And to give you a bit of context, my view on this has developed from working with many people who can pass a calculus exam and perform the necessarily steps to compute a derivative, but yet have almost no intuition about what a derivative means and how to use it and reason about it. Calculus historically focused on computation over intuition because that was what was needed to do practical work with calculus. Today the computation can take second place to the intuition because we have powerful tools that can take care of all the computation for you.
> Today the computation can take second place to the intuition because we have powerful tools that can take care of all the computation for you.
and that tool is backprop. if you do not understand what the chain rule is and what it is doing, that tool will be magic to you and you are blindly trusting its correctness. seeing that alot of risk is involved in using AI models in real life, blindly trusting your model is not a good approach
i agree that simply regurgitating rules of calculus is pointless to understanding. but thats definitely not what i mean when i talk about the need to understand the chain rule
ML is a mathematically intensive subject. there is no going around this fact
but that's my point. knowing how to compile a program does not make me a compiler engineer. in that sense feel free to use ML tools, but don't be fooled into thinking you will get a job as an ML engineer if you do not know what the chain rule is, or why we need to take a derivative in order to optimise a loss function. in fact, don't even be fooled into thinking you will get into a ML uni degree if you don't know what the chain rule is. i actually don't understand what is the problem. spend 10 minutes reading up on it and i am sure you will get it. i think an unwarranted phobia of mathematics is what is at play here
If by understand, you mean understand and not regurgitate it when asked as a trivia question - I agree with you. However, there are different interpretations of the chain rule.
Why should I, though? I remember the concept from calculus. I know pytorch keeps track of the various stuff I do to a vector and calculates a gradient based on it. What more do I need to know when all I want to do is to play with applications, not implement backprop myself?