Articles, Blog What is backpropagation really doing? | Deep learning, chapter 3 Related posts: Gradient descent, how neural networks learn | Deep learning, chapter 2 But what is a Neural Network? | Deep learning, chapter 1 Deep Learning – Computerphile Heroes of Deep Learning: Andrew Ng interviews Andrej Karpathy 3 blue 1 brown 3 brown 1 blue 3b1b 3brown1blue and blue brown deep learning machine learning Mathematics neural networks one the three three blue one brown three brown one blue you Post navigation Interactive Classroom 3 – Knowledge building with ICT, SwitzerlandLearning Outcomes Presentation 27 Comments Skylight July 1, 2019 at 12:44 pm Reply I remember seeing this a year back and not really understanding what is going on. I have recently started Andrew Ng's Machine Learning course and now this video feels a lot clearer 😀 timkolm2 July 3, 2019 at 5:34 pm Reply @3Blue1Brown Why was there no explanation on how the biases should be adjusted?Otherwise great series! Ryan Denziloe July 7, 2019 at 5:08 pm Reply The stochastic gradient descent analogy of the quickly stumbling drunk man is just perfect. Deepak July 8, 2019 at 6:00 pm Reply I like that pi. Raver July 11, 2019 at 2:42 am Reply I love these moments in maths or other sciences, where you have this one singular moment of pure enlightenment. You got me at 7:37 ….I got goose bumps when the first column of "+" appeared. This is just brilliant. Ajay Hemanth July 11, 2019 at 9:51 am Reply very good explanation … user 53503 July 13, 2019 at 1:50 pm Reply Love this channel mondlos July 16, 2019 at 9:27 pm Reply for non natives: "bang for your buck" = "effect" mondlos July 16, 2019 at 10:24 pm Reply 1. Gradient descent (step): You calculate the negative gradient of your multidimensional cost function at a random point and then move to the point where the (displacement) vector, i.e. the gradient vector is pointing to and calculate the gradient again. You do this until you find a point where the magnitude of the gradient vector is very small (approaching zero) – meaning that you do the gradient descent until you find a local minimum of the cost function. 2. impact on the result a.k.a. bang for the buck: focussing on a single input, the weights must be changed (nudged) by particular values proportional to the activation of the neurons they are referring to. The learning rate alpha is the proportionality factor. Let`s focus on the last layer. You take a single neuron and calculate the needed changes for the activations of the neurons in the previous layer. You do that for every neuron in this layer and add all these desired changes for every activation of the neurons in the previous layer. 3. backpropagation: it´s called backpropagation because you first calculate the weights of the last layer. Then you move on to the second last hidden layer until the last hidden layer. Awesome Kid July 23, 2019 at 4:30 pm Reply is a cost function the same as a loss function? - RedBlazerFlame - July 29, 2019 at 3:17 am Reply <!DOCTYPE html><html><head><title>What is backpropagation really doing? | Deep learning, chapter 3</title></head><body><div id="player" link="https://www.youtube.com/watch?v=Ilg3gGewQ5U"><p class="topic" id="plan"> … </p> … <p class="topic" id="Stochastic gradient descent">It will take a long time to get the average cost for all the data in your training set. So instead of calculating the average cost for everything in your dataset, you instead take a mini batch and calculate the cost. Sure it is less accurate, but you will get a significant computational speedup!</p>…</div> </body></html> 남재현 July 29, 2019 at 4:45 pm Reply 저거… 자막이 bias를 편견이라고 해석하네요… 그리고 여러 단어가 실제 문맥 맥락과 다른 의미로 번역되었어요… bfbfbfb bfbfbf July 30, 2019 at 9:49 am Reply какой ублюдский перевод субтитров, выблядок что это сделал – вандал, просто не вникая в тему забил в гугл транслит англ субтитры и все, выблядок смерти тебе Jyoti Prasad Pal July 30, 2019 at 3:54 pm Reply You are great…. Sarim Mehdi August 1, 2019 at 7:09 pm Reply Quite frustrating I have to end up here on this guy's channel. I don't like his video format (what's up with this weird music as if this is some super mysterious stuff? It actually turns away a lot of people and I don't think he realizes that he is actually making it look way more complicated than it should be). However, I don't see any proper video on backpropagation so I have to make use of this. But still unliking because of the poor presentation of rather simple ideas. Why do people not use a simple pen and paper approach as should be the case. Weird animations like these just scare a lot of people away. DowzerWTP72 August 6, 2019 at 11:01 pm Reply I'm forcing myself to watch all these videos before going on to write my own code to do this! I've got some awesome ideas that I can't wait to implement, plus in my next year of University, I will be doing computer vision and machine learning, so having this under my belt will be really great! I just wanna start programming though!! Vitalijs Fescenko August 7, 2019 at 2:53 pm Reply Not drunk, just stochastic Phil Myday August 9, 2019 at 6:56 am Reply Wonderful videos! You would have been my favorite teacher in education. generaldefence August 9, 2019 at 10:29 pm Reply I use Brave rewards on you man you're that worth it 😀 Sanwal Yousaf August 13, 2019 at 3:32 am Reply The song in the background reminds me of the theme song of CityVille, the Facebook game from 2010. Charlie Angkor August 14, 2019 at 8:50 pm Reply once I took the description of visual system by Hubel and Wiesel and programmed a neural network to simulate it and it was amazing. Then I used the tricks I learned from the brain to program a language cortex. That is even more amazing because I just flush down a language corpus without even telling it what encoding it is in or where the word boundaries are and it learns all the phrases words suffixes prefixes by itself. How do I figure out what I programmed when I dont understand the neural network theory at all? Like how its called and stuff? I find it boring to study and a lot of mental effort. I am lazy to decipher the ideas from the mathematical formulae. Tianze Wang August 15, 2019 at 1:30 am Reply So, how could we make a donation? mixbaal0 August 22, 2019 at 3:44 am Reply Wittgestein said something like "What we can talk about, we can say it clearly, otherwise is better to keep silence". You talk very, very clear. Congratulations! Mario G. August 25, 2019 at 12:06 pm Reply 11:51 I think there's a mistake in one of the equations. You have z superscript L superscript L instead of just z superscript L. Zachary Thatcher August 27, 2019 at 3:03 pm Reply What kind of resources do you need to apply neural networks and AI to researching other fields? I know that learning is computationally expensive, but if I am doing image object recognition on a large dataset, is there any description of time or space complexity as a function of the size of the dataset or the number of levels/nodes in a neural network? Bat YOK September 1, 2019 at 9:20 pm Reply You must be wicked smart! Cause the way you explain things is just awesome Lyaman Agabekova September 2, 2019 at 4:59 pm Reply Thank you for the explanation. Amazing channel! Leave a Reply Cancel Save my name, email, and website in this browser for the next time I comment.