Activation functions: A function applied elementwise to the output of each neuron in a neural network, the activation functions are used to introduce non-linearity in the network.