Artificial Neural Networks in Machine Learning

What are Artificial Neural Networks?

• The term "Artificial Neural Network" comes from replicating the structure of the human brain, which is composed of biological neural networks.
• Artificial Neural Networks (ANNs) are computational models inspired by the human brain's structure and functioning.
• They consist of interconnected nodes, or artificial neurons, organized into layers.
• ANNs are used for tasks such as pattern recognition, classification, and regression.

Hebbian Network (Hebb Net)

• Hebbian learning is a principle stating that if two neurons are activated simultaneously, the connection between them strengthens.
• Hebbian Networks, or Hebb Nets, implement this learning rule.
• The Hebbian rule was the first learning rule. In 1949, Donald Hebb created it as the learning algorithm for unsupervised neural networks.
• It is used for pattern classification. It is a single layer neural network, i.e. it has one input layer and one output layer.

Learning Rule:

Example:

Perceptron

• The perceptron serves as a fundamental component of an Artificial Neural Network.
• A Perceptron is the simplest form of a neural network.
• It takes multiple binary inputs, applies weights to them, sums the results.
• Perceptron is considered as a single-layer neural network that consists of four main components.

Example:

• In a Perceptron for cancer prediction, inputs could represent factors like tumor size and age.
• The network learns to predict cancer (output: 1) or no cancer (output: 0) based on weighted features.

Adaline (Adaptive Linear Neuron)

• Adaline is an improvement over the Perceptron, introducing a continuous activation function and the ability to learn from errors.
• It was created by Professor Bernard Widrow and his student Ted Hoff at Stanford University in 1960.
• It employs gradient descent to minimize the cost function, adjusting weights iteratively.

Components:

• Inputs and Weights: Similar to a Perceptron.
• Summation Function: Calculates the weighted sum of inputs.
• Activation Function: Uses the weighted sum to produce the output.

Example:

Multilayer Neural Network

• Multilayer Perceptron also known as Artificial Neural Networks consists of more than one perceptron which is grouped together to form a multiple layer neural network.
• A Multilayer Neural Network consists of multiple layers of neurons, including input, hidden, and output layers.
• Feedforward Neural Networks (FNN), also known as Multi-layer Perceptrons (MLP),
• They are a type of artificial neural network designed for information processing in a forward direction.
• A multi-layered perceptron model is effective for tackling complex non linear problems.
• The presence of hidden layers allows these networks to learn complex relationships and patterns.

Components:

• Input Layer: Receives input features.
• Hidden Layers: Process information, capturing intricate patterns.
• Output Layer: Produces final predictions.

Example:

Architecture

Input Layer

• Role: Receives Features.
• The input layer is the first layer of the neural network, receiving the raw features or input data.
• Each neuron in this layer represents a specific feature.

Hidden Layers

• Role: Process Information.
• Hidden layers come between the input and output layers and are where the neural network learns complex patterns and relationships within the data.
• Each neuron in a hidden layer processes information based on learned weights and activation functions.

Output Layer

• Role: Produces Predictions.
• The output layer produces the final predictions or outcomes based on the processed information from the hidden layers.
• The number of neurons in this layer depends on the nature of the task (e.g., binary classification, multi-class classification, regression).

Connections (Weights)

• Role: Determine Signal Strength.
• Connections between neurons are represented by weights.
• These weights determine the strength of the signal between neurons.
• During training, the neural network adjusts these weights to minimize the difference between predicted and actual outputs.

Example:

Activation Functions

• Activation functions brings non-linearity to neural networks, allowing them to learn complex patterns and relationships in data.
• An activation function acts as a gate, checking if an incoming value surpasses a specific threshold.
• Activation functions is also known as Transfer Function.
• These functions determine the output of a neuron or a layer and play a crucial role in the learning process.

Popular Activation Functions

• Sigmoid Function: Squashes input values between 0 and 1.
• Example: Used in the output layer of binary classification models.
• Hyperbolic Tangent (tanh): Similar to the sigmoid but squashes values between -1 and 1.
• Example: Often used in hidden layers of neural networks.
• Rectified Linear Unit (ReLU): Outputs the input for positive values; zero for negative values.
• Example: Widely used in hidden layers due to simplicity and effectiveness.

Example:

Loss Function

Types of Loss Functions

• Mean Squared Error (MSE)
• This involves calculating the average squared difference between predicted and actual values.
• Example: Regression tasks where precise numeric predictions are crucial.

• Binary Cross-Entropy
• Measures the binary classification error.
• Example: Used when the output is binary, such as spam or not spam classification.

• Categorical Cross-Entropy
• Extends binary cross-entropy to multi-class classification.
• Example: Classifying images into multiple categories.

Example:

Hyperparameters

• Hyperparameters are configuration settings external to the model that impact its learning process.
• These settings are not learned from the data but need to be set beforehand.
• random_state: It is a parameter that specifies the random number used for weights and bias initialization.
• Learning Rate: Determines the step size during gradient descent.
• Batch Size: Specifies the number of training samples used in one iteration.
• Activation function for the hidden layer. Examples, identity, logistic, tanh, and relu.

Gradient Descent

• Gradient Descent is a fundamental optimization algorithm used in machine learning and deep learning to iteratively train models by minimizing the error or cost function.
• It operates by adjusting the model's parameters, often represented as weights, in the direction that reduces the error.

Key Concepts

Stochastic Gradient Descent (SGD)

• SGD is a variant of gradient descent where parameters are updated using a subset (batch) of the training data rather than the entire dataset.
• This approach reduces computational requirements and is particularly useful for large datasets.

Gradient Ascent

• While gradient descent minimizes the cost function, gradient ascent maximizes a function.
• It's used in scenarios where the goal is to find the maximum, such as maximizing likelihood in probabilistic models.
  

Conclusion

• Artificial Neural Networks (ANNs) mimic the brain's structure for tasks like pattern recognition.
• They evolve from basic components like Perceptrons to more sophisticated Multilayer Neural Networks with input, hidden, and output layers.