How to do Xavier initialization on TensorFlow

In TensorFlow, using Xavier initialization (also known as Glorot initialization) helps maintain consistent variance between inputs and outputs, which is crucial for training deep learning networks. Xavier initialization is particularly suitable for neural networks with activation functions such as Sigmoid or Tanh. The following sections detail how to apply Xavier initialization in TensorFlow.

1. Using TensorFlow 1.x

In TensorFlow 1.x, you can use Xavier initialization via tf.contrib.layers.xavier_initializer():

python
import tensorflow as tf

# Create variables with Xavier initialization
weights = tf.Variable(tf.contrib.layers.xavier_initializer()(shape=[input_dim, output_dim]))

2. Using TensorFlow 2.x

In TensorFlow 2.x, tf.contrib has been deprecated. Instead, you can use GlorotUniform or GlorotNormal from tf.keras, which are variants of Xavier initialization. By default, the Keras Dense layer uses GlorotUniform initialization:

python
import tensorflow as tf
from tensorflow.keras.layers import Dense

# Use Dense layer with default GlorotUniform initialization
model = tf.keras.Sequential([
    Dense(128, activation='relu', input_shape=(input_dim,)),
    Dense(output_dim)
])

If you need to explicitly specify Xavier initialization (e.g., using a normal distribution), you can do the following:

python
from tensorflow.keras.initializers import GlorotNormal

# Explicitly use Xavier normal distribution for initialization
model = tf.keras.Sequential([
    Dense(128, activation='relu', kernel_initializer=GlorotNormal(), input_shape=(input_dim,)),
    Dense(output_dim, kernel_initializer=GlorotNormal())
])

Example Application

Suppose we are developing a neural network for handwritten digit classification, with an input layer dimension of 784 (28x28 pixel images) and an output layer dimension of 10 (for 10 digit classes). We can use Xavier initialization to help the model achieve better performance during the initial training phase:

python
from tensorflow.keras.layers import Flatten

# Create model
model = tf.keras.Sequential([
    Flatten(input_shape=(28, 28)),
    Dense(256, activation='relu', kernel_initializer=GlorotNormal()),
    Dense(10, activation='softmax', kernel_initializer=GlorotNormal())
])

# Compile model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Model training
model.fit(x_train, y_train, epochs=10, validation_data=(x_test, y_test))

By using Xavier initialization, we ensure that the variance of inputs and outputs remains balanced across layers, which helps avoid gradient vanishing or exploding issues during training, allowing the model to converge faster.

This covers the basic methods and example applications for using Xavier initialization in TensorFlow. I hope this helps you understand how to implement this initialization strategy in your specific projects.