How to understand static shape and dynamic shape in TensorFlow?

In TensorFlow, understanding static shapes and dynamic shapes is crucial for developing efficient and flexible models.

Static Shapes

Static shapes refer to the dimensions defined at the time of Tensor creation. This shape is established during the graph construction phase, and once set, it cannot be modified. Static shapes are essential for graph optimization and performance improvement because they enable TensorFlow to perform more comprehensive static analysis and optimizations during compilation.

In code implementation, we typically define the Tensor's shape directly using tf.placeholder or the constructor to set static shapes. For example:

python
import tensorflow as tf

# Using tf.placeholder to define a static shape
x = tf.placeholder(tf.float32, shape=[None, 10])
print(x.shape)  # Output: (?, 10), where ? indicates that the size of this dimension can vary at runtime, but 10 is fixed

Once the static shape of a Tensor is determined, it cannot be altered; attempting to modify it will result in an error.

Dynamic Shapes

Dynamic shapes allow us to change the Tensor's shape during the graph execution phase. This is particularly useful when handling data with varying batch sizes or dynamic sequence lengths. While dynamic shapes provide greater flexibility, they may incur some performance trade-offs.

Dynamic shape modifications are typically implemented using the tf.reshape function, which enables shape changes during graph execution. For example:

python
import tensorflow as tf

# Define a static shape
x = tf.placeholder(tf.float32, shape=[None, 10])
print(x.shape)  # Output: (?, 10)

# Define dynamic shape change
x_dynamic = tf.reshape(x, [2, -1])
print(x_dynamic)  # Output shape is determined at runtime, depending on the input data

with tf.Session() as sess:
    # Provide actual data and execute
    feed_dict = {x: [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]]}
    result = sess.run(x_dynamic, feed_dict=feed_dict)
    print(result.shape)  # Output: (2, 10)

In this example, the static shape of x is (?, 10), indicating that the first dimension can vary at runtime while the second dimension is fixed at 10. Using tf.reshape, we dynamically reshape it to the shape (2, -1), where -1 automatically calculates the size of this dimension to maintain the total number of elements unchanged.

Summary

Static shapes, once set, cannot be modified and facilitate graph optimization; dynamic shapes provide flexibility by allowing Tensor shape adjustments during runtime. In practical applications, effectively leveraging the characteristics of both shape types can enhance the design and optimization of TensorFlow models.