Eager mode

check mode

To check mode of an execution of a function:

tf.executing_eagerly()

and more info about it

In a tf graph:

• breakpoints are not working
• tensor values not inspectable
• any python side-effects (appending to a list, printing with print, etc) will only happen once, when graph func is traced
• graph is traced if same type and same shape
• graph is traced every time if passing python scalar or list
• and many more and more

If eager mode, plain python code related to tf (except model's logic) will not be transformed into graph. To disable eager mode:

tf.compat.v1.disable_eager_execution()  # tf.compat.v1.enable_eager_execution

In eager mode, python code can be manually transformed to graph:

model.compile(run_eagerly=True) # default

@tf.function

In eager mode, more memory is required. And the random counter is very large compared with graph mode.

# tf 2.1 eager
from tensorflow.python.framework import random_seed
tf.random.set_seed(0)
random_seed.get_seed(None)  # counter (0, 1654615998)
random_seed.get_seed(None)  # counter (0, 2087043557)

# tf 2.1 graph
from tensorflow.python.framework import random_seed
tf.random.set_seed(0)
random_seed.get_seed(None)  # counter (0, 1)
random_seed.get_seed(None)  # counter (0, 3)

# tf 2.1 different graph
from tensorflow.python.framework import random_seed
tf.random.set_seed(0)
random_seed.get_seed(None)  # counter (0, 2)
random_seed.get_seed(None)  # counter (0, 4)

# tf 2.1 random counter increasing sequence for a model:
# weights first, then dataset.shuffle, the last is dropout
# weights or dropout consume random counter only when they exist both in build and call method of tf.keras.layers.Layer

tf eager mode is still developing and sometimes eager function and graph function cannot interact desirably:

• tf below 2.1, a function with tuple as input cannot accept sequence like list
• take care of empty input

  # since tf.function transform code to graph, we expect it can interact correctly with tf's graph tensors
  # however, error occurs if input_signatures exist for model function which requires keras's learning phase(whether training)
  # error occurs might due to keras_learning_phase is symbolic(graph) tensor, but func is first a eager function then transformed to graph
  @tf.function(input_signatures=[])
  model_func():
      depends on learning phase
  
  # Okay, black box success
  @tf.function
  model_func():
      depends on learning phase
  

to resolve above issues:

tf.config.run_functions_eagerly(True)  # tf >= 2.3
tf.config.experimental_run_functions_eagerly(True)  # tf >= 2.0