Determinism

Ops determinism + value calculation determinism => reproducibility

Enable determinism starting from tf 2.8

tf.config.experimental.enable_op_determinism()

tf.keras.utils.set_random_seed(42)

tf ops:

• If neither the global seed nor the operation seed is set: A randomly picked seed is used for this op.
• If the graph-level seed is set, but the operation seed is not: The system deterministically picks an operation seed in conjunction with the graph-level seed so that it gets a unique random sequence. Within the same version of tensorflow and user code, this sequence is deterministic. However across different versions, this sequence might change. If the code depends on particular seeds to work, specify both graph-level and operation-level seeds explicitly.
• If the operation seed is set, but the global seed is not set: A default global seed and the specified operation seed are used to determine the random sequence.
• If both the global and the operation seed are set: Both seeds are used in conjunction to determine the random sequence.

calculation determinism: These differences are often caused by the use of asynchronous threads within the op nondeterministically changing the order in which floating-point numbers are added. Most of these cases of nondeterminism occur on GPUs, which have thousands of hardware threads that are used to run ops. Enabling determinism directs such ops to use a different algorithm, one that does not use threads in a nondeterministic way.

tf.keras.utils.set_random_seed(42) == tf.random.set_seed(42) + random.seed(42) + numpy.random.seed(42)