-
Notifications
You must be signed in to change notification settings - Fork 915
fix: resolve TensorFlow keras lazy loader recursion error #3126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Set TF_USE_LEGACY_KERAS=1 to force TensorFlow 2.19.0 to use tf_keras instead of Keras 3.x if the environment has Keras 3.x installed. This work around prevents TensorFlow's lazy loader from infinitely recursing when trying to determine which Keras version to use. See: https://keras.io/getting_started/#tensorflow--keras-2-backwards-compatibility. This fixes generate_test_models.py which was failing with: "RecursionError: maximum recursion depth exceeded in comparison" at line checking "if self._tfll_keras_version == 'keras_3'" BUG=tensorflow#3125
Oops, must ensure that tf_keras is installed. Have I mentioned how much I dislike our non-hermetic Python setup in Bazel? I might fix that soon. |
Thanks Ryan. Should we also add this for all other files that use Keras? Or does issue only occur for this one? |
Also is it not an issue for the CI because we have only one keras version in those testing environments? |
AFACT, the problem only rears its head when Keras >= 3 is visible alongside (older versions of?) Tensorflow. I'm not 100% sure of the circumstances in which this happens. Perhaps it only happens with some of our Bazel targets due to varying dependency sets and Bazel's sandboxing 🤷♂️. I had Keras 3 in my environment, because I was doing model compression with TensorFlow Model Optimization Toolkit, which has a dependency on Keras 3. Maybe the workaround in this PR isn't the right solution. I can think of at least two other possibilities:
Those seemed like changes with possibly farther-reaching impact, hence my affinity for the workaround. I'll experiment with it a bit more and follow up. |
Ugh. Turns out I was wrong about the underlying problem. The workaround does indeed avoid importing Keras, but that "fixes" a different underlying problem. The underlying problem is that Keras imports the module
There appears to be no problem with Tensorflow coexisting with Keras >= 3. In fact, Keras >= 3 has been in our Bazel Python dependencies alongside Tensorflow 2.18 for some time now without any issues. I will withdraw this PR, and open or reopen an issue about configuring Bazel's Python toolchain to be hermetic. Backtrace:
|
Set TF_USE_LEGACY_KERAS=1 to force TensorFlow 2.19.0 to use tf_keras
instead of Keras 3.x if the build environment has both keras 3.10.0 and
tf_keras 2.19.0 installed. This prevents TensorFlow's lazy loader from
entering infinite recursion when trying to determine which Keras version
to use for tf.keras.models.Sequential().
This fixes runtime_test which was failing with:
"RecursionError: maximum recursion depth exceeded in comparison"
at line checking "if self._tfll_keras_version == 'keras_3'"
BUG=#3125