Error replicating training a TensorFlow model with Keras

🤗Transformers
1

I want to use a fine tuned transformers model for one of my own projects so started with replicating the documentation example to make sure everything worked. (Fine-tune a pretrained model)

I’m on iOS (updated) and running on an M1 chip so used the tensorflow-metal install since the basic tensorflow / pytorch were both giving me issues (this also seemed to be the most common issue when trying to look this error up). Every step of the documentation example works up until the very last one for actually fitting the model. Code and error both are below:

model.fit(tokenized_data, labels)

Error:

2023-01-19 17:12:14.960409: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/polymorphism/function_cache.py", line 117, in lookup
    dispatch_key = self._dispatch_table.dispatch(key)
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/polymorphism/type_dispatch.py", line 78, in dispatch
    if request in self._dispatch_table:
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/polymorphism/function_cache.py", line 77, in __hash__
    return hash((self.call_context, self.function_type))
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/polymorphism/function_type.py", line 246, in __hash__
    return hash(
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/polymorphism/function_type.py", line 106, in __hash__
    return hash((self.name, self.kind, self.optional, self.type_constraint))
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/trace_type/default_types.py", line 207, in __hash__
    return hash(self.components)
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/trace_type/default_types.py", line 207, in __hash__
    return hash(self.components)
  File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tensorflow/core/function/trace_type/default_types.py", line 584, in __hash__
    return hash((self.identifier, self.base))
ValueError: Cannot generate a hashable key for IteratorSpec(({'input_ids': TensorSpec(shape=(None, 47), dtype=tf.int64, name=None), 'token_type_ids': TensorSpec(shape=(None, 47), dtype=tf.int64, name=None), 'attention_mask': TensorSpec(shape=(None, 47), dtype=tf.int64, name=None)}, TensorSpec(shape=(None,), dtype=tf.int64, name=None)),) because the _serialize() method returned an unsupproted value of type <class 'transformers.tokenization_utils_base.BatchEncoding'>

Any help would be much appreciated!

2

Hi! I think the issue is that you need to convert the BatchEncoding returned by the tokenizer into a dict for Keras. Skipping this step results in the error above!

Try this instead :slightly_smiling_face: :

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokenized_data = tokenizer(dataset["sentence"], return_tensors="np", padding=True)
tokenized_data = dict(tokenized_data)
3

Thanks for your help! I tried this and I wound up with a new error:

2023-01-20 16:09:49.860122: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2023-01-20 16:09:59.469589: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type GPU is enabled.
2023-01-20 16:10:03.135639: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x11fe451e0
....(this repeats a bunch of times)
2023-01-20 16:10:06.430412: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x11fe451e0
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
Cell In[19], line 1
----> 1 model.fit(tokenized_data, labels)

File ~/miniconda/lib/python3.10/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File ~/miniconda/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:52, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50 try:
     51   ctx.ensure_initialized()
---> 52   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                       inputs, attrs, num_outputs)
     54 except core._NotOkStatusException as e:
     55   if name is not None:

NotFoundError: Graph execution error:

Detected at node 'StatefulPartitionedCall_199' defined at (most recent call last):
    File "/Users/lexipalmer/miniconda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/Users/lexipalmer/miniconda/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel_launcher.py", line 17, in <module>
      app.launch_new_instance()
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/traitlets/config/application.py", line 1041, in launch_instance
      app.start()
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 724, in start
      self.io_loop.start()
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 215, in start
      self.asyncio_loop.run_forever()
    File "/Users/lexipalmer/miniconda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
      self._run_once()
    File "/Users/lexipalmer/miniconda/lib/python3.10/asyncio/base_events.py", line 1899, in _run_once
      handle._run()
    File "/Users/lexipalmer/miniconda/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 512, in dispatch_queue
      await self.process_one()
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 501, in process_one
      await dispatch(*args)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 408, in dispatch_shell
      await result
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 731, in execute_request
      reply_content = await reply_content
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 417, in do_execute
      res = shell.run_cell(
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 540, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 2945, in run_cell
      result = self._run_cell(
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3000, in _run_cell
      return runner(coro)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3203, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3382, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3442, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/var/folders/ny/h_bygvy53h16kd57z4lsmsvh0000gn/T/ipykernel_63158/3344439326.py", line 1, in <module>
      model.fit(tokenized_data, labels)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/engine/training.py", line 1650, in fit
      tmp_logs = self.train_function(iterator)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/engine/training.py", line 1249, in train_function
      return step_function(self, iterator)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/engine/training.py", line 1233, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/engine/training.py", line 1222, in run_step
      outputs = model.train_step(data)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/transformers/modeling_tf_utils.py", line 1557, in train_step
      self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 527, in minimize
      self.apply_gradients(grads_and_vars)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1140, in apply_gradients
      return super().apply_gradients(grads_and_vars, name=name)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 634, in apply_gradients
      iteration = self._internal_apply_gradients(grads_and_vars)
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1166, in _internal_apply_gradients
      return tf.__internal__.distribute.interim.maybe_merge_call(
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1216, in _distributed_apply_gradients_fn
      distribution.extended.update(
    File "/Users/lexipalmer/miniconda/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1211, in apply_grad_to_update_var
      return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: 'StatefulPartitionedCall_199'
could not find registered platform with id: 0x11fe451e0
	 [[{{node StatefulPartitionedCall_199}}]] [Op:__inference_train_function_34668]
4

Hey everyone,

It seems that Tensorflow handles more with OrderedDict. So I invite you to replace

tokenized_data = dict(tokenized_data)

with

from collections import OderedDict
tokenized_data = OderedDict(tokenized_data)

It should work. Hope it helps