Strange Error While Attempting to Load DataSet

John6666 · March 28, 2025, 2:55pm

TensorFlow error?

Keras fit method not working with dataset iterator

opened 02:27PM - 14 Jun 18 UTC

closed 06:08PM - 21 Jun 18 UTC

### System information - **Have I written custom code (as opposed to using a st…ock example script provided in TensorFlow)**: Custom - **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10 - **TensorFlow installed from (source or binary)**: Binary - **TensorFlow version (use command below)**: 1.9-rc0 - **Python version**: 3.6 - **Bazel version (if compiling from source)**: NA - **GCC/Compiler version (if compiling from source)**: NA - **CUDA/cuDNN version**: NA - **GPU model and memory**: NA - **Exact command to reproduce**: `model.fit(get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)` and `model.fit(get_iterator,get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)` ### Describe the problem When I pass one dataset iterator to `fit` method, I get: > Please provide data as a list or tuple of 2 elements - input and target pair. Received Tensor("IteratorGetNext_4:0", shape=(2, ?), dtype=float32) When I pass two iterators I get the error: > ValueError: You passed a dataset or dataset iterator (<tensorflow.python.data.ops.iterator_ops.Iterator object at 0x000001FEABE88748>) as input `x` to your model. In that case, you should not specify a target (`y`) argument, since the dataset or dataset iterator generates both input data and target data. Received: <tensorflow.python.data.ops.iterator_ops.Iterator object at 0x000001FEABE88748> When I create a new dataset after zipping the original x and y data set and pass that to `fit `I get the error described in https://github.com/tensorflow/tensorflow/issues/19912 According to 1.9-rc0 method release notes iterators should be usable with keras training methods. Please provide a solution or provide clarification in the documentation. ### Source code / logs ``` dataset= tf.contrib.data.make_csv_dataset(file_name,48,select_columns= ['Load_residential_multi_0','Load_residential_multi_1'],shuffle=False) dataset = dataset.map(lambda x: tf.stack(list(x.values()))) get_iterator = dataset.make_one_shot_iterator() get_batch = get_iterator.get_next() #Building and training a single layer model using Keras (Available within TensorFlow) model = Sequential() #Input Layer model.add(InputLayer(input_shape=(48,),name='InputLayer'))#,input_tensor =dataset #model.add(BatchNormalization(axis=1)) #Normalizing values #Layer1 model.add(Dense(units=5,activation='relu',name='FeedForward1')) #Add a feed forward layer #Layer2 model.add(Dense(units=5,activation='relu',name='FeedForward2')) #Add a feed forward layer #Output layer model.add(Dense(units=48,name='OutputLayer')) #Specify los function and optimizer model.compile(loss='mse',optimizer='adam',metrics=['mae']) #Summarize model model.summary() #Train the model model.fit(get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1) #model.fit(get_iterator,get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1) ```

github.com/keras-team/keras

PyDataset Documentation and Best Practices

opened 01:09PM - 21 Aug 24 UTC

dryglicki

type:support

**Keras Version:** 3.5.0 **Tensorflow Version:** 2.17.0 **What I want to do:…** Use PyDataset class in a data distributed environment. --- I would like to ask about the status of PyDataset and some of its best uses and practices. I have a functioning PyDataset class that ingests and processes HDF files: ``` class HDFDataset(K.utils.PyDataset): ''' Keras data loader to replace Tensorflow's Dataset API. Reads HDF5 files. Inputs: file_list: list list of file names, pre-globbed batch_size: int size of batches shuffle: bool whether or not to shuffle the dataset at the end of each epoch lons_lats: bool whether or not to include longitudes and latitudes -- Additional keyword arguments -- workers=1 use_multiprocessing=False max_queue_size=10 ''' def __init__(self, file_list: list | tuple | set, batch_size: int, shuffle: bool = False, lons_lats: bool = False, subsample: bool = False, **kwargs): super(HDFDataset, self).__init__(**kwargs) self.shuffle = shuffle self.batch_size = batch_size self.tmplen = len(self.file_list) self.subsample = subsample if self.subsample: self.slice = slice(64, 192) self.time_slice = slice(0,6) def __len__(self): return self.tmplen // self.batch_size def _extract_data_from_hdf5(self, file_list): input_list = ['priors', 'model'] output_list = ['forecast'] # Preparing input dictionary inputs_dict = {} for name in input_list: new_var = f'input_{name}' inputs_dict[new_var] = [] outputs = [] for f in file_list: with h5py.File(f, 'r') as h5: for k in input_list: new_var = f'input_{k}' if self.subsample: inputs_dict[new_var].append(h5.get(k)[:, self.slice, self.slice, :]) else: inputs_dict[new_var].append(h5.get(k)[...]) for k in output_list: if self.subsample: outputs.append(h5.get(k)[0:6, self.slice, self.slice, :]) else: outputs.append(h5.get(k)[0:6, self.slice, self.slice, :]) for k in input_list: nv = f'input_{k}' inputs_dict[nv] = np.stack(inputs_dict[nv], axis = 0) outputs = np.stack(outputs, axis = 0) return inputs_dict, outputs def __getitem__(self, idx: int): if idx >= self.__len__(): raise StopIteration low = idx * self.batch_size high = min(low + self.batch_size, self.tmplen) inputs, outputs = self._extract_data_from_hdf5(self.file_list[low:high]) return [inputs, outputs] def on_epoch_end(self): if self.shuffle: random.shuffle(self.file_list) # In-place shuffle ``` This works for my case really nicely. It avoids the memory leak nightmare with which I have been dealing by directly trying to use the `tf.data` API (https://github.com/tensorflow/tensorflow/issues/72014) for multiple inputs from the same file. But the documentation on PyDataset stinks! Looking inside the [source code](https://github.com/keras-team/keras/blob/v3.5.0/keras/src/trainers/data_adapters/py_dataset_adapter.py), PyDataset has an Adapter class that will make a Tensorflow data generator. Does this automatically get called during `fit()`? Is it best practice to call the data generator directly so I can distribute the dataset via [TF's experimental distribute dataset function](https://www.tensorflow.org/api_docs/python/tf/distribute/Strategy#experimental_distribute_dataset)? In the source, there is also a `PyDatasetEnqueuer` class. Do I need this? Why is this here? Who is the target audience? Is the expectation of the Enquerer in the PyDataset class also the reason I need to raise a `StopIteration` command in `__getitem__`? Also digging into source, at [this point](https://github.com/keras-team/keras/blob/v3.5.0/keras/src/trainers/data_adapters/py_dataset_adapter.py), the shuffle is hard-coded to 8. That probably needs to go. Anyway, I don't have any specific programming questions here, but I would like to know what best practices are, how do I use `PyDataset` in a (Tensorflow) distributed data environment, and so on.

Topic		Replies	Views
Datasets.load_datasets fails 🤗Datasets	12	850	October 11, 2024
Cannot load dataset on Kaggle 🤗Datasets	4	3145	August 16, 2023
Unable to Load Dataset Using `load_dataset` 🤗Datasets	10	378	March 11, 2025
TypeError: Couldn't cast array of type int64 to null 🤗Datasets	3	126	February 6, 2025
DatasetGenerationError. Failed to parse string: as a scalar of type double Beginners	3	112	January 7, 2025

Strange Error While Attempting to Load DataSet

Related topics