Embedding models fail possibly on scientific characters no matter how the pdf is parsed

[mranaarsal95]

I am trying to create a RAG pipline using Llamaindex. Anytime I try to use an embedding model on these papers fpr semantic chunking or anything, it does not follow through, which is I think is possibly due to the presence of scientific characters. I’m working with large amount of papers so it’s not possible to audit each separately. Are there any options other than removing the chunks/papers that are problematic? The documents have been parsed in Llamaparse and saved in markdown all with SimpleDirectoryReader - same issue arose in the past when I parsed with PyMuPDF and SimpleDirectoryReader, so my thoughts go towards the embedding process as the main issue. The parsed data itself has been properly preserved in the .pkl file so I don’t think any issues with reading that. It works on simple texts, or docs[0] when tested individually but the entire process fails together.

from llama_index.core import SimpleDirectoryReader
from llama_parse import LlamaParse
import nest_asyncio
import os
nest_asyncio.apply()
from dotenv import load_dotenv

load_dotenv()
first_api = os.getenv("LLAMA_CLOUD_API_KEY")

parser = LlamaParse(api_key = first_api
                    ,result_type="markdown", user_prompt="Make sure the structure of all info is preserved",
                    extract_charts=True,auto_mode_trigger_on_table_in_page=True,  auto_mode_trigger_on_image_in_page=True,
)

documents = SimpleDirectoryReader(
    input_dir="./docs", 
    file_extractor={"pdf": parser}
).load_data()
print("docs ingested")
print(documents[0].text[:100])

import pickle

with open("mds ", 'wb') as f: 
    pickle.dump(documents, f)

(changed the file extension to Chengwei Semiconductor afterwards)

from llama_index.core.node_parser import SemanticSplitterNodeParser, SentenceSplitter
import pickle
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
import nest_asyncio
nest_asyncio.apply()
from dotenv import load_dotenv
import os
load_dotenv()

embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

with open("/Users/rifah/Desktop/p/species/mds.pkl", "rb") as f:
    docs = pickle.load(f)


splitter = SemanticSplitterNodeParser(buffer_size=1, embed_model=embed_model,include_metadata=True)

nodes = splitter.get_nodes_from_documents(docs) 

2026-07-24 16:27:07.093685: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/opt/anaconda3/lib/python3.12/site-packages/keras/src/export/tf2onnx_lib.py:8: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
  if not hasattr(np, "object"):
Embedding attempt failed: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
Embedding attempt failed: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
Embedding attempt failed: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/model.py:557, in BaseModel.preprocess(self, inputs, prompt, **kwargs)
    556 try:
--> 557     preprocessed = self[0].preprocess(inputs, prompt=prompt, **kwargs)
    558 except TypeError:

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py:988, in Transformer.preprocess(self, inputs, prompt, processing_kwargs, **kwargs)
    987 with suggest_extra_on_exception():
--> 988     processor_output = self._call_processor(
    989         modality,
    990         processor_inputs,
    991         modality_kwargs,
    992         common_kwargs,
    993         chat_template_kwargs=chat_template_kwargs,
    994     )
    996 if num_images_per_sample is not None and "image_grid_thw" in processor_output:

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py:1241, in Transformer._call_processor(self, modality, processor_inputs, modality_kwargs, common_kwargs, chat_template_kwargs)
   1239     return self._call_multimodal_processor(modality, processor_inputs, modality_kwargs, common_kwargs)
-> 1241 return self._call_single_modality_processor(modality, processor_inputs, modality_kwargs, common_kwargs)

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py:1302, in Transformer._call_single_modality_processor(self, modality, processor_inputs, modality_kwargs, common_kwargs)
   1301     primary_input = processor_inputs.pop(modality_type)
-> 1302     return self.processor(primary_input, **processor_inputs, **call_kwargs)
   1303 return self.processor(**processor_inputs, **call_kwargs)

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:3078, in PreTrainedTokenizerBase.__call__(self, text, text_pair, text_target, text_pair_target, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   3077         self._switch_to_input_mode()
-> 3078     encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
   3079 if text_target is not None:

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:3166, in PreTrainedTokenizerBase._call_one(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens, **kwargs)
   3165     batch_text_or_text_pairs = list(zip(text, text_pair)) if text_pair is not None else text
-> 3166     return self.batch_encode_plus(
   3167         batch_text_or_text_pairs=batch_text_or_text_pairs,
   3168         add_special_tokens=add_special_tokens,
   3169         padding=padding,
   3170         truncation=truncation,
   3171         max_length=max_length,
   3172         stride=stride,
   3173         is_split_into_words=is_split_into_words,
   3174         pad_to_multiple_of=pad_to_multiple_of,
   3175         padding_side=padding_side,
   3176         return_tensors=return_tensors,
   3177         return_token_type_ids=return_token_type_ids,
   3178         return_attention_mask=return_attention_mask,
   3179         return_overflowing_tokens=return_overflowing_tokens,
   3180         return_special_tokens_mask=return_special_tokens_mask,
   3181         return_offsets_mapping=return_offsets_mapping,
   3182         return_length=return_length,
   3183         verbose=verbose,
   3184         split_special_tokens=split_special_tokens,
   3185         **kwargs,
   3186     )
   3187 else:

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:3367, in PreTrainedTokenizerBase.batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens, **kwargs)
   3358 padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
   3359     padding=padding,
   3360     truncation=truncation,
   (...)
   3364     **kwargs,
   3365 )
-> 3367 return self._batch_encode_plus(
   3368     batch_text_or_text_pairs=batch_text_or_text_pairs,
   3369     add_special_tokens=add_special_tokens,
   3370     padding_strategy=padding_strategy,
   3371     truncation_strategy=truncation_strategy,
   3372     max_length=max_length,
   3373     stride=stride,
   3374     is_split_into_words=is_split_into_words,
   3375     pad_to_multiple_of=pad_to_multiple_of,
   3376     padding_side=padding_side,
   3377     return_tensors=return_tensors,
   3378     return_token_type_ids=return_token_type_ids,
   3379     return_attention_mask=return_attention_mask,
   3380     return_overflowing_tokens=return_overflowing_tokens,
   3381     return_special_tokens_mask=return_special_tokens_mask,
   3382     return_offsets_mapping=return_offsets_mapping,
   3383     return_length=return_length,
   3384     verbose=verbose,
   3385     split_special_tokens=split_special_tokens,
   3386     **kwargs,
   3387 )

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py:553, in PreTrainedTokenizerFast._batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens)
    551     self._tokenizer.encode_special_tokens = split_special_tokens
--> 553 encodings = self._tokenizer.encode_batch(
    554     batch_text_or_text_pairs,
    555     add_special_tokens=add_special_tokens,
    556     is_pretokenized=is_split_into_words,
    557 )
    559 # Convert encoding to dict
    560 # `Tokens` has type: tuple[
    561 #                       list[dict[str, list[list[int]]]] or list[dict[str, 2D-Tensor]],
    562 #                       list[EncodingFast]
    563 #                    ]
    564 # with nested dimensions corresponding to batch, overflows, sequence length

TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
Cell In[4], line 21
     15     docs = pickle.load(f)
     18 splitter = SemanticSplitterNodeParser(buffer_size=1, embed_model=embed_model,include_metadata=True)
---> 21 nodes = splitter.get_nodes_from_documents(docs)

File /opt/anaconda3/lib/python3.12/site-packages/llama_index/core/node_parser/interface.py:176, in NodeParser.get_nodes_from_documents(self, documents, show_progress, **kwargs)
    171 doc_id_to_document = {doc.id_: doc for doc in documents}
    173 with self.callback_manager.event(
    174     CBEventType.NODE_PARSING, payload={EventPayload.DOCUMENTS: documents}
    175 ) as event:
--> 176     nodes = self._parse_nodes(documents, show_progress=show_progress, **kwargs)
    177     nodes = self._postprocess_parsed_nodes(nodes, doc_id_to_document)
    179     event.on_end({EventPayload.NODES: nodes})

File /opt/anaconda3/lib/python3.12/site-packages/llama_index_instrumentation/dispatcher.py:413, in Dispatcher.span.<locals>.wrapper(func, instance, args, kwargs)
    410             _logger.debug(f"Failed to reset active_span_id: {e}")
    412 try:
--> 413     result = func(*args, **kwargs)
    414     if isinstance(result, asyncio.Future):
    415         # If the result is a Future, wrap it
    416         new_future = asyncio.ensure_future(result)

File /opt/anaconda3/lib/python3.12/site-packages/llama_index/core/node_parser/text/semantic_splitter.py:137, in SemanticSplitterNodeParser._parse_nodes(self, nodes, show_progress, **kwargs)
    134 nodes_with_progress = get_tqdm_iterable(nodes, show_progress, "Parsing nodes")
    136 for node in nodes_with_progress:
--> 137     nodes = self.build_semantic_nodes_from_documents([node], show_progress)
    138     all_nodes.extend(nodes)
    140 return all_nodes

File /opt/anaconda3/lib/python3.12/site-packages/llama_index/core/node_parser/text/semantic_splitter.py:173, in SemanticSplitterNodeParser.build_semantic_nodes_from_documents(self, documents, show_progress)
    169 text_splits = self.sentence_splitter(text)
    171 sentences = self._build_sentence_groups(text_splits)
--> 173 combined_sentence_embeddings = self.embed_model.get_text_embedding_batch(
    174     [s["combined_sentence"] for s in sentences],
    175     show_progress=show_progress,
    176 )
    178 for i, embedding in enumerate(combined_sentence_embeddings):
    179     sentences[i]["combined_sentence_embedding"] = embedding

File /opt/anaconda3/lib/python3.12/site-packages/llama_index_instrumentation/dispatcher.py:413, in Dispatcher.span.<locals>.wrapper(func, instance, args, kwargs)
    410             _logger.debug(f"Failed to reset active_span_id: {e}")
    412 try:
--> 413     result = func(*args, **kwargs)
    414     if isinstance(result, asyncio.Future):
    415         # If the result is a Future, wrap it
    416         new_future = asyncio.ensure_future(result)

File /opt/anaconda3/lib/python3.12/site-packages/llama_index/core/base/embeddings/base.py:517, in BaseEmbedding.get_text_embedding_batch(self, texts, show_progress, **kwargs)
    515     self.rate_limiter.acquire()
    516 if not self.embeddings_cache:
--> 517     embeddings = self._get_text_embeddings(cur_batch)
    518 elif self.embeddings_cache is not None:
    519     embeddings = self._get_text_embeddings_cached(cur_batch)

File /opt/anaconda3/lib/python3.12/site-packages/llama_index/embeddings/huggingface/base.py:333, in HuggingFaceEmbedding._get_text_embeddings(self, texts)
    322 def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
    323     """
    324     Generates Embeddings for text.
    325 
   (...)
    331 
    332     """
--> 333     return self._embed(texts, prompt_name="text")

File /opt/anaconda3/lib/python3.12/site-packages/llama_index/embeddings/huggingface/base.py:268, in HuggingFaceEmbedding._embed(self, inputs, prompt_name)
    248 def _embed(
    249     self,
    250     inputs: List[Union[str, BytesIO]],
    251     prompt_name: Optional[str] = None,
    252 ) -> List[List[float]]:
    253     """
    254     Generates Embeddings with input validation and retry mechanism.
    255 
   (...)
    266 
    267     """
--> 268     return self._embed_with_retry(inputs, prompt_name)

File /opt/anaconda3/lib/python3.12/site-packages/tenacity/__init__.py:331, in BaseRetrying.wraps.<locals>.wrapped_f(*args, **kw)
    329 copy = self.copy()
    330 wrapped_f.statistics = copy.statistics  # type: ignore[attr-defined]
--> 331 return copy(f, *args, **kw)

File /opt/anaconda3/lib/python3.12/site-packages/tenacity/__init__.py:470, in Retrying.__call__(self, fn, *args, **kwargs)
    468 retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)
    469 while True:
--> 470     do = self.iter(retry_state=retry_state)
    471     if isinstance(do, DoAttempt):
    472         try:

File /opt/anaconda3/lib/python3.12/site-packages/tenacity/__init__.py:371, in BaseRetrying.iter(self, retry_state)
    369 result = None
    370 for action in self.iter_state.actions:
--> 371     result = action(retry_state)
    372 return result

File /opt/anaconda3/lib/python3.12/site-packages/tenacity/__init__.py:413, in BaseRetrying._post_stop_check_actions.<locals>.exc_check(rs)
    411 retry_exc = self.retry_error_cls(fut)
    412 if self.reraise:
--> 413     raise retry_exc.reraise()
    414 raise retry_exc from fut.exception()

File /opt/anaconda3/lib/python3.12/site-packages/tenacity/__init__.py:184, in RetryError.reraise(self)
    182 def reraise(self) -> t.NoReturn:
    183     if self.last_attempt.failed:
--> 184         raise self.last_attempt.result()
    185     raise self

File /opt/anaconda3/lib/python3.12/concurrent/futures/_base.py:449, in Future.result(self, timeout)
    447     raise CancelledError()
    448 elif self._state == FINISHED:
--> 449     return self.__get_result()
    451 self._condition.wait(timeout)
    453 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File /opt/anaconda3/lib/python3.12/concurrent/futures/_base.py:401, in Future.__get_result(self)
    399 if self._exception:
    400     try:
--> 401         raise self._exception
    402     finally:
    403         # Break a reference cycle with the exception in self._exception
    404         self = None

File /opt/anaconda3/lib/python3.12/site-packages/tenacity/__init__.py:473, in Retrying.__call__(self, fn, *args, **kwargs)
    471 if isinstance(do, DoAttempt):
    472     try:
--> 473         result = fn(*args, **kwargs)
    474     except BaseException:  # noqa: B902
    475         retry_state.set_exception(sys.exc_info())  # type: ignore[arg-type]

File /opt/anaconda3/lib/python3.12/site-packages/llama_index/embeddings/huggingface/base.py:236, in HuggingFaceEmbedding._embed_with_retry(self, inputs, prompt_name)
    234         self._model.stop_multi_process_pool(pool=pool)
    235     else:
--> 236         emb = self._model.encode(
    237             inputs,
    238             batch_size=self.embed_batch_size,
    239             prompt_name=prompt_name,
    240             normalize_embeddings=self.normalize,
    241             show_progress_bar=self.show_progress_bar,
    242         )
    243     return emb.tolist()
    244 except Exception as e:

File /opt/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/util/decorators.py:41, in deprecated_kwargs.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
     39         else:
     40             kwargs.pop(old_name)
---> 41 return func(*args, **kwargs)

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py:654, in SentenceTransformer.encode(self, inputs, prompt_name, prompt, batch_size, show_progress_bar, output_value, precision, convert_to_numpy, convert_to_tensor, device, normalize_embeddings, truncate_dim, pool, chunk_size, **kwargs)
    652 for start_index in trange(0, len(inputs_sorted), batch_size, desc="Batches", disable=not show_progress_bar):
    653     inputs_batch = inputs_sorted[start_index : start_index + batch_size]
--> 654     features = self.preprocess(inputs_batch, prompt=prompt, **kwargs)
    656     if is_hpu:
    657         features = self._pad_features_for_hpu(features)

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/model.py:561, in BaseModel.preprocess(self, inputs, prompt, **kwargs)
    559     if prompt and modality == "text":
    560         inputs = [(prompt + inp[0],) + inp[1:] if isinstance(inp, tuple) else prompt + inp for inp in inputs]
--> 561     preprocessed = self[0].preprocess(inputs, **kwargs)
    562 except AttributeError:
    563     if prompt and modality == "text":

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py:988, in Transformer.preprocess(self, inputs, prompt, processing_kwargs, **kwargs)
    985     num_images_per_sample, num_videos_per_sample = _count_media_per_sample(processor_inputs["message"])
    987 with suggest_extra_on_exception():
--> 988     processor_output = self._call_processor(
    989         modality,
    990         processor_inputs,
    991         modality_kwargs,
    992         common_kwargs,
    993         chat_template_kwargs=chat_template_kwargs,
    994     )
    996 if num_images_per_sample is not None and "image_grid_thw" in processor_output:
    997     processor_output["num_images_per_sample"] = torch.tensor(num_images_per_sample, dtype=torch.long)

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py:1241, in Transformer._call_processor(self, modality, processor_inputs, modality_kwargs, common_kwargs, chat_template_kwargs)
   1238 if isinstance(self.processor, ProcessorMixin):
   1239     return self._call_multimodal_processor(modality, processor_inputs, modality_kwargs, common_kwargs)
-> 1241 return self._call_single_modality_processor(modality, processor_inputs, modality_kwargs, common_kwargs)

File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py:1302, in Transformer._call_single_modality_processor(self, modality, processor_inputs, modality_kwargs, common_kwargs)
   1300     if modality_type in processor_inputs:
   1301         primary_input = processor_inputs.pop(modality_type)
-> 1302         return self.processor(primary_input, **processor_inputs, **call_kwargs)
   1303     return self.processor(**processor_inputs, **call_kwargs)
   1305 raise RuntimeError(
   1306     f"Could not determine how to call processor of type {type(self.processor).__name__} "
   1307     f"for modality '{format_modality(modality)}'"
   1308 )

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:3078, in PreTrainedTokenizerBase.__call__(self, text, text_pair, text_target, text_pair_target, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   3076     if not self._in_target_context_manager:
   3077         self._switch_to_input_mode()
-> 3078     encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
   3079 if text_target is not None:
   3080     self._switch_to_target_mode()

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:3166, in PreTrainedTokenizerBase._call_one(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens, **kwargs)
   3161         raise ValueError(
   3162             f"batch length of `text`: {len(text)} does not match batch length of `text_pair`:"
   3163             f" {len(text_pair)}."
   3164         )
   3165     batch_text_or_text_pairs = list(zip(text, text_pair)) if text_pair is not None else text
-> 3166     return self.batch_encode_plus(
   3167         batch_text_or_text_pairs=batch_text_or_text_pairs,
   3168         add_special_tokens=add_special_tokens,
   3169         padding=padding,
   3170         truncation=truncation,
   3171         max_length=max_length,
   3172         stride=stride,
   3173         is_split_into_words=is_split_into_words,
   3174         pad_to_multiple_of=pad_to_multiple_of,
   3175         padding_side=padding_side,
   3176         return_tensors=return_tensors,
   3177         return_token_type_ids=return_token_type_ids,
   3178         return_attention_mask=return_attention_mask,
   3179         return_overflowing_tokens=return_overflowing_tokens,
   3180         return_special_tokens_mask=return_special_tokens_mask,
   3181         return_offsets_mapping=return_offsets_mapping,
   3182         return_length=return_length,
   3183         verbose=verbose,
   3184         split_special_tokens=split_special_tokens,
   3185         **kwargs,
   3186     )
   3187 else:
   3188     return self.encode_plus(
   3189         text=text,
   3190         text_pair=text_pair,
   (...)
   3208         **kwargs,
   3209     )

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:3367, in PreTrainedTokenizerBase.batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens, **kwargs)
   3357 # Backward compatibility for 'truncation_strategy', 'pad_to_max_length'
   3358 padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
   3359     padding=padding,
   3360     truncation=truncation,
   (...)
   3364     **kwargs,
   3365 )
-> 3367 return self._batch_encode_plus(
   3368     batch_text_or_text_pairs=batch_text_or_text_pairs,
   3369     add_special_tokens=add_special_tokens,
   3370     padding_strategy=padding_strategy,
   3371     truncation_strategy=truncation_strategy,
   3372     max_length=max_length,
   3373     stride=stride,
   3374     is_split_into_words=is_split_into_words,
   3375     pad_to_multiple_of=pad_to_multiple_of,
   3376     padding_side=padding_side,
   3377     return_tensors=return_tensors,
   3378     return_token_type_ids=return_token_type_ids,
   3379     return_attention_mask=return_attention_mask,
   3380     return_overflowing_tokens=return_overflowing_tokens,
   3381     return_special_tokens_mask=return_special_tokens_mask,
   3382     return_offsets_mapping=return_offsets_mapping,
   3383     return_length=return_length,
   3384     verbose=verbose,
   3385     split_special_tokens=split_special_tokens,
   3386     **kwargs,
   3387 )

File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py:553, in PreTrainedTokenizerFast._batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens)
    550 if self._tokenizer.encode_special_tokens != split_special_tokens:
    551     self._tokenizer.encode_special_tokens = split_special_tokens
--> 553 encodings = self._tokenizer.encode_batch(
    554     batch_text_or_text_pairs,
    555     add_special_tokens=add_special_tokens,
    556     is_pretokenized=is_split_into_words,
    557 )
    559 # Convert encoding to dict
    560 # `Tokens` has type: tuple[
    561 #                       list[dict[str, list[list[int]]]] or list[dict[str, 2D-Tensor]],
    562 #                       list[EncodingFast]
    563 #                    ]
    564 # with nested dimensions corresponding to batch, overflows, sequence length
    565 tokens_and_encodings = [
    566     self._convert_encoding(
    567         encoding=encoding,
   (...)
    576     for encoding in encodings
    577 ]

TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

Are there any fixes or workarounds?

[John6666]

For now, I tried narrowing down the cause a little:

---

I do not think you need to start by deleting the problematic papers or replacing the embedding model.

The exception you are getting,

TypeError: TextEncodeInput must be Union[
    TextInputSequence,
    Tuple[InputSequence, InputSequence]
]

looks more like a failure at the input boundary of the fast tokenizer than an error that specifically means “unsupported scientific characters.”

There is one particularly close public reproducer: Hindsight issue #1875 reports the same exception with the same BAAI/bge-small-en-v1.5 model when a Python string contains an unpaired UTF-16 surrogate. I also tried a small controlled reproduction with the same BGE model, then through SentenceTransformers, LlamaIndex’s HuggingFaceEmbedding, and finally SemanticSplitterNodeParser.

In that test:

• ordinary ASCII worked;
• Greek letters such as α β γ Δ λ μ Ω worked;
• both µM and μM worked;
• ±, ΔG, Unicode minus signs, superscripts/subscripts, H₂O, 10⁻⁶, cm⁻¹, U+FFFD (�), emoji, NUL, and empty strings all passed the tokenizer/embedding path;
• an isolated high surrogate and an isolated low surrogate both reproduced the same TextEncodeInput exception;
• removing only that surrogate from the embedding copy made the same inputs pass again;
• putting one surrogate-containing sentence into a SemanticSplitterNodeParser(buffer_size=1, ...) document reproduced the failure there as well.

That does not establish that your corpus contains a surrogate. The same generic error can have other causes; for example, vLLM issue #40292 shows the same TextEncodeInput error caused by a wrong input type rather than malformed Unicode.

But given your traceback and the fact that docs[0] works, I think a corpus-wide surrogate/strict-UTF-8 scan is a very cheap first test before changing parsers or models.

The first thing I would try

This does not perform any embeddings and does not require manually inspecting every paper:

def find_surrogates(text):
    return [
        (i, f"U+{ord(ch):04X}")
        for i, ch in enumerate(text)
        if 0xD800 <= ord(ch) <= 0xDFFF
    ]

bad_docs = []

for doc_i, doc in enumerate(docs):
    text = doc.text
    hits = find_surrogates(text)

    try:
        text.encode("utf-8", errors="strict")
        utf8_error = None
    except UnicodeEncodeError as e:
        utf8_error = e

    if hits or utf8_error is not None:
        bad_docs.append(doc_i)

        print(f"\nDocument {doc_i}")
        print("surrogates:", hits[:20])

        if utf8_error is not None:
            p = utf8_error.start
            print("UTF-8 error:", utf8_error)
            print(
                "context:",
                ascii(text[max(0, p - 80): p + 80])
            )

print("\nBad documents:", bad_docs)

If that reports a hit, I would not modify the original parsed text. I would keep the raw/source representation intact and make a separate embedding-safe representation.

Conceptually:

PDF / parsed source
        |
        +--> raw text kept unchanged
        |
        +--> embedding-input validation/sanitation
                  |
                  +--> SemanticSplitter
                  +--> embeddings
                  +--> vector index

That keeps “making the embedder accept the input” separate from “what the scientific document actually says.”

If the offending characters really are isolated surrogates, one narrow option for the embedding copy is:

import re

_SURROGATES = re.compile(r"[\ud800-\udfff]")

def embedding_text(text):
    return _SURROGATES.sub("", text)

Stripping is not the only possible policy; replacement or quarantine may be preferable depending on the corpus. The important part is that I would apply it only after confirming the condition, and only at the embedding boundary rather than destructively rewriting all scientific text.

The Hindsight project eventually implemented centralized sanitation for this same class of failure; their current code also removes surrogate code units before downstream processing.

If the scan finds nothing, I would follow this branch instead:

all-document scan clean
        |
        v
run SemanticSplitter one Document at a time
        |
        v
identify the first failing Document
        |
        v
identify the failing combined_sentence / embedding batch
        |
        +--> direct SentenceTransformer/tokenizer also fails
        |        |
        |        +--> inspect exact string + package versions
        |             + tokenizer/version-specific edge case
        |
        +--> direct SentenceTransformer/tokenizer succeeds
                 |
                 +--> investigate the LlamaIndex wrapper /
                      batching / splitter path

So there is still a useful next step even if the surrogate hypothesis is wrong.

Why I think the input itself is worth checking first

The relevant point is that this exception is not saying “the model does not know this character.”

The Hugging Face tokenizers API accepts text inputs according to its TextEncodeInput contract. The Python fast-tokenizer binding ultimately has to turn the Python input into Rust strings.

An unpaired surrogate is an unusual case because Python can hold it inside a str, for example:

x = "bad \udce2 text"

print(type(x))
# <class 'str'>

x.encode("utf-8")
# UnicodeEncodeError: surrogates not allowed

So a value can look like an ordinary Python str to upstream Python code while still failing when it crosses into a component expecting valid Unicode/UTF-8 text.

That also explains why a pickle round-trip does not rule this out. Pickle preserves Python objects; it is not a UTF-8 validity check. A Python string containing an isolated surrogate can be pickled and unpickled with that code point still present.

This is also consistent with the low-level path in the Rust-backed tokenizer rather than with a normal vocabulary miss.

For an ordinary valid but unknown character, a BERT tokenizer can tokenize it as an unknown token; that is a different situation from failing to construct a valid text input in the first place.

The BGE model card explicitly supports SentenceTransformers usage, so I would not interpret this exception by itself as evidence that bge-small-en-v1.5 fundamentally cannot handle scientific text.

Why docs[0] working is a useful clue

In the current LlamaIndex implementation, SemanticSplitterNodeParser processes documents individually.

Roughly:

doc.text
  -> sentence_splitter(...)
  -> sentence strings
  -> neighboring sentences combined
  -> get_text_embedding_batch(...)

The implementation builds a combined_sentence for every sentence by adding the configured neighboring sentence buffer, then embeds those combined strings.

So if your installed version behaves similarly, this observation:

splitter.get_nodes_from_documents([docs[0]])  # works
splitter.get_nodes_from_documents(docs)       # fails

fits quite naturally with a later document containing a content-dependent trigger.

It is not proof, because your installed LlamaIndex version may differ from current main, but it makes “the corpus is simply too large” less compelling than “something in a later document is different.”

With buffer_size=1, one bad sentence can also appear in several embedding inputs:

previous + BAD
previous + BAD + next
BAD + next

So when debugging this, the most precise object to inspect is eventually the exact combined sentence passed to the embedder, not only the original sentence.

I would also treat embed_batch_size=1 as a diagnostic tool rather than a fix. In my controlled test, a batch containing a lone surrogate failed with both batch size 1 and batch size 10.

A slightly more targeted isolation path if the initial scan is clean

If the raw-document scan finds no invalid UTF-8/surrogates, you can isolate the first failing document without inspecting them manually:

bad_doc_i = None
bad_doc = None

for i, doc in enumerate(docs):
    try:
        splitter.get_nodes_from_documents([doc])
    except Exception as e:
        bad_doc_i = i
        bad_doc = doc
        print("First failing document:", i)
        print(type(e).__name__, e)
        break

At that point I would avoid changing the whole ingestion pipeline.

Instead, use that one document as the reproducer.

If necessary, temporarily use embed_batch_size=1, or reproduce the splitter’s sentence-window construction, and identify the exact string that fails.

Then test that exact string directly:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

model.encode([candidate_text])

If that fails identically, LlamaIndex is probably only exposing a lower-level input/tokenizer problem.

If it succeeds, then the interesting boundary moves upward toward batching, wrapper behavior, or the exact versions of:

llama-index-core
llama-index-embeddings-huggingface
sentence-transformers
transformers
tokenizers

That would also give you a much smaller reproducer for a GitHub issue if one is needed.

I would avoid globally normalizing or removing scientific symbols

I would be careful with a workaround such as:

unicodedata.normalize("NFKC", all_text)

or converting every non-ASCII character to ASCII.

That may make some pipelines easier to process, but it mixes two different jobs:

1. removing malformed input that cannot cross the tokenizer boundary;
2. changing valid scientific notation.

The Unicode Consortium’s Unicode Normalization Forms specification (UAX #15) explicitly warns that NFKC/NFKD should not be blindly applied to arbitrary text because compatibility normalization removes distinctions that may carry semantic information.

For example, compatibility normalization can collapse superscripts/subscripts:

i⁹ -> i9
i₉ -> i9

In scientific corpora those distinctions may matter.

In my small probe, NFKC also changed examples such as:

µ    -> μ
10⁻⁶ -> 10−6
H₂O  -> H2O
E₀   -> E0

None of that fixed an isolated surrogate anyway.

So if the immediate failure is malformed Unicode, I think narrow validation/sanitation at the embedding boundary is a safer design than general-purpose normalization of the source corpus.

There is also a separate scientific-PDF problem worth keeping distinct

Your intuition that scientific characters may be involved is still relevant in another sense.

Even after the tokenizer crash is fixed, scientific PDFs can contain a separate class of extraction errors where the extracted Unicode is syntactically valid but semantically wrong.

A useful example is this PyMuPDF discussion, where a scientific table visually containing Greek letters and superscript/subscript notation extracted with errors such as:

λ -> l
ε -> 3
minus sign -> missing / U+FFFD / another value

The maintainer explains that some PDF fonts simply do not contain a complete reverse mapping from the displayed glyph back to the intended Unicode character.

That means this:

parser produced valid Python text

does not necessarily imply:

the extracted scientific text is semantically faithful

LANL’s experimental poppler-science project is interesting for exactly this reason. It is specifically aimed at extracting Unicode, superscripts/subscripts, and scientific PDF structure more faithfully for search/RAG use cases.

Their README gives a particularly good example:

displayed:  0.15 µM
extracted:  0.15 mM

Both strings are perfectly valid text and perfectly plausible scientific units, but they differ by a factor of 1000.

I would therefore treat this as a second, independent quality-control layer:

Layer A: Can the text safely reach the tokenizer?
         -> your current crash

Layer B: Does the extracted text still mean what the PDF displays?
         -> scientific glyph/unit fidelity

Layer C: Does the chunking/retrieval strategy work well?
         -> RAG quality

Fixing Layer A does not require redesigning B or C.

For Layer B, a tiny set of corpus-specific canaries may be more useful than reprocessing everything:

µM / mM
λ
ΔG
10⁻⁶
cm⁻¹
H₂O
E₀
superscript/subscript isotope notation

Compare a small sample against what is visibly present in the source PDF. If those are preserved, that gives more confidence than merely checking that parsing completed without an exception.

One later check: long combined sentences

This is not a likely explanation for the TextEncodeInput exception, so I would not start here.

But after the immediate error is fixed, it may be worth checking unusually long extracted sentences.

BAAI/bge-small-en-v1.5 has a maximum sequence length of 512 tokens according to its model configuration/tokenizer configuration.

Semantic splitting embeds the neighboring-sentence windows used to decide breakpoints. If a PDF parser accidentally produces enormous “sentences” because punctuation or reading order was lost, those windows can be truncated even though nothing crashes.

That would be a chunk-quality problem, not the present input-type failure, but it is an easy thing to inspect later if semantic boundaries still look strange.

So my default path here would be:

1. Keep the current overall RAG goal and pipeline.
2. Scan doc.text for invalid UTF-8 / surrogate code points.
3. If found, preserve raw text and sanitize only the embedding view.
4. If not found, isolate the first failing document and exact embedding input.
5. Only then decide whether this is a tokenizer/version issue,
   a LlamaIndex-wrapper issue, or something parser-specific.
6. Separately validate a few scientifically meaningful symbols/units
   so a crash workaround does not silently damage retrieval quality.

That seems cheaper and more informative than starting over with another embedding model or another PDF parser, and it gives a fairly clean branch either way.