Sliding window for Long Documents

shreyans92dhankhar · June 20, 2022, 7:51am

Hi,

Is there any way to chunk a large document with left and right context? The default param in tokenizer provide only left context, is there a way we can provide right context also and predict for the central part not for the context.

Similar to the approach mention in this paper: https://arxiv.org/pdf/2011.06993.pdf

melhoushi · February 9, 2023, 10:34pm

return_over_flowing_tokens might help:

github.com

huggingface/notebooks/blob/main/examples/question_answering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X4cRE8IbIrIV"
   },
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "MOsHUjgdIrIW",

This file has been truncated. show original

Topic		Replies	Views
On-the-fly splitting for datasets with long texts 🤗Transformers	0	590	November 8, 2021
Token Classification: How to tokenize and align labels with overflow and stride? 🤗Tokenizers	3	5290	September 13, 2022
Huggingface distilbert-base-uncased-finetuned-sst-2-english runs out of ram with only a few kb? Beginners	0	306	May 12, 2022
DataCollator for list of inputs? Intermediate	0	377	November 1, 2022
Using of context_window in parameter Amazon SageMaker	0	637	June 23, 2023

Sliding window for Long Documents

Related Topics