Wav2vec2 for dementia screening based on spontaneous speech

wav2vec2 for dementia screening

Dementia is hard to diagnose. And there is no known cure, maybe because it too late by the time the diagnosis is confirmed. It is understood that the development of dementia commences 10 to 15 years before the symptoms first appear. Also, language, especially spontaneous speech, is a promising indicator/biomarker for diagnosing dementia and other cognitive disorders.

2. Language

The model will be trained on English audio.

3. Model

As far as I know, wav2vec2 is the best candidate.

4. Datasets

There is dementia classification from dementia bank. However, it is a binary classification dataset for dementia and no dementia.
To predict dementia 10 to 15 before the onslaught of symptoms, we would need longitudinal data on individuals who develop dementia. The only such available datasets I know is the Framingham heart study, and it is text only.
I have been building a list of public figures diagnosed with dementia and scrapping videos of youtube into different categories; after symptoms, two years before symptoms, five years before symptoms etc. Over the next week, I will build a streamlit app to extract 8 to 10 secs of audio files of the person of interest from the video.
Maybe we can use data from dementia bank or other sources for no dementia class??

Possible links to publicly available datasets include:

  • Dementia Bank
  • Google sheet with list of public figures with dementia and YouTube urls

5. Training scripts

TODO

6. Challenges

The dataset is too small, too noisy?
No dataset on time for no dementia

7. Desired project outcome

A proof of concept streamlit app that this works?

8. Reads

The following links can be helpful to better understand the project and
what has previously been done.

  • IBM efforts using Farmingham data for dementia prediction
4 Likes

Really like this project! Hope more people will be interested :slight_smile:

1 Like

I am hoping for the same :wink:

Actually I am interested to this project, unfortunately I am already in other projects. Maybe next time :slight_smile:

1 Like

Hi! Are you still looking for people to work with? I’m very interested in doing this project

1 Like

Yup, we are still open for people :wink:

Cool can I join?

2 Likes

Awesome two people is enough - let’s create the project!

Think this is a really cool & interesting project and we should be able to demo it well! Here I think it’ll actually make most sense to fine-tune a pretrained Wav2Vec2 model, maybe this one: facebook/wav2vec2-base-100k-voxpopuli · Hugging Face ?

There is currently no official fine-tuning script for Wav2Vec2, but it should be relatively easy to adapt the pretraining script: https://github.com/huggingface/transformers/pull/12271 or the official one in PyTorch.

Let’s create the project - very excited about this one :slight_smile:

1 Like

@shreyasgite cool project, let me know if I can give a hand. Is there a channel on discord for this project?

2 Likes

Great, added you to the official project @mattbui

I would love to be a part of this project. I actually used wav2vec2 embeddings for participating in a Alzheimer dementia challenge. The wav2vec2 embeddings didn’t work so well for the task but this might be because there wasn’t enough data. I’m more then happy to try this out again.

1 Like

I made a discord channel

1 Like

This is awesome @birgermoell and @mattbui

Thanks Partick for setting the vector:)

Hey @shreyasgite I would love to join and contribute to the project.
If it is late can I follow the group’s progress. Really want to get going with a small project in the area

How is the group coordinating?
Thanks

@asharma85 You can join the channel on Discord

It keeps showing me this when I try to join on discord

also @shreyasgite , make sure you right click on your server icon → invite and copy the link provided. You cannot access a discord server via the link in your browser.

1 Like

@asharma85 Flax-HuggingFace-Community-Week