Using the prompt to switch model

Hello,

I am new to gradio and tried to implement a way to switch models by identifying keyword inside the prompt

see

the problem seems to be that whatever is selected
it is ignored by gradio

Can someone explain, what is wrong in my code
and if this is feasible or not?

Thanks

Hi @blogclif, you can’t use gr.Interface.load() with a fn, you can only load a single model at a time. However, it is possible to do what you have in mind by noting the fact that an a model loaded with gr.Interface.load() can be used as a function. So you could do something like this:

models = [
 gr.Interface.load("model1name"),
 gr.Interface.load("model2name")
 ...
]

def prediction(model_choice, input):
  model = models[model_choice]
  output = model(input)
  return output = model

gr.Interface(fn=prediciton, inputs=[gr.Dropdown(["model1name", "model2name"], type="index"), gr.Textbox()], outputs=gr.Image()].launch()

This is just a code skeleton but let me know if it makes sense!

2 Likes

Thanks a lot for looking into it

I will try, I like the fact that the prompt was including the model in a text format ; that seemed really transparent, but having to select the model via a menu list is surely better, and more reliable

thank a lot

hi, this is a cool example using this approach app.py · Omnibus/maximum_diffusion at a451648ba3f1b4e498072f0e4a462d4c265fe713 from this project Maximum Diffusion - a Hugging Face Space by Omnibus currently down but I’ve ping @Omnibus to look at the error.

2 Likes

Thanks for the nudge on this, and I have that space up and running again if it helps out here.

@blogclif
I’m also new to Gradio, but I have found that sometimes using a Textbox as a toggle switch with something like Textbox.change(fn,in,out) can lend itself towards building complex decision trees. That and iterating a single event to multiple functions. I’m pretty sure that this is considered really hacky by “standards”, but as an old wise man once said, “it isn’t stupid if it works”.

This one has a lot of that going on, too:

2 Likes

Thanks a lot for the help, I am writing some more code
yesterday and today
let’s see what will work best.

I will most likely go in the direction of a dropdown menu ; as I understand now that extracting the model from the prompt is not accepted by gradio, because gradio needs two variables, and won’t accept the second variable after it has been called.

I guess, that was the misunderstanding I had at the very beginning

2 Likes

right on, if it saves time, I know that @anzorq had the model selection dropdown perfected on:

I couldn’t tell you how it works exactly, but it sure does.

@Omnibus

I did some more testing and found another user who had the interface I was looking for

there is a new problem now

This error is popping up

OSError: We couldn’t connect to ‘https://huggingface.co’ to load this model, couldn’t find it in the cached files and it looks like models/ItsJayQz/Marvel_WhatIf_Diffusion is not the path to a directory containing a model_index.json file.

for all the models

1 Like

I know that when using gr.Interface.load(“models/ItsJayQz/Marvel_WhatIf_Diffusion”) would probably work, because the “models/” prefix does something other than note a URL location.

https://huggingface.co/models/ItsJayQz/Marvel_WhatIf_Diffusion doesn’t exist because:
ItsJayQz/Marvel_WhatIf_Diffusion · Hugging Face is the models URL

I think just remove the “models/” prefix in your case when using:
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)

also, if you have any trouble with torch_dtype=torch.float16 on CPU change to torch_dtype=torch.get_default_dtype, which cured a problem for me.

[edit]
torch_dtype=torch.get_default_dtype()
is the correct syntax on that suggestion

1 Like

will try, here was the link to the other space that had the interface I was trying to have in the first place.

I left a comment in his discussion thread

Check this one out, it looks like it has all the elements you are using. This isn’t my code, I just modified it to work on CPU

1 Like

Feels like your solution above is working

removing the “models/” and using get_default_dtype

now in the log I see

Downloading: 71%|███████ | 2.47G/3.46G [00:35<00:18, 54.4MB/s]

it still shows an ‘Error’ in the end

and the logs are still scrolling … I can’t really get an error this time

I think I know all of the errors now, lol. Check the container logs, vs. build logs for the error when the program is running like that and see if there are clues.

This is what I have:

Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:

pip install accelerate

.
Traceback (most recent call last):
File “/home/user/.local/lib/python3.8/site-packages/gradio/routes.py”, line 321, in run_predict
output = await app.blocks.process_api(
File “/home/user/.local/lib/python3.8/site-packages/gradio/blocks.py”, line 1015, in process_api
result = await self.call_function(fn_index, inputs, iterator, request)
File “/home/user/.local/lib/python3.8/site-packages/gradio/blocks.py”, line 856, in call_function
prediction = await anyio.to_thread.run_sync(
File “/home/user/.local/lib/python3.8/site-packages/anyio/to_thread.py”, line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File “/home/user/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py”, line 937, in run_sync_in_worker_thread
return await future
File “/home/user/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py”, line 867, in run
result = context.run(func, *args)
File “app.py”, line 18, in TextToImage
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.get_default_dtype)
File “/home/user/.local/lib/python3.8/site-packages/diffusers/pipeline_utils.py”, line 708, in from_pretrained
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File “/home/user/.local/lib/python3.8/site-packages/transformers/modeling_utils.py”, line 2325, in from_pretrained
dtype_orig = cls._set_default_torch_dtype(torch_dtype)
File “/home/user/.local/lib/python3.8/site-packages/transformers/modeling_utils.py”, line 1102, in _set_default_torch_dtype
if not dtype.is_floating_point:
AttributeError: ‘builtin_function_or_method’ object has no attribute ‘is_floating_point’
Traceback (most recent call last):
File “/home/user/.local/lib/python3.8/site-packages/gradio/routes.py”, line 321, in run_predict
output = await app.blocks.process_api(
File “/home/user/.local/lib/python3.8/site-packages/gradio/blocks.py”, line 1013, in process_api
inputs = self.preprocess_data(fn_index, inputs, state)
File “/home/user/.local/lib/python3.8/site-packages/gradio/blocks.py”, line 923, in preprocess_data
processed_input.append(block.preprocess(inputs[i]))
IndexError: list index out of range
Traceback (most recent call last):
File “/home/user/.local/lib/python3.8/site-packages/diffusers/configuration_utils.py”, line 326, in load_config
config_file = hf_hub_download(
File “/home/user/.local/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py”, line 114, in _inner_fn
validate_repo_id(arg_value)
File “/home/user/.local/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py”, line 172, in validate_repo_id
raise HFValidationError(
huggingface_hub.utils.validators.HFValidationError: Repo id must use alphanumeric chars or ‘-’, '', ‘.’, ‘–’ and ‘…’ are forbidden, ‘-’ and ‘.’ cannot start or end the name, max length is 96: ‘’.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/user/.local/lib/python3.8/site-packages/gradio/routes.py”, line 321, in run_predict
output = await app.blocks.process_api(
File “/home/user/.local/lib/python3.8/site-packages/gradio/blocks.py”, line 1015, in process_api
result = await self.call_function(fn_index, inputs, iterator, request)
File “/home/user/.local/lib/python3.8/site-packages/gradio/blocks.py”, line 856, in call_function
prediction = await anyio.to_thread.run_sync(
File “/home/user/.local/lib/python3.8/site-packages/anyio/to_thread.py”, line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File “/home/user/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py”, line 937, in run_sync_in_worker_thread
return await future
File “/home/user/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py”, line 867, in run
result = context.run(func, *args)
File “app.py”, line 18, in TextToImage
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.get_default_dtype)
File “/home/user/.local/lib/python3.8/site-packages/diffusers/pipeline_utils.py”, line 459, in from_pretrained
config_dict = cls.load_config(
File “/home/user/.local/lib/python3.8/site-packages/diffusers/configuration_utils.py”, line 363, in load_config
raise EnvironmentError(
OSError: We couldn’t connect to ‘https://huggingface.co’ to load this model, couldn’t find it in the cached files and it looks like is not the path to a directory containing a model_index.json file.
Checkout your internet connection or see how to run the library in offline mode at ‘Installation’.


To get this error I left the model as ‘blank’

oh yeah, I know that one. The accelerate is not a big deal, and is probably not causing the error, but you can add “accelerate” to your requirements.txt and then “import accelerate” in app.py. I think the error is simply my advice to use torch_dtype=torch.get_default_dtype, try changing that to torch_dtype=torch.get_default_dtype() because I think it’s a function.

1 Like

@Omnibus
Oh well, this time the generating part never ends
there are no error it just won’t finish

I give up

I tried it this model "ItsJayQz/Marvel_WhatIf_Diffusion” and it worked. When I started I was 7/7 on the queue, but when I checked back some time later, it had completed, so it works.

Processing on normal CPU can take 15 mins per image if only one user is processing, and depending on the model. I’ve found that some models will just never complete, depending on the time of day, phase of the Moon, or other factors.

You can speed up processing by using Gradio.Interface.load(“models/Repo/Name”)
I have no idea where the processing is happening using this method, but I suspect it has something to do with Quantum Super-positioning, or Machine Elves maybe. The processing does not happen on the page running that code though as far as I can tell.

If you try everything, something will work!

1 Like

Thanks again, for your assistance,

I let it run for 3 hours and just closed the tab (after taking this screenshot)

the logs are showing a lot of downloads and sometime I see this

Fetching 16 files: 100%|██████████| 16/16 [02:48<00:00, 10.53s/it]
/home/user/.local/lib/python3.8/site-packages/transformers/models/clip/feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
warnings.warn(

if this would load the models once and for all?
that could really speed up the process

It looks like you are 1/1 on the queue, so there is a process ahead of you that is probably stuck on one of the models that are processing, but not succeeding or erroring out. You could eliminate the offending model, or make your queue larger using:

sandbox.queue(concurrency_count=10)
sandbox.launch()

This will allow 10 processes to run concurrently before enforcing the queue, but 10 processes will each be slower than 1 process running at a time. If there is a model that never completes or errors out, this will clog up the queue anyway eventually, so best to find the offending model and remove it from the options.

1 Like