I am currently using Vertex AI’s Multimodal Embedding Model (https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings)
I was able to get the Image and Text examples running just fine using the Python SDK and POST, however, when I try to run the following Video example (from the above link), I get the following error
import vertexai
from vertexai.vision_models import MultiModalEmbeddingModel, Video
project_id = 'multimodal-project'
location = 'us-central1'
vertexai.init(project=project_id, location=location)
# Document metadata
video_path = 'gs://multi-assets/videos/dog_jumping_short.mp4' # small 1 MB 7 second video file
description = 'dogs jumping'
model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
video = Video.load_from_file(video_path)
embeddings = model.get_embeddings(
video=video
)
# Video Embeddings are segmented based on the video_segment_config.
print("Video Embeddings:")
for video_embedding in embeddings.video_embeddings:
print(
f"Video Segment: {video_embedding.start_offset_sec} - {video_embedding.end_offset_sec}"
)
print(f"Embedding: {video_embedding.embedding}")
print(f"Text Embedding: {embeddings.text_embedding}")
The error I get when running this sample code is related to Deadline
Traceback (most recent call last):
File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
return callable_(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Video embedding failed with the following error: Deadline"
debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.81.234:443 {grpc_message:"Video embedding failed with the following error: Deadline", grpc_status:16, created_time:"2024-05-23T15:53:15.429704-04:00"}"
Note that I have been able to embed contextual_text
and image
just fine with the same authentication, so I am fairly certain this has nothing to do with authentication even though the error says so.
I have also tried to POST using the following cURL, but I get the same error response Deadline
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://us-central1-aiplatform.googleapis.com/v1/projects/multi/locations/us-central1/publishers/google/models/multimodalembedding@001:predict"
Anyone know what I can do here?
P.S. I work arounded this using the base64 endpoint instead of the gcs endpoint, and it works, but it’s limited to videos no longer than 2 minutes (because the embedding splitting doesn’t work for base64).