Call rust function in python

🤗Tokenizers
1

I want to write a rust function that takes the python tokenizer (the tokenizer might be set in python code). for example,
define this in rust

fn foo(tokenizer: PyTokenizer, ...) { ... }

and call it in python

tokenizer = AutoTokenizer.from_pretrained(...)
# set tokenizer, e.g. pad_id
foo(tokenizer, ...)

Is there any guide to achieve this

2 Likes
2

There are a few ways to achieve what you’re wanting to do. I’d recommend PyO3 & Maturin.

Implementation can seem a bit convoluted if you don’t know what you’re doing. If that’s the case then check out rustimport it allows you to skip the complicated PyO3 implementation but still import Rust files or crates from Python, and it generates the pyo3 binding code automatically for you. BUT you give up some of the more granular control you have by building the bindings yourself.

1 Like