How Libraries Run Rust Inside Python (with PyO3) — Bob Belderbos

Every time you validate data with Pydantic v2, the data-validation library most Python apps reach for, a Rust extension does the work. Its core, pydantic-core, is built with PyO3, the same toolchain we’ll use here.

This post builds that same kind of bridge, small enough to read in one sitting: a JSON parser written in Rust, exposed to Python, so you can import it like any other package. The last step, turning the Rust result into Python objects, is the one to understand before you port anything: for a parser like this, it can cost more than the parsing itself.

The four steps from Rust to import

Getting Rust code into Python takes four steps:

1. Write a normal Rust module.
2. Annotate it with PyO3 macros.
3. Let maturin compile and install it.
4. Import the result.

#[pyfunction] and #[pymodule] are the two Rust macros that do the wiring. A Rust attribute macro is close to a Python decorator: it rewrites the function it sits on, here adding the glue that lets Python call it and handles the type conversions and reference counting at the boundary.

Maturin then compiles the crate to a shared library (.so, .dylib, .dll) and drops it into your virtual environment, so import just works. I walk through this whole setup, from cargo new to the first import, in How to run Rust in Python with PyO3 and Maturin.

That first tutorial returns a single number. This one picks up where it left off, because the interesting part starts once you return a structure instead of a scalar.

The parser produces a Rust value first

The structure this parser returns is a JSON tree, and it’s the running example for the rest of this post. In our Python to Rust cohort, students spend six weeks writing a JSON parser from scratch in Rust, a hand-rolled tokenizer and recursive-descent parser with no serde, then expose it to Python through PyO3. Josh’s version beat CPython’s C json module on real-world fixtures; Jochen’s ran up to 3.5x faster than the Python version.

The public reference implementation, the clean version students start from, is the code I’ll walk through here.

The parser produces a plain Rust enum. A Rust enum holds one of several shapes, and each variant can carry data, so it maps a JSON tree cleanly:

pub enum JsonValue 🔥

That tree lives entirely in Rust. Python never sees it. The PyO3 layer is a thin adapter on top.

Exposing one function

Exposing a function to Python takes two lines:

#[pyfunction]
fn parse_json<'py>(py: Python<'py>, input: &str) -> PyResult<Bound<'py, PyAny>> 🔥

For a Python reader, the signature is the most interesting part:

• py: Python<‘py> is a token representing access to the Python interpreter and is what you pass to PyO3 APIs that need access to Python objects. On traditional Python builds, this access is associated with holding the GIL. PyO3 hands it to you and you pass it along wherever you touch a Python object.
• Bound<‘py, PyAny> is a handle to a Python object of any type, the Rust side of what you’d think of as a PyObject.
• PyResult is Result: return the value, or an error PyO3 raises as a Python exception.
• ? propagates that error. If parse fails, the function returns early and Python sees an exception; otherwise it unwraps the JsonValue and moves on.

So parse(input)? does the real work, and .into_pyobject(py) builds the Python objects the caller asked for. That last call is where the cost lives: it has to create Python objects for the nodes in the tree, and on a large document that can add up to more work than the parse itself.

The return trip is the expensive part

Here is why that conversion is not free. .into_pyobject walks the entire JsonValue tree and rebuilds it as native Python objects: a dict per object, a list per array, a float or str per leaf. You provide that translation by implementing the IntoPyObject trait, which PyO3 calls to convert a Rust value into a Python one:

impl<'py> IntoPyObject<'py> for JsonValue {
    fn into_pyobject(self, py: Python<'py>) -> Result<Self::Output, Self::Error> {
        match self {
            JsonValue::Null => Ok(py.None().into_bound(py)),
            JsonValue::Number(n) => Ok(n.into_pyobject(py)?.to_owned().into_any()),
            JsonValue::Object(obj) => 💬

        }
    }
}

A document with 100,000 values means on the order of 100,000 Python objects being created at the boundary, all after parsing is completely done. On a large document this materialization loop, not the parsing, can dominate the end-to-end time.

Errors cross the boundary the same way

The return value is not the only thing that has to translate. A parse failure is a typed Rust error, and Python wants an exception. One From impl, the trait Rust uses to convert one type into another, lets ? do the work:

impl From<JsonError> for PyErr {
    fn from(err: JsonError) -> PyErr {
        match err {
            JsonError::UnterminatedString { position } => PyValueError::new_err(
                format!("Unterminated string starting at position {position}")
            ),

        }
    }
}

Now malformed input raises a ValueError carrying the offset where parsing broke. The file-reading path gets the same treatment for free: std::io::Error already converts to the matching Python exception, so a missing path raises FileNotFoundError.

The caller gets Python semantics without the Rust layer leaking through.

What this means for your own port

If the Rust function you’re porting returns a scalar, port it and move on. The boundary is usually small enough to ignore.

If it returns a large structure, the conversion is your real cost, and it is the next thing to optimize once the parser itself is fast. Preallocating the PyDict can help at the margins, but the bigger win is architectural: don’t materialize the whole tree if the caller won’t touch all of it. Hand back a lazy, Rust-backed view and build Python objects on demand.

So when you reach for PyO3, profile the boundary, not just the algorithm. Getting Rust to run fast is the easy half. What you build on the way out, the trip from Rust values to Python objects, is the half that decides whether the port was worth it.