"SGLang Frontend Language" (https://docs.sglang.ai/frontend/frontend.html) provides a relatively minimal interface that I find acceptable.

e.g. tool use:

@function
def tool_use(s, question):
    s += assistant(
        "To answer this question: "
        + question
        + ". I need to use a "
        + gen("tool", choices=["calculator", "search engine"])
        + ". "
    )

    if s["tool"] == "calculator":
        s += assistant("The math expression is: " + gen("expression"))
    elif s["tool"] == "search engine":
        s += assistant("The key word to search is: " + gen("word"))

and images:

@function
def image_qa(s, image_file, question):
    s += user(image(image_file) + question)
    s += assistant(gen("answer", max_tokens=256))


image_url = "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
image_bytes, _ = load_image(image_url)
state = image_qa(image_bytes, "What is in the image?")
print_highlight(state["answer"])

And the server has some additional endpoints too (https://docs.sglang.ai/backend/native_api.html) that may make certain tasks easier (e.g. /clasify, /v1/rerank).