I'm just putting down my thought as someone who had explored LLM couple of months back for internal projects. When it comes to observability, there are lot of things into factor. I'll to summarize them by each points:

• Cost: Like others mentioned, if you're someone who is deploying LLMs on cloud then you need to be vary careful about your cost. Most "out-of-the-box" remote LLMs have costing measurement available as part of the service you purchase (E.g OpenAI, Anthropic). But in other cases, if you are hosting the model on your own for many good reasons (like control, versioning, etc) then you need to measure there metrics on your own. So that boils down to resource monitoring as you pay by resource usage in cloud.
• Resource Monitoring: Treat LLM like an black-box API (it really is) and use the same resource monitoring stack like Prometheus, Grafana, Telegraf, Splunk, ELK, etc.
• AI-Metrics: These includes things like Tokens per second, Model Responses, User Prompt. There is a framework by Open Telemetry called OpenLit which is meant for instrumenting these metrices. They seem to support a large group of LLM related stacks including VectorDBs, Frameworks (langchain, llamaindex...), and LLM providers. So this would be something that I'd use to setup a monitoring solution stack for collecting AI metrics.

If you want to further evaluate the model responses (good/bad), then it would require you to build some mechanism to bring feedbacks into the loop (like collect user rating on the text output, store them with some request_id to map it with the model response).