Easy way to log input/output in llama.cpp? (server and chat)
Posted by aayehh@reddit | LocalLLaMA | View on Reddit | 6 comments
Hi. I been trying to automatically log the inputs and outputs in the CLI and API webgui in llama.cpp. Looking for an efficient one.
pluckytree@reddit
Just a quick note here in case it helps anyone else...
llama.cpp's cli still doesn't seem to support logging the conversations to files. (Just logging internal events / debug info.) And I couldn't work out a way to pipe the output either (very strange).
My workaround... I use VSCodium (Visual Studio Code fork) to run llama.cpp / pi / other closer-to-model wrappers in terminals (also closer to project code when I need it but works nicely as a general chat interface too). Then you can use various extensions to auto-log terminal outputs (e.g. I use one called "TermLog").
aayehh@reddit (OP)
i just enabed verbose/debug output in llama.cpp and piping the logs into Grafana + Loki, as someone said here.
AdamDhahabi@reddit
For API logging:
Install Apache on your Windows machine and create a VHost with ProxyPass & ProxyPassReverse, now you access llama-server on the Vhost its port.
Inspect HTTP requests with SecAuditEngine (Apache httpd module) https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-Frequently-Asked-Questions-(FAQ)
How do I get ModSecurity to inspect request and response bodies? You need to set: SecRequestBodyAccess On / SecResponseBodyAccess On
Also this: SecRuleEngine DetectionOnly / SecAuditLogParts ABCIJDEFHZ has to contain C (request body) and/or E (response body)
Nepherpitu@reddit
Enable debug output for llamacpp and ask any llm how to setup grafana + loki, or graylog, or anything similar. Then ask llm how to forward output from llamacpp to logging stack. You will get infinite logging storage with search and fancy formatters.
Nepherpitu@reddit
I mean forwarding any cli application output to syslogs/graylog/etc. It takes 15 minutes to setup on windows, but everything is hardcoded. Just ask llm for something like "llamacpp.exe > logger.ps1" and adjust for your system and requirements.
aayehh@reddit (OP)
Im kinda new to this and chagpt been helping to do this with a ps1 script. Did lot of troubleshoot and still coudnt set up properly. Logging the cli chats was okay but the API logs doesnt work at all. Anyways thanks, ill look into what u said.