llamacpp
๐ฆ Release: ๐๐ฅ๐ฅ๐๐ฆ๐๐ ๐๐ซ๐ฆ โจ
Recently I had the need to monitor all the locally available #Ollama instances. Of course, you can ๐ค๐ถ๐ณ๐ญ yourself, but why not have a tool in the history of btop/abtop ..
Only one ๐ด๐ฉ๐ฆ๐ญ๐ญ script utilizing ๐ค๐ถ๐ณ๐ญ, ๐ข๐ธ๐ฌ and ๐ซ๐ฒ. The name is the architecture: one llama is a pet, several across machines is a farm.
Tons of hotkeys to change update frequency, re-discovery of servers, probing their VRAM, cycling color themes. GitHub Actions ensure high quality code.
So give it a try and/or write a better version, #GPLv3 allows it: https://github.com/marcelpetrick/ollamaFarm/
Ollama Farmer out ๐ง๐ปโ๐พ๐ฆ
๐ ๐ฅ๐ฎ๐ป ๐ฎ ๐ฎ๐ณ๐ ๐ ๐ผ๐ฑ๐ฒ๐น ๐ผ๐ป ๐ฎ๐ป ๐ด๐๐ ๐๐ฃ๐จ.
๐ฐ๐ฌ ๐บ๐ถ๐ป๐๐๐ฒ๐. That’s all it took me to get Bonsai 27B running locally on an RTX A2000 Laptop GPU with just 8GB of VRAM.
Bonsai 27B is based on Qwen 3.6, but uses PrismML’s custom native ๐-๐๐ข๐ญ ๐๐จ๐ง๐ฌ๐๐ข format, reducing the model to just 3.9GB. A specialized ๐ฅ๐ฅ๐๐ฆ๐.๐๐ฉ๐ฉ fork implements custom CUDA kernels for the 1-bit inference path, making it possible to run the model directly on an 8GB GPU. The runtime exposes an OpenAI-compatible API, so existing tools and agents work without modification.
Performance is a different topic: 15 down to 9 tokens/s.
If you want to give it a try: find my notes and setup-scripts at GitHub: https://github.com/marcelpetrick/codingWithGPT/tree/master/bonsaiTestrun
