Great to see the community pushing MiniCPM5-1B inference further!🥳
@VukRosic99 has open-sourced an FP8 serving optimization that achieves significant speedups on RTX 3060 without modifying the official checkpoint. The reproducible benchmarks and practical exploration of efficient local inference are great contributions to the community.👍
We would love to see more results across different NVIDIA GPUs and workloads.😍
My AI startup made an LLM nearly 80% faster. An autonomous research system took MiniCPM5-1B from about 165 to 287 tokens/second on an RTX 3060. I show what work...