DeepSeek发布开源并行文件系统3FS(Fire-Flyer File System),专为现代SSD和RDMA网络优化。180节点集群可实现6.6 TiB/s聚合读取吞吐量,25节点GraySort测试达3.66 TiB/min,单节点KVCache查找峰值超40 GiB/s。采用分离式架构与强一致性语义,支持训练数据预处理、检查点存取及V3/R1推理的KVCache查找。同步开源Smallpond数据处理框架。
🚀 Day 5 of #OpenSourceWeek: 3FS, Thruster for All DeepSeek Data Access
Fire-Flyer File System (3FS) - a parallel file system that utilizes the full bandwidth of modern SSDs and RDMA networks.
⚡ 6.6 TiB/s aggregate read throughput in a 180-node cluster ⚡ 3.66 TiB/min throughput on GraySort benchmark in a 25-node cluster ⚡ 40+ GiB/s peak throughput per client node for KVCache lookup 🧬 Disaggregated architecture with strong consistency semantics ✅ Training data preprocessing, dataset loading, checkpoint saving/reloading, embedding vector search & KVCache lookups for inference in V3/R1
📥 3FS → https://github.com/deepseek-ai/3FS ⛲ Smallpond - data processing framework on 3FS → https://github.com/deepseek-ai/smallpond