opencsghq/csghub-xnet repository overview

⁠xet-core Server

xet-core is a high-performance Go backend server designed for modern AI/ML storage workloads. It provides a robust, secure, and scalable storage foundation for managing large-scale model files, datasets, and unstructured data.

⁠Key Features

• Content-Addressed Storage (CAS): Each data chunk is identified by its SHA-256 hash, ensuring immutability, automatic integrity verification, and built-in deduplication for storage efficiency.

• Merkle Tree Integrity: Files composed of multiple chunks are verified via Merkle trees, enabling efficient full-file validation and partial verification without downloading entire files.

• FoundationDB Backend: Uses FoundationDB as the underlying distributed key-value store, providing strong consistency, high availability, and horizontal scalability.

• Security: All data is encrypted in transit via TLS, with strict authentication and authorization controls. Regular background audits proactively detect data corruption.

• Durability: Supports replication and erasure coding strategies to protect against data loss and ensure high availability across geographically distributed nodes.

• Monitoring: Integrated monitoring and alerting tracks node health, operation success rates, and audit results for rapid issue detection and resolution.