Public Repository

Last pushed: 3 months ago
Short Description
babymumu/fastdht是一个搭建在centos上的分布式内存高性能哈希表。
Full Description

FastDHT介绍

FastDHT 是一个高性能的分布式哈希系统 (DHT) ,使用 Berkeley DB 做数据存储,使用 libevent 做网络IO处理,提供 Java 版的客户端接口包。适合用来存储用户在线、会话等小数据量信息。

FastDHT存储Key Value Pair支持两种存储方式:缓存方式的MPOOL和持久存储方式的BDB。Key包括三部分:Namespace, ObjectID和Key。 Key可设置过期时间,自动清除过期数据.Server端划分group,同group数据互相备份,并且可自动压缩binlog.服务端可使用单线程,多线程模式。

FastDHT一些特性:

虚拟farm,便于扩容;

分布式算法client端实现,不需要中心服务器;

二进制通信协议,支持Proxy;

使用libevent,异步IO方式,支持大并发;

自动failover;

支持长连接。

FastDHT集群由一个或多个组(group)组成,每个组由一台或多台服务器组成,同组服务器上存储的数据是相同的,数据同步只在同组的服务器之间进行。组内各个服务器是对等的,对数据进行存取时,可以根据key的hash值来决定使用哪台服务器。数据同步采用推(Push)的方式,由源服务器主动将数据同步到本组的其他服务器。FastDHT集群由一个或多个组(group)组成,每个组由一台或多台服务器组成,同组服务器上存储的数据是相同的,数据同步只在同组的服务器之间进行。组内各个服务器是对等的,对数据进行存取时,可以根据key的hash值来决定使用哪台服务器。数据同步采用推(Push)的方式,由源服务器主动将数据同步到本组的其他服务器。

FastDHT由客户端决定应该选择哪台服务器,为例避免查表,应该根据key的hash code来选择服务器,算法描述如下:
计算出key的hash值(hash_code)

group_index = hash_code % group_count

new_hash_code = hash_code高16位和低16位互换

server_index = new_hash_code % 组内server_count

计算server_index和group_index时使用了不同的hash code,是因为如果group_count和组内server_count相等,例如都等于2,那么对于一个组来说,任何key值都将选中其中一台固定的服务器(server_index == group_index)。
FastDHT is a high performance distributed hash table, it can be used as a cache system or a persistent storage for key value pairs.

FastDHT is a high performance distributed hash table (DHT) which based key value pairs. It can store mass key value pairs such as filename mapping, session data and user related data.

FastDHT uses the Berkeley DB as data storage to support mass data and libevent as network IO to support huge connections. FastDHT implements data replication by it's binlog file, so it only uses the basic storage function of the Berkeley DB.

FastDHT cluster composes of one or many groups and a group contains one or more servers. The data on a group is same and data is synchronized only among the servers of the same group. The servers of a group are coordinative, so which server is selected to access according to the key's hash code. The source server pushes the data to other servers in the group.

The client of FastDHT decides which server to be selected. When select which server to access, we use the key's hash code to avoid lookuping the constrast table. The step of the algorithm is: calculate the hash code of the key group_index = hash_code % group_count new_hash_code = (hash_code << 16) | (hash_code >> 16) server_index = new_hash_code % server_count_in_the_group

In FastDHT, the key contains three fields: namespace, object ID and key name. These three concepts are like those of the database system: namespace vs database name object ID vs table name * key name vs field name The purpose of namespace is resolving the probable data conflict between multiple users such as the different applications or products. The purpose of object ID is convenient for organization and management of object related data such as user's data and increasing the whole performance. The sytem is more flexible as namespace and object ID are imported. These two fields can be empty in some cases. The input of the hash function (it's result is hash code) is: If namespace and object ID are not empty, the concatenation of these two fields, or the key name.

FastDHT imports the concept of logic group which is used to avoid re-hashing when the system scales. A physical group is composed of one or more real servers. It can contains one or more logic groups. A FastDHT server supports several logic groups and the data of a logic group is stored to a single Berkeley DB file. This design provides more convenience for scaling. We can use a larger number of logic groups at the beginning. When system scaled, one or more logic groups are migrated to the new physical group(s). All things we need to do are: copy the Berkeley DB data file to the new servers change the config file restart the programs of the FastDHT servers restart the programs of the FastDHT clients if necessary

FastDHT supports timeout and every key has timeout attribute. FastDHT can be used to store session data which is more efficient and more simple than the traditional database.

文章推荐 https://my.oschina.net/babymumu/blog/916324

Docker Pull Command
Owner
babymumu

Comments (0)