[{"_1":2,"_159":-5,"_160":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_138":139},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":11,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_129":-5,"_130":11,"_131":21,"_132":11,"_133":-5,"_134":11,"_135":-5,"_136":137},"canonicalUrl","https://hub.docker.com/r/freignat91/agrid/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","freignat91","reponame","agrid","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":11,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2016-12-26T11:09:00.780727Z","description","High available file storage designed to be easy to use and scale. ","is_private","last_updated","2017-08-31T22:16:39.445333Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",2942,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","Purpose\n-----------\n\nAgrid is a high available file storage service. Agrid can handle from 1 to several hundred storage nodes. Each file are cut and spread on nodes to be stored on high available way. Files can be saved with associated metadata and retrieve with their metadata.\n\nClients can use Command line interface (CLI) or Go API to store and retrieve files, the Go API allows also direct file manipulations create/open, read/write, seek, close, on the cluster\n\nClients can monitor file transfers on a separated process\n\nAgrid is a docker service. It enough to pull its image freignat91/agrid:latest or build it using make build and create the docker service to use it on a swarm cluster. It can be scaled using docker service scale command.\n\nAgrid uses grpc protocol for communication between nodes and between nodes and clients. Under 20 nodes, all nodes are completely connected one to each other, up to 20 nodes Agrid create a grid and the communication between nodes are not direct anymore.\n\nAgrid use Ant like behavior to found the shortest path between two nodes. The path are dynamically adapted regarding the nodes workload to stay the shortest in term of time. The grid become more efficient while it is used. see:\n\nhttps://github.com/freignat91/agrid/tree/master/docs\n- Agrid-grid-building.pptx\n- Agrid-Ant-net.pptx\n\nConfiguration using System Variables:\n---------------------------------------------------\nGRPCPORT: grpc server port used by nodes\nNB_DUPLICATE: number of time a file is replicated in the cluster when it is stored: default 3\nNB_DUPLICATE_ACK: number of acknowledge replications before a file is considered stored and acknowledge the client: default 1\nNB_LINE_CONNECT: number of \"line\" type connection in grid: default 0 means computed automatically\nNB_CROSS_CONNECT: number of \"cross\" type connection in grid: default 0 means computed automatically\nBUFFER_SIZE: messages waiting to be treated buffer size: default 10000\nPARALLEL_SENDER: maximum number of messages to be sent treated in parallel: default 100\nPARALLEL_RECEIVER: maximum number of received messages treated in parallel: default 100\nDATA_PATH: path in container where file data is stored: default: /data (should be mapped on host using mount docker argument (--mount type=bind,source=/[hostpath],target=/data)\n\nInstall\n--------\nDocker 1.12.3 min should be installed\npull this image or rebuild it:\n - clone the git project: https://github.com/freignat91/agrid \n - execute make install to build the agrid command line executable\n - execute make build to create a image freignat91/agrid:latest\n\nagrid can't be used as a single container, it needs to be started as a service;\n\ncreate agrid service, for instance with 5 nodes, using a publish port 30103 and network aNetwork\n[dataPath] should exist on the host, it contains all the data saved by agrid\n\ndocker service create --network aNetwork --name agrid \\\n --publish 30103:30103 \\\n --mount type=bind,source=[dataPath],target=/data \\\n --replicas=5 \\\n freignat91/agrid:latest\n\nResilience\n--------------\n\nFor resilience reason, it's better to have a separated disk file system for each node (each node on its own VM), but for test reason it's possible to have several nodes on the same file system or have architecture with several nodes on several VMs, but the used mermory on file retrieve command will be higher than normal.\n\nNode crash\n---------------\n\nIf a node crash (agrid itself, or disk file system failure or VM failure), docker will restart the node. When the new node restart, it will try to get it's previous file system or ask the other nodes to resend the blocks he handles (this last part is targeted for 0.1.5 version)\n\nScale out\n-------------\n\nTo scale out, it's enough to use docker service scale agrid=xxx command. Agrid will recompute its grid recreating all the node connections accordingly to the number of nodes\n\nScale in\n----------\nTo scale in, it's enough to use docker service scale agrid=xxx command. Agrid will recompute its grid recreating all the node connections accordingly to the number of nodes. Warning, to do not lose files, it's important to scale in with a difference between the number of nodes lower than the NB_DUPLICATE parameter and let time to Agrid to reorganize the files blocks between each scale command\n\nGrid simulation\n--------------------\n\nTo simulate nodes connections using different parameters as, node number, line connections, cross connections,and see the grid topology, use the cli command:\n\nagrid grid simul [nodes] <--line xx > <--cross yy>\n\n[nodes] the number of nodes\n<--line xx> optionally: xx the number of line connections\n<--cross yy> optionally: yy the number of cross connections\nthis command as not effect on the real cluster grid connections.see (./docs/Agrid-grid-building.pptx, ./docs/Agrid-Ant-net.pptx)\n\nUsers\n--------\nAgrid use a \"common\" file space by default, everyone can access to this space. Even if files can be encrypted with a specific key, it's possible to create a user to ensure more confidentiality. Create a user, create at the same time a dedicated file space that no one can access except the user. A user can see and act only on its own file space. To authenticate a user a token a given at user creation by the cluster, this token should be provided for all commands used with a user. It's also possible to provide your own token at user creation and use it for all commandes with this user.\n\nCLI\n-----\n\nAgrid command lines is implemented using the Agrid Go API\n\ncommon options:\n- verbose: display more informations messages\n- server: format addr1:port,addr2:port, ... list of the cluster servers (can list less servers than really in the cluster).\n\n# create a user\n\nagrid user create [username] <--token>\n\nCreate a user with its own file space in the cluster. This command return a token used to authenticate the user when executing any other command\n\n[username] the user name to create\n<--token> set the token for this user, without the token is computed by the server\n\n# remove a user\n\nagrid user remove [user] <--force>\n\nRemove a user. All files in its file space should have been removed first\n\n[user] the user to remove, format userName:token\n<--force> if this option exist then the user is removed with all its associated files, if not the user is removed only if its file space is empty.\n\n# store a new version of a file on cluster:\n\nagrid file store [source] [target] <--thread> <--key> <--user>\n\n[source]: the full pathname of the local file to store\n[target]: the full pathname of the file in the cluster\n<--thread number>: optionally: number of threads used to store the file (default 1), each thread open a grpc connection on a distinct node and handle a part of the file (all the block n if n%nbThread==threadNumber)\n<--key>: optionally: AES key to encrypt the file\n<--user userName:token>: to store on the usee file space, token is given at token creation\n<--meta key1=value1,key2=value2, ...>\nby default, the file is stored 3 times (config NB_DUPLICATE parameter default value)\n\n# retrieve a file from cluster\n\nRetrieve a file from cluster using duplicated blocks if some are missing\n\nagrid file retrieve [source] [target] <--thread> <--key> <--user>\n\n[source]: the full pathname of the file to get in cluster\n[target]: the full pathname of the file to write locally\n<--thread>: optionally: number of threads used to retrieve the file (default 1), each thread open a grpc connection on a distinct node and handle a part of the file (all the block n if n%nbThread==threadNumber)\n<--key>: optionally: AES key to encrypt the file\n<--user userName:token>: to store on the usee file space, token is given at user creation\n<--meta> metadata file name, default: no metadata file\n<--version nn> retrieve the version nn, default: the last version\n\n# list the files on the cluster\n\nagrid file ls [path] <--user>\n\n[path]: path name on the cluster to list, default /\n<--user userName:token>: to store on the user file space, token is given at user creation\n<--version> if exist, list all versions for all selected files\n\n# remove a file on the cluster\n\nagrid file rm [pathname] <-r> <--user>\n\n[pathname]: full pathname of the file to remove on the cluster\n<-r>: to remove a folder recursively\n<--user userName:token>: to store on the usee file space, token is given at user creation\n<--version nn> remove a specific version of the file, default: remove all the versions\n\n# monitor file transfers\n\nagrid file monitor <--type> <--user>\n\n<--user userName:token>: to store on the usee file space, token is given at user creation\n<--type xx> monitor only the file transfer with type=xx\n\n# list the cluster nodes\n\nagrid node ls\n\nping a cluster node\n\nagrid node ping |node]\n\n[node] the node name to ping\n\nAPI\n-----\n\nAgrid is usable using Go api API github.com/freignat91/agrid/agridapi\n\nUsage\n```\n import \"github.com/freignat91/agrid/agridapi\"\n ...\n api := agridapi.New(\"localhost:10315\")\n err, fileList := api.FileLs(\"/\")\n ...\n```\n\n# func (api *AgridAPI) UserCreate(name string, token string) (string, error)\n\nCreate a new user, return a token to authenticate the user Argument\n\nname: the user name to create\ntoken: if equal to \"\", the token is computed by server, if not it'sused as the user token.\n\n# func (api *AgridAPI) UserRemove(name string, force bool) error\n\nRemove a user Argument\n\nname: the user name to remove, format userName:token\nforce : if true the user is removed with all its associated files, if false the user is removed only if its file space is empty.\n\n# func (api *AgridAPI) SetUser(user string)\n\nSet the current user and authenticate it with the token, then every api function will be executed with this user \nArguments:\n\nuser: format userName:token, or \"\" to set the common user\n\n# func (api *AgridAPI) FileLs(folder string, version bool) ([]string, error)\n\nList the file stored on the cluster Argument:\n\nfolder: Folder under which the files are listed\nversion: if true list all the versions of the selected files\n\n# func (api *AgridAPI) FileStore(localFile string, clusterPathname string, meta *[]string, nbThread int, key string) (int, error)\n\nStore a file on the cluster, rerturn the version stored and error if happend. Arguments:\n\nlocalFile: pathname of the local file to store\nclusterPathName: pathname of the file on the cluster\nmetadata associated to the file and stored with the file\nnbThread: number of threads used to store the file (each thread open a distinc grpc connection)\nkey: AES key to encrypt the file on the cluster\n\n# func (api *AgridAPI) FileRetrieve(clusterPathname string, localFile string, version int, nbThread int, key string) (map[string]string, int, error)\n\nRetrieve a file from the cluster and return its metadata, its version and error if happend Arguments:\n\nclusterPathname: pathname of the file to get on the cluster\nlocalFile: pathname of the file to write locally\nversion: to retrieve a specific version, default: the last version is retrieved\nnbThread: number of threads used to store the file (each thread open a distinc grpc connection)\nkey: AES key to decrypt the file\n\n# func (api *AgridAPI) FileRm(clusterPathname string, version int, recursive bool) (error, bool)\n\nRemove a file on the cluster Arguments:\n\nclusterPathname: pathname of the file to remove on the cluster\nrecusive: if true remove all files under the clusterPathname\nversion: to remove a specifiv version, default: all versions are removed\n\n# func (api *AgridAPI) FileSetTransferEventCallback(fileType string, callbackFunction interface{}) error\n\nSet a function callback executed at each TransferEvent Arguments:\n\nfileType: callback only the transfer of files having type \"fileType\"\ncallbackFunction: function executed at each TransfertEvent, should have type: func(*AgridAPI.TransferEvent) error\n\n# func (api *AgridAPI) NodePing(node string, debugTrace bool) (string, error)\n\nPing a node Arguments:\n\nnode: node name to ping\ndebugTrace: if true, trace the message especially in the node logs.\n\n# func (api *AgridAPI) NodePingFromTo(node1 string, node2 string, debugTrace bool) (string, error)\n\nPing a node from another node Arguments:\n\nnode1: node name of the node which execute the ping\nnode2: targetted node name\ndebugTrace: if true, trace the message especially in the node logs.\n\n# func (api *AgridAPI) NodeSetLogLevel(node string, logLevel string) error\n\nSet the logLevel on a node(s) Arguments:\n\nnode: targetted node\nlogLevel: error, warn, info, debug\n\n# func (api *AgridAPI) NodeLs() ([]string, error)\n\nList the node of the cluster\n\n# func (api *AgridAPI) NodeLs() ([]string, error)\n\nList the node of the cluster\n\n# func (api *AgridAPI) CreateFile(name string, key string) *AFile, error\n\nCreate a new file on cluster and return an AFile instance Argument:\n\nkey: if key != \"\", encrypte the data with the key (AES256)\n\n# func (api *AgridAPI) OpenFile(name string, verison int, key string) *AFile, error\n\nOpen an existing file on cluster, move the current position to the end of file and return an AFile instance Arguments:\n\nversion: file version to retrieve, if 0, retrieve the last version\nkey: if key != \"\", encrypte the data with the key (AES256)\nFunctions on AFile (experimental part)\n\n# func (a *AFile) Write(data []byte) (int, error)\n\nWrite data in file at the current position\n\n# func (a *AFile) WriteAt(data []byte, at int64) (int, error)\n\nWrite data in file at the position 'at'\n\n# func (a *AFile) WriteString(data []byte) (int, error)\n\nWrite the string data in file at the current position\n\n# func (a *AFile) WriteStringAt(data []byte, at int64) (int, error)\n\nWrite the string data in file at the position 'at'\n\n# func (a *AFile) Read(data []byte) (int, error)\n\nRead data in file at the current position\n\n# func (a *AFile) ReadAt(data []byte, at int64) (int, error)\n\nRead data in file at the position 'at'\n\n# func (a *AFile) Seek(pos int64, whence int) (int64, error)\n\nMove the current possition to 'pos', relative to whence:\n\n0: from the begining of the file\n1: from the current position\n2: from the end of the file\n\n# func (a *AFile) Sync() error\n\nSave the memory data on the cluster\n\n# func (a *AFile) Close() error\n\nSave the memory data on the cluster and free all AFile instance ressources\n","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":21,"_83":11,"_84":11},"read","write","admin","user","storage_size",614814270,"orgData","recentTags",[91],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},"content_type","creator",1146218,"full_size",138981691,"id",7078523,"images",[101],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},"architecture","amd64","digest","sha256:da1c965583c83f38db4c234f8cae6569ea26078e94c2f8154f524ebaca4fdc69","features","","last_pulled","2026-09-24T18:44:43.011088623Z","last_pushed","2017-08-31T22:16:36Z","os","linux","os_features","os_version","size","variant","2017-08-31T22:16:35.735269Z","last_updater","last_updater_username","media_type","latest","repository",1135100,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"routes/_layout.r.$namespace.$reponame.tags",{"_38":140,"_147":148,"_136":157,"_158":11},{"_40":141,"_42":142,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":143,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":144,"_79":11,"_80":146,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":145},[78],{"_82":21,"_83":11,"_84":11},"tags",{"_149":64,"_150":-5,"_151":-5,"_152":153},"count","next","previous","results",[154],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":155,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},[156],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},[],"canDeleteTags","actionData","errors"]

freignat91/agrid

Tags for freignat91/agrid