https://github.com/src-d/datasets/tree/master/PublicGitArchive/pga-create
10K+
pga-create is the tool to generate the Public Git Archive.
The docker image is intended to be used in two different step through the commands select-repos and index-repos plus and additional step using borges to download the repositories.
The select-repos and index-repos commands will make usage of the volume mounted on /pga to persist the required data between steps and the final one containing the PGA.
select repos needs a volume mounted on /pga. This step will leave the processed data and the list of repositories to be included in the PGA in that volume.
$ docker run --rm --name pga-create_1 \
-v /mymounted/vol:/pga \
-e "STARS=1000" \
pga-create:test select-repos
The environment variable STARS configures the number of stars to filter repositories that will be included in the PGA list.
Using borges and the generated list of the repositories (/mymounted/vol/data/pga.list) in the previous step you can download the repositories. It's important that the bucket-size used by the borges consumer and in the next step are the same.
To generate the pga-index:
$ docker run --rm --name pga-create_1 \
-v /mymounted/vol:/pga \
--network host \
-e "PGA_VERSION=v2-test" \
-e "BUCKET_SIZE=2" \
pga-create:test index-repos
It's important to note two things: "BUCKET_SIZE=2" should be the same used by borges and in the mounted volume-v /mymounted/vol:/pga/ must have a /mymounted/vol/root-repositories where we have placed the repositories downloaded by borges.
After this step we will have two sets of data under the mounted volume:
# DATA generated by pga-create
$ ls /mymounted/vol/data
index.csv index_v2-test.csv languages.gz pga.list repositories.gz stars.gz
# PGA: repositories (siva files) and the `index_XXX.tar.gz`
$ ls /mymounted/vol/root-repositories
00 06 0c 12 18 1e 24 2a 30 36 3c 42 48 4e 54 5a 60 66 6c 72 78 7e 84 8a 90 96 9c a2 a8 ae b4 ba c0 c6 cc d2 d8 de e4 ea f0 f6 fc
01 07 0d 13 19 1f 25 2b 31 37 3d 43 49 4f 55 5b 61 67 6d 73 79 7f 85 8b 91 97 9d a3 a9 af b5 bb c1 c7 cd d3 d9 df e5 eb f1 f7 fd
02 08 0e 14 1a 20 26 2c 32 38 3e 44 4a 50 56 5c 62 68 6e 74 7a 80 86 8c 92 98 9e a4 aa b0 b6 bc c2 c8 ce d4 da e0 e6 ec f2 f8 fe
03 09 0f 15 1b 21 27 2d 33 39 3f 45 4b 51 57 5d 63 69 6f 75 7b 81 87 8d 93 99 9f a5 ab b1 b7 bd c3 c9 cf d5 db e1 e7 ed f3 f9 ff
04 0a 10 16 1c 22 28 2e 34 3a 40 46 4c 52 58 5e 64 6a 70 76 7c 82 88 8e 94 9a a0 a6 ac b2 b8 be c4 ca d0 d6 dc e2 e8 ee f4 fa index_v2-test.tar.gz
05 0b 11 17 1d 23 29 2f 35 3b 41 47 4d 53 59 5f 65 6b 71 77 7d 83 89 8f 95 9b a1 a7 ad b3 b9 bf c5 cb d1 d7 dd e3 e9 ef f5 fb
$ ls /mymounted/vol/root-repositories/index_v2-test.tar.gz
index_v2-test.tar.gz
Content type
Image
Digest
Size
21.1 MB
Last updated
about 7 years ago
docker pull srcd/pga-create