\n\nThis demo does the full face recognition pipeline on every frame.\nIn practice, object tracking\n[like dlib's](http://blog.dlib.net/2015/02/dlib-1813-released.html)\nshould be used once the face recognizer has predicted a face.\n\nTo run on your system, after following the setup directions\nbelow, install the requirements for the web demo with\n`./install-deps.sh` and `sudo pip install -r requirements.txt`\nfrom the `demos/web` directory.\nThe application is split into a processing server and static\nweb pages that communicate via web sockets.\nStart the server with `./demos/web/server.py` and\nserve the static website with `python2 -m SimpleHTTPServer 8000`\nfrom the `/demos/web` directory.\nYou should now be able to access the demo from your browser\nat `http://localhost:8000`.\nThe saved faces are only available for the browser session.\n\n# Demo 2: Comparing two images\nThe [comparison demo](demos/compare.py) outputs the predicted similarity\nscore of two faces by computing the squared L2 distance between\ntheir representations.\nA lower score indicates two faces are more likely of the same person.\nSince the representations are on the unit hypersphere, the\nscores range from 0 (the same picture) to 4.0.\nThe following distances between images of John Lennon and\nEric Clapton were generated with\n`./demos/compare.py images/examples/{lennon*,clapton*}`.\n\n| Lennon 1 | Lennon 2 | Clapton 1 | Clapton 2 |\n|---|---|---|---|\n|

|\n\nThe following table shows that a distance threshold of `0.5` would\ndistinguish these two people.\nIn practice, further experimentation should be done on the distance threshold.\nOn our LFW experiments, the mean threshold across multiple\nexperiments is 0.71 ± 0.027,\nsee [accuracies.txt](evaluation/lfw.nn4.v1.epoch-177/accuracies.txt).\n\n| Image 1 | Image 2 | Distance |\n|---|---|---|\n| Lennon 1 | Lennon 2 | 0.310 |\n| Lennon 1 | Clapton 1 | 1.241 |\n| Lennon 1 | Clapton 2 | 1.056 |\n| Lennon 2 | Clapton 1 | 1.386 |\n| Lennon 2 | Clapton 2 | 1.073 |\n| Clapton 1 | Clapton 2 | 0.259 |\n\n# Demo 3: Training a Classifier\nOpenFace's core provides a feature extraction method to\nobtain a low-dimensional representation of any face.\n[demos/classifier.py](demos/classifier.py) shows a demo of\nhow these representations can be used to create a face classifier.\n\nThis is trained on about 6000 total images of the following people,\nwhich are the people with the most images in our dataset.\nClassifiers can be created with far less images per\nperson.\n\n+ America Ferrera\n+ Amy Adams\n+ Anne Hathaway\n+ Ben Stiller\n+ Bradley Cooper\n+ David Boreanaz\n+ Emily Deschanel\n+ Eva Longoria\n+ Jon Hamm\n+ Steve Carell\n\nThis demo uses [scikit-learn](http://scikit-learn.org) to perform\na grid search over SVM parameters.\nFor 1000's of images, training the SVMs takes seconds.\nOur trained model obtains 87% accuracy on this set of data.\n[models/get-models.sh](models/get-models.sh)\nwill automatically download this classifier and place\nit in `models/openface/celeb-classifier.nn4.v1.pkl`.\n\nFor an example, consider the following small set of images\nthe model has no knowledge of.\nFor an unknown person, a prediction still needs to be made, but\nthe confidence score is usually lower.\n\nRun the classifier on your images with:\n\n```\n./demos/classifier.py infer ./models/openface/celeb-classifier.nn4.v1.pkl ./your-image.png\n```\n\n| Person | Image | Prediction | Confidence |\n|---|---|---|---|\n| Carell |

| SteveCarell | 0.78 |\n| Adams |

| AmyAdams | 0.87 |\n| Lennon 1 (Unknown) |

| DavidBoreanaz | 0.28 |\n| Lennon 2 (Unknown) |

| DavidBoreanaz | 0.56 |\n\n# Cool demos, but I want numbers. What's the accuracy?\nEven though the public datasets we trained on have orders of magnitude less data\nthan private industry datasets, the accuracy is remarkably high\non the standard\n[LFW](http://vis-www.cs.umass.edu/lfw/results.html)\nbenchmark.\nWe had to fallback to using the deep funneled versions for\n152 of 13233 images because dlib failed to detect a face or landmarks.\nWe obtain a mean accuracy of 0.8483 ± 0.0172 with an AUC of 0.923.\nFor comparison, training with Google-scale data results in an\naccuracy of .9963 ± 0.009.\n\n![](images/nn4.v1.lfw.roc.png)\n\nThis can be generated with the following commands from the root `openface`\ndirectory, assuming you have downloaded and placed the raw and\ndeep funneled LFW data from [here](http://vis-www.cs.umass.edu/lfw/)\nin `./data/lfw/raw` and `./data/lfw/deepfunneled`.\n\n1. Install prerequisites as below.\n2. Preprocess the raw `lfw` images, change `8` to however many\n separate processes you want to run:\n `for N in {1..8}; do ./util/align-dlib.py data/lfw/raw align affine data/lfw/dlib-affine-sz:96 --size 96 &; done`.\n Fallback to deep funneled versions for images that dlib failed\n to align:\n `./util/align-dlib.py data/lfw/raw align affine data/lfw/dlib-affine-sz:96 --size 96 --fallbackLfw data/lfw/deepfunneled`\n3. Generate representations with `./batch-represent/main.lua -outDir evaluation/lfw.nn4.v1.reps -model models/openface/nn4.v1.t7 -data data/lfw/dlib-affine-sz:96`\n4. Generate the ROC curve from the `evaluation` directory with `./lfw-roc.py --workDir lfw.nn4.v1.reps`.\n This creates `roc.pdf` in the `lfw.nn4.v1.reps` directory.\n\n---\n\nIf you're interested in higher accuracy open source code, see:\n\n1. [Oxford's VGG Face Descriptor](http://www.robots.ox.ac.uk/~vgg/software/vgg_face/),\n which is licensed for non-commercial research purposes.\n They've released their softmax network, which obtains .9727 accuracy\n on the LFW and will release their triplet network (0.9913 accuracy)\n and data soon.\n\n Their softmax model doesn't embed features like FaceNet,\n which makes tasks like classification and clustering more difficult.\n Their triplet model hasn't yet been released, but will provide\n embeddings similar to FaceNet.\n The triplet model will be supported by OpenFace once it's released.\n2. [AlfredXiangWu/face_verification_experiment](https://github.com/AlfredXiangWu/face_verification_experiment),\n which uses Caffe and doesn't yet have a license.\n The accuracy on the LFW is .9777.\n This model doesn't embed features like FaceNet,\n which makes tasks like classification and clustering more difficult.\n\n# Help Wanted!\nAs the following table shows, the forefront of deep learning research\nis driven by large private datasets.\nIn face recognition, there are no open source implementations or\nmodels trained on these datasets.\nIf you have access to a large dataset, we are very interested\nin training a new OpenFace model with it.\nPlease contact Brandon Amos at [bamos@cs.cmu.edu](mailto:bamos@cs.cmu.edu).\n\n| Dataset | Public | #Photos | #People |\n|---|---|---|---|\n| [DeepFace](https://research.facebook.com/publications/480567225376225/deepface-closing-the-gap-to-human-level-performance-in-face-verification/) (Facebook) | No | 4.4 Million | 4k |\n| [Web-Scale Training...](http://arxiv.org/abs/1406.5266) (Facebook) | No | 500 Million | 10 Million |\n| FaceNet (Google) | No | 100-200 Million | 8 Million |\n| [FaceScrub](http://vintage.winklerbros.net/facescrub.html) | Yes | 100k | 500 |\n| [CASIA-WebFace](http://arxiv.org/abs/1411.7923) | Yes | 500k | 10k |\n\n\n# Visualizing representations with t-SNE\n[t-SNE](http://lvdmaaten.github.io/tsne/) is a dimensionality\nreduction technique that can be used to visualize the\n128-dimensional features OpenFace produces.\nThe following shows the visualization of the three people\nin the training and testing dataset with the most images.\n\n**Training**\n\n![](images/train-tsne.png)\n\n**Testing**\n\n![](images/val-tsne.png)\n\nThese can be generated with the following commands from the root\n`openface` directory.\n\n1. Install prerequisites as below.\n2. Preprocess the raw `lfw` images, change `8` to however many\n separate processes you want to run:\n `for N in {1..8}; do ./util/align-dlib.py align affine --size 96 &; done`.\n3. Generate representations with `./batch-represent/main.lua -outDir -model models/openface/nn4.v1.t7 -data `\n4. Generate t-SNE visualization with `./util/tsne.py --names ... `\n This creates `tsne.pdf` in ``.\n\n# Visualizing layer outputs\nVisualizing the output feature maps of each layer\nis sometimes helpful to understand what features\nthe network has learned to extract.\nWith faces, the locations of the eyes, nose, and\nmouth should play an important role.\n\n[demos/vis-outputs.lua](demos/vis-outputs.lua)\noutputs the feature maps from an aligned image.\nThe following shows the first 39 filters of the\nfirst convolutional layer on two images\nof John Lennon.\n\n![](images/nn4.v1.conv1.lennon-1.png)\n![](images/nn4.v1.conv1.lennon-2.png)\n\n\n# Model Definitions\nModel definitions should be kept in [models/openface](models/openface),\nwhere we have provided definitions of the [NN2](models/openface/nn2.def.lua)\nand [nn4](models/openface/nn4.def.lua) as described in the paper,\nbut with batch normalization and no normalization in the lower layers.\nThe inception layers are introduced in\n[Going Deeper with Convolutions](http://arxiv.org/abs/1409.4842)\nby Christian Szegedy et al.\n\n# Pre-trained Models\nPre-trained models are versioned and should be released with\na corresponding model definition.\nWe currently only provide a pre-trained model for `nn4.v1`\nbecause we have limited access to large-scale face recognition\ndatasets.\n\n## nn4.v1\nThis model has been trained by combining the two largest (of August 2015)\npublicly-available face recognition datasets based on names:\n[FaceScrub](http://vintage.winklerbros.net/facescrub.html)\nand [CASIA-WebFace](http://arxiv.org/abs/1411.7923).\nThis model was trained for about 300 hours on a Tesla K40 GPU.\n\nThe following plot shows the triplet loss on the training\nand test set.\nEach training epoch is defined to be 1000 minibatches, where\neach minibatch processes 100 triplets.\nEach testing epoch is defined to be 300 minibatches,\nwhere each minibatch processes 100 triplets.\nSemi-hard triplets are used on the training set, and\nrandom triplets are used on the testing set.\nOur `nn4.v1` model is from epoch 177.\n\nThe LFW section above shows that this model obtains a mean\naccuracy of 0.8483 ± 0.0172 with an AUC of 0.923.\n\n![](images/nn4.v1.loss.png)\n\n# How long does processing a face take?\nThe processing time depends on the size of your image for\nface detection and alignment.\nThese only run on the CPU and take from 100-200ms to over\na second.\nThe neural network uses a fixed-size input and has\na more consistent runtime.\nAveraging over 500 forward passes of random input, the latency is\n77.47 ms ± 50.69 ms on our 3.70 GHz CPU and\n21.13 ms ± 6.15 ms on our Tesla K40 GPU,\nobtained with\n[util/profile-network.lua](util/profile-network.lua)\n\n# Usage\n## Existing Models\nSee [the image comparison demo](demos/compare.py) for a complete example\nwritten in Python using a naive Torch subprocess to process the faces.\n\n```Python\nimport openface\nfrom openface.alignment import NaiveDlib # Depends on dlib.\n\n# `args` are parsed command-line arguments.\n\nalign = NaiveDlib(args.dlibFaceMean, args.dlibFacePredictor)\nnet = openface.TorchWrap(args.networkModel, imgDim=args.imgDim, cuda=args.cuda)\n\n# `img` is a numpy matrix containing the RGB pixels of the image.\nbb = align.getLargestFaceBoundingBox(img)\nalignedFace = align.alignImg(\"affine\", args.imgDim, img, bb)\nrep1 = net.forwardImage(alignedFace)\n\n# `rep2` obtained similarly.\nd = rep1 - rep2\ndistance = np.dot(d, d)\n```\n\n# Training new models\nThis repository also contains our training infrastructure to promote an\nopen ecosystem and enable quicker bootstrapping for new research and development.\nWarning: Training is computationally expensive and takes a few\nweeks on our Tesla K40 GPU.\n\nA rough overview of training is:\n\n1. Create a directory for your raw images so that images from different\n people are in different subdirectories. The names of the labels or\n images do not matter, and each person can have a different amount of images.\n The images should be formatted as `jpg` or `png` and have\n a lowercase extension.\n\n ```\n $ tree data/mydataset/raw\n person-1\n ├── image-1.jpg\n ├── image-2.png\n ...\n └── image-p.png\n\n ...\n\n person-m\n ├── image-1.png\n ├── image-2.jpg\n ...\n └── image-q.png\n ```\n\n\n2. Preprocess the raw images, change `8` to however many\n separate processes you want to run:\n `for N in {1..8}; do ./util/align-dlib.py align affine --size 96 &; done`.\n Prune out directories with less than N (I use 10) images\n per class with `./util/prune-dataset.py --numImagesThreshold ` and\n then split the dataset into `train` and `val` subdirectories\n with `./util/create-train-val-split.py

`.\n3. Run [training/main.lua](training/main.lua) to start training the model.\n Edit the dataset options in [training/opts.lua](training/opts.lua) or\n pass them as command-line parameters.\n This will output the loss and in-progress models to `training/work`.\n4. Visualize the loss with [training/plot-loss.py](training/plot-loss.py).\n\n# Setup\nThe following instructions are for Linux and OSX only.\nPlease contribute modifications and build instructions if you\nare interested in running this on other operating systems.\n\nWe strongly recommend using the [Docker](https://www.docker.com/)\ncontainer unless you are experienced with building\nLinux software from source.\n\nAlso note that in OSX, you may have to change the hashbangs\nfrom `python2` to `python`.\n\n## Check out git submodules\nClone with `--recursive` or run `git submodule init && git submodule update`\nafter checking out.\n\n## Download the models\nRun [models/get-models.sh](models/get-models.sh)\nto download pre-trained OpenFace\nmodels on the combined CASIA-WebFace and FaceScrub database.\nThis also downloads dlib's pre-trained model for face landmark detection.\nThis will incur about 500MB of network traffic for the compressed\nmodels that will decompress to about 1GB on disk.\n\nBe sure the md5 checksums match the following.\nUse `md5sum` in Linux and `md5` in OSX.\n\n```\nopenface(master)$ md5sum models/{dlib/*.dat,openface/*.{pkl,t7}}\n73fde5e05226548677a050913eed4e04 models/dlib/shape_predictor_68_face_landmarks.dat\nc0675d57dc976df601b085f4af67ecb9 models/openface/celeb-classifier.nn4.v1.pkl\na59a5ec1938370cd401b257619848960 models/openface/nn4.v1.t7\n```\n\n## With Docker\nThis repo can be deployed as a container with [Docker](https://www.docker.com/)\nfor CPU mode.\nBe sure you have checked out the submodules and downloaded\nthe models as described above.\nDepending on your Docker configuration, you may need to\nrun the docker commands as root.\n\nTo use, place your images in `openface` on your host and\naccess them from the shared Docker directory.\n\n```\ndocker build -t openface ./docker\ndocker run -t -i -v $PWD:/openface openface /bin/bash\ncd /openface\n./demos/compare.py images/examples/{lennon*,clapton*}\n```\n\n### Docker in OSX\nIn OSX, follow the\n[Docker Mac OSX Installation Guide](https://docs.docker.com/installation/mac/)\nand start a docker machine and connect your shell to it\nbefore trying to build the container.\nIn the simplest case, this can be done with:\n\n```\ndocker-machine create --driver virtualbox default\neval $(docker-machine env default)\n```\n\n## By hand\nBe sure you have checked out the submodules and downloaded the models as\ndescribed above.\nSee the [Dockerfile](docker/Dockerfile) as a reference.\n\nThis project uses `python2` because of the `opencv`\nand `dlib` dependencies.\nInstall the packages the Dockerfile uses with your package manager.\nWith `pip2`, install `numpy`, `pandas`, `scipy`, `scikit-learn`, and `scikit-image`.\n\nNext, manually install the following.\n\n### OpenCV\nDownload [OpenCV 2.4.11](https://github.com/Itseez/opencv/archive/2.4.11.zip)\nand follow their\n[build instructions](http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html).\n\n### dlib\ndlib can alternatively by installed from [pypi](https://pypi.python.org/pypi/dlib),\nbut might be slower than building manually because they are not\ncompiled with AVX support.\n\ndlib requires boost libraries to be installed.\n\nTo build manually, start by\ndownloading\n[dlib v18.16](https://github.com/davisking/dlib/releases/download/v18.16/dlib-18.16.tar.bz2),\nthen:\n\n```\nmkdir -p ~/src\ncd ~/src\ntar xf dlib-18.16.tar.bz2\ncd dlib-18.16/python_examples\nmkdir build\ncd build\ncmake ../../tools/python\ncmake --build . --config Release\ncp dlib.so ..\n```\n\nAt this point, you should be able to start your `python2`\ninterpreter and successfully run `import cv2; import dlib`.\n\nIn OSX, you may get a `Fatal Python error: PyThreadState_Get: no current thread`.\nYou may be able to resolve by rebuilding `python` and `boost-python`\nas reported in [#21](https://github.com/cmusatyalab/openface/issues/21),\nbut please file a new issue with us or [dlib](https://github.com/davisking/dlib)\nif you are unable to resolve this.\n\n### Torch\nInstall [Torch](http://torch.ch) from the instructions on their website\nand install the [dpnn](https://github.com/nicholas-leonard/dpnn)\nand [nn](https://github.com/torch/nn) libraries with\n`luarocks install dpnn` and `luarocks install nn`.\n\nIf you want CUDA support, also install\n[cudnn.torch](https://github.com/soumith/cudnn.torch).\n\nAt this point, the command-line program `th` should\nbe available in your shell.\n\n# Citations\n\n[![DOI](https://zenodo.org/badge/doi/10.5281/zenodo.32041.svg)](http://dx.doi.org/10.5281/zenodo.32041)\n\nPlease cite this repository if you use this in academic works.\n\n\n```\n@misc{amos2015openface,\n author = {Amos, Brandon and Harkes, Jan and Pillai, Padmanabhan and Elgazzar, Khalid and Satyanarayanan, Mahadev},\n title = {OpenFace 0.1.1: Face recognition with Google's FaceNet deep neural network},\n month = oct,\n year = 2015,\n doi = {10.5281/zenodo.32148},\n url = {http://dx.doi.org/10.5281/zenodo.32148}\n}\n```\n\n# Acknowledgements\n+ The fantastic Torch ecosystem and community.\n+ [Alfredo Canziani's](https://github.com/Atcold)\n implementation of FaceNet's loss function in\n [torch-TripletEmbedding](https://github.com/Atcold/torch-TripletEmbedding)\n+ [Nicholas Léonard](https://github.com/nicholas-leonard)\n for quickly merging my pull requests to\n [nicholas-leonard/dpnn](https://github.com/nicholas-leonard/dpnn)\n modifying the inception layer.\n+ [Francisco Massa](https://github.com/fmassa)\n and\n [Andrej Karpathy](http://cs.stanford.edu/people/karpathy/)\n for\n quickly releasing [nn.Normalize](https://github.com/torch/nn/pull/341)\n after I expressed interest in using it.\n+ [Soumith Chintala](https://github.com/soumith) for\n help with the [fbcunn](https://github.com/facebook/fbcunn)\n example code.\n+ NVIDIA's academic\n [hardware grant program](https://developer.nvidia.com/academic_hw_seeding)\n for providing the Tesla K40 used to train the model.\n+ [Davis King's](https://github.com/davisking) [dlib](https://github.com/davisking/dlib)\n library for face detection and alignment.\n+ Zhuo Chen, Kiryong Ha, Wenlu Hu,\n [Rahul Sukthankar](http://www.cs.cmu.edu/~rahuls/), and\n Junjue Wang for insightful discussions.\n\n# Licensing\nThe source code and trained models `nn4.v1.t7` and\n`celeb-classifier.nn4.v1.t7` are copyright\nCarnegie Mellon University and licensed under the\n[Apache 2.0 License](./LICENSE).\nPortions from the following third party sources have\nbeen modified and are included in this repository.\nThese portions are noted in the source files and are\ncopyright their respective authors with\nthe licenses listed.\n\nProject | Modified | License\n---|---|---|\n[Atcold/torch-TripletEmbedding](https://github.com/Atcold/torch-TripletEmbedding) | No | MIT\n[facebook/fbnn](https://github.com","has_starred","hub_user","immutable_tags_settings",{"_76":11,"_77":78},"enabled","rules",[79],".*","is_automated","permissions",{"_83":21,"_84":11,"_85":11},"read","write","admin","user","storage_size",39614656013,"orgData","recentTags",[92],{"_93":44,"_94":95,"_96":97,"_98":99,"_100":101,"_102":103,"_50":119,"_120":95,"_121":33,"_122":56,"_57":123,"_124":125,"_126":110,"_127":119,"_128":66,"_129":21},"content_type","creator",181230,"digest","sha256:da25a74d603ac023452334b7adaf7f830efd19bce230840051c40bbae5fb0de8","full_size",1341129639,"id",1222298,"images",[104],{"_105":106,"_96":97,"_107":108,"_109":110,"_111":112,"_113":108,"_114":108,"_115":-5,"_116":117,"_63":66,"_118":-5},"architecture","amd64","features","","last_pulled","2026-10-05T03:20:42.498250931Z","last_pushed","2016-01-04T21:01:28Z","os","os_features","os_version","size",1341129671,"variant","2016-01-04T21:01:27.878431Z","last_updater","last_updater_username","media_type","latest","repository",376614,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxRunInCloud","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":144,"_151":152,"_138":161,"_162":11},{"_40":145,"_42":146,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":147,"_57":35,"_32":33,"_58":59,"_60":-5,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":69,"_70":71,"_72":11,"_73":33,"_74":148,"_80":21,"_81":150,"_86":33,"_87":88},[],[44],[56],{"_76":11,"_77":149},[79],{"_83":21,"_84":11,"_85":11},"tags",{"_153":64,"_154":-5,"_155":-5,"_156":157},"count","next","previous","results",[158],{"_93":44,"_94":95,"_96":97,"_98":99,"_100":101,"_102":159,"_50":119,"_120":95,"_121":33,"_122":56,"_57":123,"_124":125,"_126":110,"_127":119,"_128":66,"_129":21},[160],{"_105":106,"_96":97,"_107":108,"_109":110,"_111":112,"_113":108,"_114":108,"_115":-5,"_116":117,"_63":66,"_118":-5},[],"canDeleteTags","actionData","errors"]

harshjv/openface

Tags for harshjv/openface