Here at EQCO, we've found ourselves having huge build times for a single docker application container, and the reason was that we weren't using docker's cache to our advantage. Each time we went to build a new version of our application container we had to recompile numpy, scipy and some other python libraries that contain C extensions. Our build time skyrocketed to about 2 hours per container build.
Our solution was to provide a cacheable base image that allows us to build on top of it. We can precompile our binary dependencies and bolt our application on top of our python base image.