ehwest/sre repository overview

Introduction This Software Readiness Engineering (SRE) analysis tool implements a statistical approach to forecasting the number and time for finding unknown bugs that will be found by future testing or production operations. The method uses time-series data describing software bugs already found during test activities. This method has been used with excellent results to make and improve executive decisions to release production software (or, alternatively, continue testing). The analysis tool runs as a web application entirely within a Docker container. The Docker container wrappers an Apache web server, a mySQl database, the OPENBugs platform for performing Monte-Carlo optimization, and all of the software required for a dedicated web-based user interface for the analysis. This makes the analysis accessible to anyone who can download and install Docker containers as desktop software. The Docker container and the analysis software is offered free under the Apache 2 license.

Software Readiness Engineering (SRE) Analysis With SRE analysis, bugs reported during the development and test process are statistically modeled as colored jellybeans being randomly selected (without replacement) from an urn containing an unknown number of colored and uncolored jellybeans. In this case, each colored jellybean represents a bug was that found; each uncolored jellybean represents a unit of testing effort where no bug is found.
A modeled selection process selects a jellybean from the urn without replacement. If a colored jellybean is selected, the time is noted and the selection process (modeling the software testing effort) continues to select another jellybean.
Now under a non-replacement policy, one would expect the time interval between successive colored jellybeans to increase steadily approaching infinity. Indeed, when all of the colored jellybeans are selected and removed, one would expect an infinite amount of time to find the next (non-existent) colored jellybean. As long as jellybeans remain in the urn to be selected, one never knows if the last colored jellybean has been removed, so as a practical matter, the selection process (modeled as the software testing process) is simply stopped, before all of the bugs have been found (and removed from the process). An analysis of the time-series data marking the discovery of individual software bugs (modeled as colored jellybeans), one may compute a number of useful (Bayesian) statistics. Using a large number of Monte-Carlo simulations, this SRE Analysis tool computes all of the interesting statistics and provides summary information and various plots, including a forecast of the timing and number of bugs remaining in the software. There are important assumptions built into this analysis. First, we assume all reported bugs are found independently and are not correlated among themselves. In addition, we assume the “unit of effort” applied to testing is constant. We also assume the definition of a “bug” whatever it is, is unchanged during the testing. Most importantly, we assume the bugs when plotted on a chart of “testing effort” on the x-axis and number of bugs found on the y-axis are expected to fit a well-known family of “S-shaped” curves.
The “goodness” of fit of the data to these S-shaped curves is expected to be “better” than a fit to a straight line. One of the statistics we compute is in fact the ratio for how much “better” it is. That ratio (DI, Distribution Index) is benchmarked against an absolute value of “3”. Below “3” means the software bugs better fit a straight line than an S-shaped curve, and the analysis cannot predict anything about the software. A DI value of “3.0” and above is “generally considered” to indicate a “good fit” and is useful for making forecasts for how many bugs remain in the software, and how much testing effort is needed to remove any number of additional bugs. The “input” to the SRE analysis tool is a MS Excel spreadsheet listing the time-series data in Column A. The output is provided as an update to a web page, with downloadable PDF files with graphs. As an aide to automating the whole process, the tool also supports all-electronic RESTful API calls for all functions, including data loading, PDF report delivery, and status on calculation progress. However, those APIs are beyond the scope of this paper.

Installation Procedures The software is contained in a “Docker” container on the Docker Hub for public repositories. The location of the Docker container is: https://hub.docker.com/r/ehwest/sre/⁠ Installing this, however, must be done by having a Docker host process retrieve the Docker image (ehwest/sre) directly from the repository. To further enable this, the user must register with Docker Hub at https://hub.docker.com⁠ and get a (free) Username and password. A Docker host must therefore be installed and running on the target PC or MAC system to pull the image into the target machine and run it. Installing and testing Docker Host on your platform is beyond the scope of this document. Note also that if your PC or MAC is ”locked down” and you do not have certain administrative rights, you will not be able to use that device as a Docker Host. When you do install and run the docker host, be sure to capture the IP address of the Docker Host. This varies, but we used “Docker Quickstart Terminal” which creates its own VM for Docker to run on, and makes the Docker Host appear at 192.168.99.100. This IP address is important and will be inserted into the browser when the Docker container is loaded. One may determine that a Docker host is running with the following command-line command:  docker –v  Docker version 1.12.5, build 7392c3b Now if you get back the version, then Docker is running. You need to log into the Docker Hub to retrieve the container as follows:  docker login  Username:  Password:  Login Succeeded  docker pull ehwest/sre  docker run –d –p 49160:22 –p 49161:80 –p 49162:3306 ehwest/sre docker run -d -p 49160:22 -p 49161:80 -p 49162:3306 ehwest/sre The parameters in the command line above provide you with ssh root/admin access to port 49160, web access at port 49161, and database access on port 49162. As a casual user, you will only need web access. However, you may wish to change the look of the web pages and may do so by logging in using SSH and modifying the files under the /var/www folder. Using the SRE Analysis Tool Pull up a web browser to this address (recalling the IP address of the Docker Host:
http://192.168.99.100:49161/sre_web_services/showprojects.htm⁠ If you see a web page that says “Welcome to the Software Readiness Estimation (SRE) Analysis Management Too” you have a working system. One “test” project is listed in a table of possibly many analysis projects. Click on the link labeled “Click Here to submit using a GUI that uses the API.” You should then see a page that has “Create A Project” table on the page. We provide two different file formats for inputting your data to be analyzed. However, the Unit of Effort Format is deprecated. Select the Time Series Format “time_series_samplefile.xlsx” from the box. This downloads to your MS Excel spreadsheet a sample file.
Using MS Excel, notice that column “A” contains time series data, with one row for each “bug” reported. There are other boxes you can populate (all optional) and will be used as an aid to keep track of the parameters for the analysis project.
Replace the sample time series data in the sample file with data describing your own software testing progress. Each row should contain exactly one time/date stamp, all unique, and in ascending order. Cell A1 only contains the words “Time Series Data” and no timestamp. After the last time-stamp, the contents of the cells should be empty. Save your modified sample file to another filename.
Within the “Create a New Project” page, select the “Time Series Format” radio checkbox, and click on the “Upload: Choose File” button. You will be presented with a navigation of your file system and you should navigate to the MS Excel spreadsheet you modified, containing your own project data. After navigating the file name, click on “Upload Chosen File” and you will then be presented with a table of Projects, with your own project listed at the top of the list. If your project is listed, you should check the Number of Input data points, and the name of the project to ensure the system has correctly parsed your MS Excel spreadsheet. Under the “Current State” /” Take Action” column, click on “Calculate” to begin the processing of your data for that project. It may take several minutes for the calculation to complete. You may refresh the screen from time-to-time to monitor progress. An automatic screen refresh will also take place at 60 second intervals.
When the calculation is done, you may scroll to the FAR RIGHT of the screen and find the “Charts and Graphs” column. Click on the PDF Charts link to download your analysis. Note the “DI” value on the charts. This figure is a log of the ratio of the fit to to the “S-shaped” family of curves to the fit of a straight line. Values of DI above 3.0 indicate a strong fit to the model. Two different families of curves are fit: “IM” and “NIM” [2] . These vary slightly in their shape at the lower portion of the curve.