hidgweb/docker-webpagetest-api repository overview

⁠WebPageTest API Wrapper for NodeJS

WebPageTest API Wrapper⁠ is a NPM⁠ package that wraps WebPageTest⁠ API for NodeJS⁠ as a module and a command-line tool.

⁠Getting started

$ npm install webpagetest -g

⁠Basics

⁠Command line

$ webpagetest test https://twitter.com/marcelduran

⁠Docker

⁠Build

$ docker build -t webpagetest-api .

⁠Run

$ docker run -it --rm webpagetest-api -k YOURAPIKEY test https://twitter.com/marcelduran

⁠Module

const WebPageTest = require('webpagetest');
const wpt = new WebPageTest('www.webpagetest.org');

wpt.runTest('https://twitter.com/marcelduran', (err, data) => {
  console.log(err || data);
});

⁠API Console Demo

marcelduran.com/webpagetest-api⁠

⁠Command Line

⁠Help

$ webpagetest --help

⁠Commands

• status [options] <id>: check test status
• results [options] <id>: get test results
• locations [options]: list locations and the number of pending tests
• testers [options]: list testers status and details
• test [options] <url_or_script>: run test, <url_or_script> can also be a path to a script file
• restart <id>: restart test
• cancel <id>: cancel running/pending test
• har <id>: get the HTTP Archive (HAR) from test
• pagespeed [options] <id>: get the Google Page Speed results (if available) from test
• utilization [options] <id>: get the CPU, bandwidth and memory utilization data from test
• request [options] <\id>: get the request data from test
• timeline [options] <id>: get the Chrome Developer Tools Timeline data (if available) from test
• netlog [options] <id>: get the Chrome Developer Tools Net log data (if available) from test
• chrometrace [options] <id>: get the Chrome Trace data (if available) from test
• console [options] <id>: get the browser console log data (if available) from test
• testinfo <id>: get test request info/details
• history [days]: get history of previously run tests
• googlecsi [options] <id>: get Google CSI data (Client Side Instrumentation)
• response [options] <id>: get response body for text resources
• waterfall [options] <id>: get the waterfall PNG image
• screenshot [options] <id>: get the fully loaded page screenshot in JPG format (PNG if in full resolution)
• video [options] <tests>: create a video from <tests> (comma separated test ids)
• player <id>: get a html5 player for a video <id>
• listen [options] [port]: start webpagetest-api server on port [7791]
• batch <file>: run commands in batch, i.e. one command per line from <file> in parallel

⁠Options

⁠Common (works for all commands)

• -s, --server <server>: the WPT server URL [https://www.webpagetest.org⁠]
• -d, --dryrun: just return the RESTful API URL
• -o, --out <file>: place the output into <file>. Defaults to stdout

The default WPT server can also be specified via environment variable WEBPAGETEST_SERVER

⁠Test (works for test command only)

• -l, --location <location>: location to test from
• -y, --connectivity <profile>: connectivity profile -- requires location to be specified -- (Cable|DSL|3GSlow|3G|3GFast|4G|LTE|Edge|2G|Dial|FIOS|Native|custom) [Cable]
• -r, --runs <number>: number of test runs [1]
• -f, --first: skip the Repeat View test
• -v, --video: capture video
• -p, --private: keep the test hidden from the test log
• -L, --label <label>: label for the test
• -i, --onload: stop test at document complete. typically, tests run until all activity stops
• -S, --noscript: disable javascript (IE, Chrome, Firefox)
• -C, --clearcerts: clear SSL certificate caches
• -R, --ignoressl: ignore SSL certificate errors, e.g. name mismatch, self-signed certificates, etc
• -T, --standards: forces all pages to load in standards mode (IE only)
• -u, --tcpdump: capture network packet trace (tcpdump)
• -O, --bodies: save response bodies for text resources
• -K, --keepua: do not add PTST to the original browser User Agent string
• -m, --dom <element>: DOM element to record for sub-measurement
• -N, --duration <seconds>: minimum test duration in seconds
• -E, --tester <name>: run the test on a specific PC (name must match exactly or the test will not run)
• -W, --mobile: (experimental) emulate mobile browser: Chrome mobile user agent, 640x960 screen, 2x scaling and fixed viewport (Chrome only)
• --device <string>: device name from mobile_devices.ini to use for mobile emulation (only when mobile=1 is specified to enable emulation and only for Chrome)
• -M, --timeline: capture Developer Tools Timeline (Chrome only)
• -J, --callstack: set between 1-5 to include the JS call stack. must be used in conjunction with timeline (increases overhead) (Chrome only)
• -q, --chrometrace: capture chrome trace (about://tracing) (Chrome only)
• --tracecategories <categories>>: trace categories (when chrometrace enabled) (Chrome only)
• -G, --netlog: capture Network Log (Chrome only)
• -Q, --datareduction: enable data reduction on Chrome 34+ Android (Chrome only)
• -x, --useragent <string>: custom user agent string (Chrome only)
• -X, --cmdline <switches>: use a list of custom command line switches (Chrome only)
• -g, --login <username>: username for authenticating tests (http authentication)
• -w, --password <password>: password for authenticating tests (http authentication)
• -t, --sensitive: discard script and http headers in the result
• -H, --noheaders: disable saving of the http headers (as well as browser status messages and CPU utilization)
• -b, --block <urls>: space-delimited list of urls to block (substring match)
• -Z, --spof <domains>: space-delimited list of domains to simulate failure by re-routing to blackhole.webpagetest.org to silently drop all requests
• -c, --custom <script>: execute arbitrary javascript at the end of a test to collect custom metrics
• -a, --authtype <type>: type of authentication: 0 = Basic, 1 = SNS [0]
• -n, --notify <e-mail>: e-mail address to notify with the test results
• -B, --pingback <url>: URL to ping when the test is complete (the test ID will be passed as an "id" parameter)
• -D, --bwdown <bandwidth>: download bandwidth in Kbps (used when specifying a custom connectivity profile)
• -U, --bwup <bandwidth>: upload bandwidth in Kbps (used when specifying a custom connectivity profile)
• -Y, --latency <time>: first-hop Round Trip Time in ms (used when specifying a custom connectivity profile)
• -P, --plr <percentage>: packet loss rate - percent of packets to drop (used when specifying a custom connectivity profile)
• -z, --noopt: disable optimization checks (for faster testing)
• -I, --noimages: disable screen shot capturing
• -F, --full: save a full-resolution version of the fully loaded screen shot as a PNG
• -j, --jpeg <level>: jpeg compression level (30-100) for the screen shots and video capture
• -A, --medianvideo: store the video from the median run when capturing video is enabled
• --htmlbody: save the content of only the base HTML response
• --tsview <id>: test name to use when submitting results to tsviewdb (for private instances that have integrated with tsviewdb)
• --tsviewconfigs <string>: configs to use when submitting results to tsviewdb (for private instances that have integrated with tsviewdb)
• --affinity <string>: string to hash test to a specific test agent. tester will be picked by index among available testers
• --priority <number>: change test priority (0-9) [enforced by API key, otherwise 5]
• --continuous: capture video continuously (unstable/experimental, may cause tests to fail)
• --spdy3: force SPDY version 3 (Chrome only)
• --swrender: force software rendering, disable GPU acceleration (Chrome only)
• --poll <interval>: poll for results after test is scheduled at every seconds [5]
• --wait <hostname:port>: wait for test results informed by agent once complete listening on : [hostname:first port available above 8000]
• --timeout <seconds>: timeout for polling and waiting results [no timeout]
• --lighthouse: perform lighthouse test (Chrome only, Linux agent only)

⁠API Key (works for test, restart and cancel commands)

• -k, --key <api_key>:API key (if assigned). Contact the WebPageTest server administrator for a key if required or request an API key for limited testing at webpagetest.org/getkey.php⁠

⁠Request (works for status, results, locations, testers and test commands)

• -e, --request <id>: echo request ID, useful to track asynchronous requests

⁠Results (works for results and test commands)

• -b, --breakdown: include the breakdown of requests and bytes by mime type
• -D, --domains: include the breakdown of requests and bytes by domain
• -p, --pagespeed: include the PageSpeed score in the response (may be slower)
• -R, --requests: include the request data in the response (slower and results in much larger responses)
• -m, --median <metric>: set the metric used to calculate median for multiple runs tests [loadTime]
• --medianrun <metric>: set the run used for median for multiple runs tests [median]
• -S, --specs <json_or_file>: set the specs for performance test suite
• -r, --reporter <name>: set performance test suite reporter output: [dot]|spec|tap|xunit|list|progress|min|nyan|landing|json|doc|markdown|teamcity

⁠Run (works for pagespeed, utilization, request, timeline, netlog, chrometrace, console, googlecsi, response, waterfall and screenshot commands)

• -r, --run <number>: which run number on a multiple runs test [1]
• -c, --cached: get the Repeat View (cached view) instead of default First View (primed cache)

⁠Image (works for waterfall and screenshot commands)

• -t, --thumbnail: get the thumbnail of actual image
• -u, --uri: return the base64 string representation (inline) of actual image

⁠Screenshot (works for screenshot command only)

• -f, --full: get full resolution screenshot in PNG format if available
• -n, --render: get the page screenshot at the Start Render point (i.e.: when something was first displayed on screen)
• -p, --complete: get the page screenshot at the Document Complete point (i.e.: when window.onload was fired)

⁠Waterfall (works for waterfall command only)

• -T, --type <chart>: set the chart type: waterfall or connection [waterfall]
• -M, --mime: set chart coloring by MIME type [false]
• -w, --width <px>: chart image width in px (300-2000) [930]
• -m, --max <seconds>: set maximum time in seconds [automatic]
• -R, --requests <items>: filter requests (e.g.:1,2,3,4-9,8) [all]
• -C, --nocpu: hide CPU utilization [false]
• -b, --nobandwidth: hide bandwidth utilization [false]
• -e, --noellipsis: hide ellipsis (...) for missing items [false]
• -l, --nolabels: hide labels for requests (URL) [false]

⁠Video (works for video command only)

• -e, --end <end_point>: frame comparison end point: [visual]=visually complete | all=last change | doc=document complete | full=fully loaded

⁠Response (works for response command only)

• -R, --request <number>: the request number [1]

⁠Listen (works for listen command only)

• -k, --key <file>: private key file to use for SSL
• -c, --cert <file>: public x509 certificate file to use for SSL

⁠Examples

⁠1. Get available locations

$ webpagetest locations

{
  "response": {
    "statusCode": 200, "statusText": "Ok",
    "data": {
      "location": [
        ...
        {
          "id": "SanJose_IE9",
          "Label": "San Jose, CA USA (IE 9,Chrome,Firefox)",
          "location": "SanJose_IE9",
          "Browser": "IE 9",
          "PendingTests": {
            "p1": 0, "p2": 0, "p3": 0, "p4": 0, "p5": 2, "p6": 2, "p7": 0,
            "p8": 0, "p9": 0, "Total": 7, "HighPriority": 2, "LowPriority": 4,
            "Testing": 1, "Idle": 0
          }
        },
        ...
      ]
    }
  }
}

⁠2. Run test on https://twitter.com/marcelduran⁠ from San Jose on IE9

$ webpagetest test https://twitter.com/marcelduran --key 1F2A3K4E5 --location SanJose_IE9

{
  "statusCode": 200,
  "statusText": "Ok",
  "data": {
    "testId": "121025_PT_N8K",
    "ownerKey": "868cb2813a0f376a977dd1a24ab041b4f12361b3",
    "jsonUrl": "https://www.webpagetest.org/results.php?test=121025_PT_N8K&f=json",
    "xmlUrl": "https://www.webpagetest.org/xmlResult.php?test=121025_PT_N8K",
    "userUrl": "https://www.webpagetest.org/results.php?test=121025_PT_N8K",
    "summaryCSV": "https://www.webpagetest.org/csv.php?test=121025_PT_N8K",
    "detailCSV": "https://www.webpagetest.org/csv.php?test=121025_PT_N8K&requests=1"
  }
}

⁠3. Check current test status

$ webpagetest status 121025_PT_N8K

{
  "statusCode": 101,
  "statusText": "Test Pending",
  "data": {
    "statusCode": 101,
    "statusText": "Test Pending",
    "testId": "121025_PT_N8K",
    "runs": 1,
    "fvonly": 0,
    "location": "SanJose_IE9"
  }
}

⁠4. Get test results

$ webpagetest results 121025_PT_N8K

{
  "response": {
    "statusCode": 200, "statusText": "Ok",
    "data": {
      "testId": "121025_PT_N8K",
      "summary": "https://www.webpagetest.org/result/121025_PT_N8K/",
      "testUrl": "https://twitter.com/marcelduran",
      "location": "SanJose_IE9",
      "connectivity": "DSL",
      "bwDown": 1500, "bwUp": 384, "latency": 50, "plr": 0,
      "completed": "Thu, 25 Oct 2012 23:42:11 +0000",
      "runs": 1, "successfulFVRuns": 1,
      "average": {
        "firstView": {
          "loadTime": 3942, "TTFB": 1518,
          "bytesIn": 963405, "bytesInDoc": 431612,
          "requests": 32, "requestsDoc": 19,
          "render": 2509, "fullyLoaded": 7765,
          "docTime": 3942, "domTime": 0,
          "titleTime": 1641, "avgRun": 1
        }
      },
      ...
    }
  }
}

⁠5. Get test waterfall thumbnail from repeat view as data URI

$ webpagetest waterfall 121025_PT_N8K --thumbnail --cached --uri

{
  "type": "image/png",
  "data": "iVBORw0KGgoAAAANSUhEUgA...RK5CYII="
}

⁠Run test on https://twitter.com/marcelduran⁠ and poll results every 5 seconds timing out in 60 seconds

$ webpagetest test https://twitter.com/marcelduran --poll 5 --timeout 60

⁠Or run test on https://twitter.com/marcelduran⁠ and wait for results listening on localhost* port 8000**

$ webpagetest test https://twitter.com/marcelduran --wait 8000

{
  "response": {
    "statusCode": 200, "statusText": "Ok",
    "data": {
      "testId": "121025_PT_N8K",
      "testUrl": "https://twitter.com/marcelduran",
      ...
      "median": {
        "firstView": {
          "loadTime": 3942, "TTFB": 1518,
          "render": 2509, "fullyLoaded": 7765,
          ...
        }
      },
      ...
    }
  }
}

* hostname and port are optional, defaults to <system hostname>:<8000> ** localhost and port must be reacheable from WebPageTest server

⁠Module

Methods and options (including the one letter shorthands) are the same when using as a Node module, however a more verbose version of both commands (methods) and options (parameters) are available and encouraged to use for code clarity.

⁠Methods

• getTestStatus(id, options, callback)
• getTestResults(id, options, callback)
• getLocations(options, callback)
• getTesters(options, callback)
• runTest(url_or_script, options, callback)
• restartTest(id, options, callback)
• cancelTest(id, options, callback)
• getHARData(id, options, callback)
• getPageSpeedData(id, options, callback)
• getUtilizationData(id, options, callback)
• getRequestData(id, options, callback)
• getTimelineData(id, options, callback)
• getNetLogData(id, options, callback)
• getChromeTraceData(id, options, callback)
• getConsoleLogData(id, options, callback)
• getTestInfo(id, options, callback)
• getHistory(days, options, callback)
• getGoogleCsiData(id, options, callback)
• getResponseBody(id, options, callback)
• getWaterfallImage(id, options, callback)
• getScreenshotImage(id, options, callback)
• createVideo(tests, options, callback)
• getEmbedVideoPlayer(id, options, callback)
• listen(port, options, callback)
• scriptToString(script)

⁠Parameters

• id: test ID string required
• options: parameters object optional, see below
• callback: the callback (error, data) _optional=> _
• url_or_script: decoded url or script string required
• port: port number optional [default: 7791]
• script: script array in the format:

[
  {command1: 'value1'},
  {command2: 123},
  {command3: ['value1', 'value2', ... , 'valueN']},
  ...
  'commandN'}
]

⁠Notes

• getWaterfallImage and getScreenshotImage callback function has a third parameter info which is an object with {type: 'image/jpeg or png', encoding: 'utf8 or binary'}
• scriptToString script array values 1-N are optional. e.g:

const script = wpt.scriptToString([
  {logData: 0},
  {navigate: 'http://foo.com/login'},
  {logData: 1},
  {setValue: ['name=username', 'johndoe']},
  {setValue: ['name=password', '12345']},
  {submitForm: 'action=http://foo.com/main'},
  'waitForComplete'
]);

wpt.runTest(script, (err, data) => {
  console.log(err || data);
});

⁠Options

⁠Common (works for all methods with options parameter)

• dryRun: Boolean, if true, method does not make an actual request to the API Server but rather returns an object with url which contains the actual URL to make the GET request to WebPageTest API Server
• server: String, if specified, overrides the WebPageTest server informed in the constructor only for that method call

⁠Test (works for runTest method only)

• location: String, location to test from
• connectivity: String, connectivity profile -- requires location to be specified -- (Cable|DSL|3GSlow|3G|3GFast|4G|LTE|Edge|2G|Dial|FIOS|Native|custom) [Cable]
• runs: Number, number of test runs [1]
• firstViewOnly: Boolean, skip the Repeat View test
• video: Boolean, capture video
• private: Boolean, keep the test hidden from the test log
• label: String, label for the test
• stopAtDocumentComplete: Boolean, stop test at document complete. typically, tests run until all activity stops
• disableJavaScript: Boolean, disable JavaScript (IE, Chrome, Firefox)
• clearCerts: Boolean, clear SSL certificate caches
• ignoreSSL: Boolean, ignore SSL certificate errors, e.g. name mismatch, self-signed certificates, etc
• disableCompatibilityView: Boolean, forces all pages to load in standards mode (IE only)
• tcpDump: Boolean, capture network packet trace (tcpdump)
• saveResponseBodies: Boolean, save response bodies for text resources
• keepOriginalUserAgent: Boolean, do not add PTST to the original browser User Agent string
• domElement: String, DOM element to record for sub-measurement
• minimumDuration: Number, minimum test duration in seconds
• tester: String, run the test on a specific PC (name must match exactly or the test will not run)
• emulateMobile: Boolean, (experimental) emulate mobile browser: Chrome mobile user agent, 640x960 screen, 2x scaling and fixed viewport (Chrome only)
• timeline: Boolean, capture Developer Tools Timeline (Chrome only)
• timelineCallStack: Boolean, set between 1-5 to include the JS call stack. must be used in conjunction with timeline (increases overhead) (Chrome only)
• chromeTrace: Boolean, capture chrome trace (about://tracing) (Chrome only)
• netLog: Boolean, capture Network Log (Chrome only)
• dataReduction: Boolean, enable data reduction on Chrome 34+ Android (Chrome only)
• userAgent: String, custom user agent string (Chrome only)
• commandLine: String, use a list of custom command line switches (Chrome only)
• login: String, username for authenticating tests (http authentication)
• password: String, password for authenticating tests (http authentication)
• sensitive: Boolean, discard script and http headers in the result
• disableHTTPHeaders: Boolean, disable saving of the http headers (as well as browser status messages and CPU utilization)
• block: String, space-delimited list of urls to block (substring match)
• spof: String, space-delimited list of domains to simulate failure by re-routing to blackhole.webpagetest.org to silently drop all requests
• customMetrics: String, execute arbitrary JavaScript at the end of a test to collect custom metrics
• authenticationType: Number, type of authentication: 0 = Basic, 1 = SNS [0]
• notifyEmail: String, e-mail address to notify with the test results
• pingback: String, URL to ping when the test is complete (the test ID will be passed as an "id" parameter)
• bandwidthDown: String, download bandwidth in Kbps (used when specifying a custom connectivity profile)
• bandwidthUp: String, upload bandwidth in Kbps (used when specifying a custom connectivity profile)
• latency: String, first-hop Round Trip Time in ms (used when specifying a custom connectivity profile)
• packetLossRate: Number, packet loss rate - percent of packets to drop (used when specifying a custom connectivity profile)
• disableOptimization: Boolean, disable optimization checks (for faster testing)
• disableScreenshot: Boolean, disable screen shot capturing
• fullResolutionScreenshot: Boolean, save a full-resolution version of the fully loaded screen shot as a PNG
• jpegQuality: Number, jpeg compression level (30-100) for the screen shots and video capture
• medianVideo: Boolean, store the video from the median run when capturing video is enabled
• htmlBody: Boolean, save the content of only the base HTML response
• tsView: String, test name to use when submitting results to tsviewdb (for private instances that have integrated with tsviewdb)
• tsViewConfigs: String, configs to use when submitting results to tsviewdb (for private instances that have integrated with tsviewdb)
• affinity: String, string to hash test to a specific test agent. tester will be picked by index among available testers
• priority: Number, change test priority (0-9) [enforced by API key, otherwise 5]
• blockAds: Boolean, block ads defined by http://adblockplus.org⁠
• continuousVideoCapture: Boolean, capture video continuously (unstable/experimental, may cause tests to fail)
• forceSpdy3: Boolean, force SPDY version 3 (Chrome only)
• forceSoftwareRendering: Boolean, force software rendering, disable GPU acceleration (Chrome only)
• pollResults: Number, poll for results after test is scheduled at every seconds [5]
• waitResults: String, wait for test results informed by agent once complete listening on : [hostname:first port available above 8000]
• timeout: String, timeout for polling and waiting results [no timeout]
• lighthouse: Boolean, perform lighthouse test (Chrome only, Linux agent only)

⁠API Key (works for runTest, restartTest and cancelTest methods)

• key: String, API key (if assigned). Contact the WebPageTest server administrator for a key if required

⁠Request (works for getTestStatus getResults getLocations getTesters and runTest metho