HUB
Sign inSign up
qti3-validator logo

feyst/qti3-validator

By feyst

•Updated about 19 hours ago

HTTP service that validates QTI 3 items, tests and packages against the 1EdTech schemas

Image
0

1.9K

feyst/qti3-validator repository overview

⁠QTI 3 Validator

CI Release Docker pulls License: MIT

An HTTP service that checks QTI 3.0 and 3.0.1 items, tests and complete content packages against the official 1EdTech schemas and rules. Send it a document or a ZIP package; it tells you whether it is valid and, if not, exactly what is wrong and where.

  • Complete checks: well-formed XML, the QTI 3 XML Schemas (XSD), the 7,400+ Schematron rules 1EdTech publishes alongside them, and additional checks⁠ on how the parts of an item fit together.
  • Precise errors: line, column and element path for every problem.
  • Small and fast: a ~21 MB image that uses about 30 MB of memory when idle, is ready in just over a second and validates a typical item in well under a millisecond.
  • Self-contained: the schemas are built in. The service never contacts the internet and never stores or logs the documents you send.

⁠Quick start

Start the service:

docker run --rm -p 8080:8080 ghcr.io/feyst/qti3-validator

Validate a package:

curl -X POST http://localhost:8080/api/validate \
  -H 'Content-Type: application/zip' \
  --data-binary @package.zip

Or a single item:

curl -X POST http://localhost:8080/api/validate \
  -H 'Content-Type: application/xml' \
  --data-binary @item.xml

The answer is a validation report⁠. summary.outcome is the verdict; see Outcomes⁠:

{
  "summary": {
    "outcome": "VALID",
    "fatals": 0,
    "errors": 0,
    "warnings": 0,
    "exceptions": 0,
    "notRun": 0,
    "totalRun": 3,
    "valid": 1
  },
  "errors": [],
  …
}

To see what findings look like, validate the example package with a problem of each kind, examples/with-errors.zip⁠.

⁠What is checked

Every document goes through four checks:

  1. Well-formedness: the document must be well-formed XML in UTF-8.
  2. XML Schema: the document must conform to the QTI XSDs of its version, including what QTI imports, such as MathML 3, SSML 1.1 and IEEE LOM.
  3. Schematron rules: the rules 1EdTech embeds in those schemas, for what an XSD cannot express, such as allowed attributes, max-choices against min-choices and ARIA roles.
  4. Additional checks⁠: whether the parts of an item, test or package fit together, for example that every interaction is bound to a declared response variable, that the files a package refers to are in it, and that values and expressions have the right types.

A package is checked file by file. It needs a QTI 3 imsmanifest.xml in its root; every XML file is validated, media files are not opened, and XML files that are not a QTI document are reported as not run, unless the manifest lists them as a QTI resource. A QTI fragment (XML that another document includes with XInclude, whether or not the manifest lists it as a resource of type imsqti_fragment_xmlv3p0) is checked as a fragment; see the report⁠.

Accepted documents are QTI items, tests, sections, stimuli, response processing templates, outcome declarations, package manifests, LOM and QTI metadata. Others, including QTI 2.x, are rejected with unsupported_document, unless a custom validator⁠ adds them.

⁠QTI versions

QTI 3.0 (3.0.0) and 3.0.1 are supported, each with its own official schemas and rules. Both use the same XML namespace, so the service picks the version like this:

SituationVersion used
?version=3.0.0 or ?version=3.0.1 in the requestThat version, for every document
A package, or a manifest on its ownThe manifest's <schemaversion>
Any other documentThe latest version, 3.0.1

The report says which version was used, in specification.version. To check 3.0 content against 3.0.1, pass ?version=3.0.1: the manifest's <schemaversion>3.0.0</schemaversion> then gives a version_overridden warning instead of an error.

⁠API

MethodPathPurpose
POST/api/validateValidate a package or a single XML document
GET/api/validatorsList the validatorId values
GET/healthLiveness and readiness check
GET/versionService and QTI versions
⁠POST /api/validate

Send a ZIP package or a single XML document as the request body. The service recognises which it is; a Content-Type is optional:

Content-TypeValidated as
application/zipA package
application/xml, text/xmlA single document
none, or any otherA package if the body is a ZIP, otherwise a single document

Query parameters, all optional:

ParameterMeaning
version3.0.0 or 3.0.1: the QTI version every document is validated against
nameThe name of the input in the report; default package.zip or document.xml
validatorIdQti30Inspector; see GET /api/validators

For example, to validate a 3.0 package against QTI 3.0.1:

curl -X POST 'http://localhost:8080/api/validate?version=3.0.1&name=toets.zip' \
  -H 'Content-Type: application/zip' \
  --data-binary @toets.zip
⁠Outcomes

Every request that could be validated gets HTTP 200 and a report, also when the content is invalid; a validation that had to stop before the end gets HTTP 422 with the report so far. Each finding is listed under its outcome (errors, warnings, fatals, …), and summary.outcome is the most severe one:

OutcomeWhenValid?
VALIDA document passed every checkyes
WARNINGSomething to look at that does not break the item, such as an outcome that is never usedyes
ERRORA document breaks the XML Schema or a rule, or a package misses its manifestno
FATALSomething could not be read at all: XML that is not well-formed, not UTF-8 or has a DOCTYPE, a document that is not QTI 3, or a body that is not a ZIPno
EXCEPTIONThe validation did not finish: it stopped because a document is too complex or took longer than REQUEST_TIMEOUT (code too_complex, HTTP 422), or the service itself failed (code internal, HTTP 500)no
NOT_RUNA check was skipped: an XML file that is not a QTI document, or the checks of a document that was not validated, such as after a FATALdoes not count

In a package, every file gets its own findings, so one package can contain all of them. examples/with-errors.zip⁠ shows each kind. The validation report⁠ describes every field and code.

⁠Status codes

A request that could not be validated at all gets an error status and a short JSON body:

{
  "code": "unsupported_version",
  "message": "version must be one of 3.0.0, 3.0.1"
}
HTTPCodeWhen
400unsupported_versionversion is not a supported version
400unknown_validatorvalidatorId is not listed by GET /api/validators
400invalid_requestThe body could not be read
408request_timeoutThe body did not arrive within REQUEST_TIMEOUT
413too_large, too_many_filesThe request, package, or the XML in a package exceeds a limit, the request is larger than all of MAX_SPOOL_SIZE, or a document has more nodes than can be checked in MAX_VALIDATION_MEMORY; the message gives the numbers
422too_complexValidation stopped before the end: a document's rules need more work than one document may take (a budget of XPath steps that grows with the size of the document, up to a fixed maximum), or validation took longer than REQUEST_TIMEOUT; the body is a report with outcome EXCEPTION and the findings so far
500internalThe service failed; the body is a report with outcome EXCEPTION
503busyNo room for the request now: no validation slot or validation memory came free within REQUEST_TIMEOUT, MAX_QUEUE requests are already waiting, or the uploads in progress fill MAX_SPOOL_SIZE and leave no room for the next bytes of this one; retry after the seconds in Retry-After

⁠Configuration

All settings are environment variables. Sizes are in bytes.

VariableDefaultDescription
ADDR:8080Address and port to listen on
MAX_REQUEST_SIZE10485760 (10 MiB)Largest single XML document
MAX_PACKAGE_SIZE104857600 (100 MiB)Largest request body for a package
MAX_FILE_SIZE10485760 (10 MiB)Largest uncompressed XML file inside a package
MAX_FILES1000Most entries a package may contain
MAX_UNCOMPRESSED_SIZE268435456 (256 MiB)Largest total of uncompressed XML in one package
MAX_ERRORS100Most errors per document from the XML Schema and the rules; the rules' warnings have a limit of the same size. See what is limited⁠
MAX_DEPTH256Deepest element nesting accepted
MAX_CONCURRENTnumber of CPUsValidations that run at the same time; further requests wait. The CPUs are those of the container's CPU limit, or GOMAXPROCS if you set it; the service itself runs with one thread more. One document under 256 KiB more runs beside them, on an express slot of its own, so large documents do not keep typical items waiting. A package never takes that slot, however small it is
MAX_QUEUE4 × MAX_CONCURRENTRequests that may wait for a validation slot or for validation memory, as many small ones (under 256 KiB, taking up to 4 MiB) as large ones; further requests get 503 at once
MAX_SPOOL_SIZE4 × MAX_PACKAGE_SIZERequest bodies held at once, counted by the bytes received: in /tmp, or in memory for a body under 256 KiB; an upload that does not fit gets 503 at once, one larger than all of it 413. Lowered automatically when /tmp is a tmpfs; see Temporary files⁠
MAX_VALIDATION_MEMORYthe memory limit less 72 MiBMemory the validations in progress may reserve together, in bytes. A validation reserves what its document takes, which its number of nodes decides rather than its size, and waits for room like for a slot, holding nothing while it waits; so the documents being validated fit in the container whatever MAX_CONCURRENT and MAX_QUEUE are. A share of it, 4 MiB per slot, is kept for typical items. Only a document that could never fit gets 413, with the numbers. Derived at startup from the memory limit of the container, or of a group above it; 1 GiB when there is none. See Memory and CPU⁠
REQUEST_TIMEOUT60sTime limit per request: receiving the body, waiting for a free slot and for validation memory, and validating. Running out while the body arrives gives 408; while waiting for a slot or memory, 503; a validation still running then stops with too_complex, 422 (the XML Schema check of the current document finishes first)
VALIDATORS_DIR/validatorsDirectory with extra validators; see Custom validators⁠

For example:

docker run --rm -p 8080:8080 \
  -e MAX_PACKAGE_SIZE=52428800 \
  -e MAX_CONCURRENT=4 \
  ghcr.io/feyst/qti3-validator

Running in production⁠ covers memory, uploads, public deployments, a read-only container, Docker Compose, Kubernetes and logging.

⁠Limitations

  • Scoring: whether response processing gives the scores you intend is not checked; the additional checks only check that it can run, and that its values and expressions have the right types.
  • Encoding: documents must be UTF-8 (ASCII is fine). A DOCTYPE is not accepted.
  • SSML: two SSML restrictions are not enforced: version and xml:lang on <ssml:speak>, and name on <ssml:mark>.

⁠Documentation

⁠License

MIT⁠. The patched copy of github.com/jacoelho/xsd in third_party/ and the fixtures from php-qti3 in testdata/ keep their own MIT licenses.

Tag summary

Content type

Image

Digest

sha256:fc29a2473…

Size

6.1 MB

Last updated

about 1 month ago

docker pull feyst/qti3-validator