Sclite is the benchmark tool that users can apply to evaluate Cognitive Speech Recognition Service. This is the docker image of Sclite which is based on SCTK(Speech Recognition Scoring Toolkit).
To run Sclite, you have to provide a .tsv file as input. The format of this input is
utteranceId reference recognition
utteranceId: the unique id for the utterance
reference: the text to be translated by Speech Recognition Service
recognition: the expected text output by Speech Recognition Service
An example of .tsv file content:
1089-134686-0007__1 what's the weather like what's the weather like
1089-134686-0007__1 hello my friend hello my friend
...
Put such .tsv input file under somewhere local (i.e. ../local/path/tsv_in/).
You also need to create a local output directory (i.e.. ../local/path/tsv_out/).
Mount both inpout and output directories by docker run command:
docker run -it -v ../local/path/tsv_in/:/tsv_in -v ../local/path/tsv_out/:/tsv_out antsu/sclite:latest
You are expected to see success information as below as an example if Sclite runs properly:
Pretzel TRN mode enabled...
No GLM file specified, using an empty GLM...
Removing output folder...
Output will be stored at /tsv_out/scoring
Pretzel trn mode is enabled...
Converting tsv to trn...
INFO:root:All done - output stored at '/tmp/ref.trn'
INFO:root:All done - output stored at '/tmp/hyp.trn'
Applying GLM rules...
DEBUG
INFO:root:Loaded 3 GLM rules.
INFO:root:Applying GLM rules to the ref file...
INFO:root:Applying GLM rules to the hyp file...
INFO:root:All done.
Running sclite...
/sctk/misc /sctk/misc
sclite: 2.10 TK Version 1.3
Begin alignment of Ref File: '/tmp/ref.trn.filt' and Hyp File: '/tsv_out/scoring/hyp.trn.filt'
Alignment# 48 for speaker 108
Writing scoring report to '/tsv_out/scoring/hyp.trn.filt.sys'
Writing raw scoring report to '/tsv_out/scoring/hyp.trn.filt.raw'
Writing overall detailed scoring report '/tsv_out/scoring/hyp.trn.filt.dtl'
Writing LUR scoring report to '/tsv_out/scoring/hyp.trn.filt.lur'
Writing string alignments to '/tsv_out/scoring/hyp.trn.filt.pra'
Writing string alignments to '/tsv_out/scoring/hyp.trn.filt.prf'
Writing SGML string alignments to '/tsv_out/scoring/hyp.trn.filt.sgml'
Successful Completion
/sctk/misc
scoring finished.
After running Sclite successfully, you are expected to get a report similar to below sample.
(This tool provides reports with different level of details. Below is scoring report.)
SYSTEM SUMMARY PERCENTAGES by SPEAKER
,----------------------------------------------------------------.
| /tsv_out/scoring/hyp.trn |
|----------------------------------------------------------------|
| SPKR | # Snt # Wrd | Corr Sub Del Ins Err S.Err |
|--------+-------------+-----------------------------------------|
| 108 | 48 806 | 94.7 4.8 0.5 1.0 6.3 50.0 |
|================================================================|
| Sum/Avg| 48 806 | 94.7 4.8 0.5 1.0 6.3 50.0 |
|================================================================|
| Mean | 48.0 806.0 | 94.7 4.8 0.5 1.0 6.3 50.0 |
| S.D. | 0.0 0.0 | 0.0 0.0 0.0 0.0 0.0 0.0 |
| Median | 48.0 806.0 | 94.7 4.8 0.5 1.0 6.3 50.0 |
`----------------------------------------------------------------'
Content type
Image
Digest
Size
52 MB
Last updated
about 7 years ago
docker pull antsu/sclite