EVAL_ST Tool 1.0.0

File: <base>/EVAL_ST/man/00README (28,698 bytes)

THE EVAL_ST TOOL

--------------------------------------------------------------------------------

The EVAL_ST tool [1] (PostScript and PDF versions of the reference [1] are in the
man subdirectory - cinc04.ps and sinc04.pdf.) is an open source tool to evaluate
and compare performance and robustness of ST episode detection algorithms. The
tool supports all standard and other relevant performance measures, aggregate
gross and average statistics, and bootstrap statistical procedure to predict
real-world clinical performance. The tool (written in C) is compilable an a wide
variety of platforms and contains an additional graphic user interface module
(LessTif/Motif environment) for use on the LINUX/UNIX operating systems.

--------------------------------------------------------------------------------

INTRODUCTION

Assessing the performance and robustness of ST segment analysers and algorithms
as well as predicting their behavior in the real-world clinical environment is
a difficult task. Availability of the European Society of Cardiology ST-T
Database (ESC DB) [3], gained development of transient ST segment episode
detectors and allowed comparison of their performance. Newly developed Long-Term
ST Database (LTST DB) [2] provides a wide variety of real-world 24-hour ambulatory
records with numerous examples of transient ischemic ST segment episodes and
transient non-ischemic heart-rate related ST segment episodes. It gained further
development and evaluation of transient ST episode detectors.

Due to relative complexity of the performance measures and of the evaluation
protocol, we developed an open-source tool EVAL_ST, Version 2.0, to objective
evaluate and compare the performance and robustness of transient ST episode
detection algorithms. Since the LTST DB contains transient ischemic and transient
non-ischemic heart-rate related ST segment episodes, we initially adapted
previously developed performance measures by adding performance matrices to
differentiate between these two types of ST episodes. For detailed description
of the performance measures and of the evaluation protocol see [1] and related
references.

The tool EVAL_ST provides first- (record-by-record) and second-order (aggregate
gross and average) performance statistics for evaluation and comparison of
transient ST episode detection algorithms. The tool allows assessing the accuracy
of: 1) detecting transient ST episodes, 2) distinguishing between ischemic and
non-ischemic heart-rate related ST episodes, 3) measuring ST episode durations
and ischemic ST episode durations, and 4) measuring ST segment deviations. The
tool also provides generation of performance distributions using a bootstrap
statistical technique for predicting real-world clinical performance and
robustness.

The core module of the tool (eval_st.c) supports command-line oriented (no
graphic display) user interface style thus enabling possible batch processing.
Input to the tool are ST segment annotation streams of a reference database
(e.g., LTST DB or ESC DB) and ST segment annotation streams of the evaluated
algorithms. Evaluation results are stored to output files.

An additional graphic user interface module in LessTif/Motif graphic environment
(geval_st.c) provides graphic display of the evaluation results on LINUX/UNIX
operating system. The main window of the graphic user interface provides display
of: overlapping reference and algorithm's ST episode annotation streams of the
records, first- and second-order performance statistics, performance matrices
regarding ST episode detection and differentiation between ischemic and
non-ischemic heart-rate related ST episodes, statistics regarding ST episode and
ischemic ST episode duration detection, scatter plot of ST segment deviation
measurements, and bootstrap estimates of expected real-world performance with
performance distributions. Detailed evaluation results are stored to output files
and also graphically displayed in secondary windows of the tool. 

We successfully used the tool for evaluating our two ST episode detection
algorithms [4,5] using the LTST DB and ESC DB. Besides the source code of the
EVAL_ST tool, the reference annotations of the LTST DB and ESC DB and the
annotations of the two algorithms are also provided.

Availability of the EVAL_ST tool to the world community should simplify as well
as promote easy and unique use of specific and complex performance measures and
evaluation protocol, help to easily compare different algorithms, encourage the
use of unique performance measures and evaluatio protocols in the field, and gain
the consistent use of standard performance measures.

--------------------------------------------------------------------------------

HOW TO COMPILE AND RUN THE EVAL_ST TOOL?

If you are using the Fedora, Core 2, you may wish to use the precompiled
versions of the EVAL_ST tool [1] (eval_st - command-line oriented interface,
geval_st - graphic user interface in Motif environment) which are located
in the bin directory (subdirectory to EVAL_ST directory).

After unziping and untaring the tool, entering the EVAL_ST directory, enter the
src directory. To compile the tool under Linux use the Makefile (Makefile_Linux)
script, to compile it under Solaris (UNIX) use Makefile_Motif_Solaris script. To
compile the tool, type following:

make eval_st

what will compile command-line oriented interface (eval_st.c) to eval_st.

make geval_st 

will compile graphic user interface in Motif environment (geval_st.c) to
geval_st.

make

will compile both, eval_st.c and geval_st.c to eval_st and geval_st.

The source files eval_st.c and geval_st.c are equal. The only difference is
value of a constant (motif) which allows conditional compiling and is set to
0 in the eval_st.c and to 1 in the geval_st.c source file.

After that, copy the compiled tool to EVAL_ST directory:

cp eval_st ..

cp eval_st.hlp ..

cp geval_st ..

cp geval_st.hlp ..

The files eval_st.hlp and geval_st.hlp contain help texts.

After that enter the EVAL_ST directory which is working (or home directory)
for evaluation:

cd ..

To run the command-line oriented interface type:

./eval_st -[a|r] <database>_<algorithm>.evl [record] -[s|i] [-e|-b [NR_TRIALS]]

To run the graphic user interface type:

./geval_st &

Detailed description of the use of the eval_st or geval_st tool is at the bottom
of this text. Working directroy (EVAL_ST), and input and output files of the tool
are described below. Output files will appear in the EVAL_ST directory. It is
also described how to add your own evaluation project.

--------------------------------------------------------------------------------

WHAT IS IN THE EVAL_ST DIRECTORY?

The directory EVAL_ST contains evaluation project files (*.evl files) and the
subdirectories associated with this files.

The names of the evaluation project files are of the form:

<database>_<algorithm>.evl 

         where <database> indicates the name of the database on which an
         algorithm was developed or tested and <algorithm> indicates the
         algorithm.

         Names of the databases are:

         ltsta, ltstb and ltstc (Long-Term ST Database, LTST DB, [2]
                                - annotation protocols A, B and C, respectively)
         esc         (European Society of Cardiology ST-T Database, ESC DB, [3])

         Names of the algorithms are:

         alg01  (Karhunen-Loeve transform-based algorithm [4])

         alg02  (Algorithm which tracks time-varying ST segment reference level
                 [5])

         Example: To evaluate the algorithm [5] using the protocol B of the
                  LTST DB you select evaluation project ltstb_alg02.evl.

The subdirectories associated with the evaluation projects are of the name
<database>_<algorithm>:

         <database>   indicates the name of the database
         <algorithm>  indicates the algorithm

         These subdirectories contain reference annotation files and algorithm's
         annotation files for the selected pair of database and algorithm.

Currently, there are following evaluation projects files in the EVAL_ST
directory:

esc_alg01.evl
esc_alg02.evl
ltsta_alg02.evl
ltstb_alg01.evl
ltstb_alg02.evl
ltstc_alg02.evl   

and following subdirectories associated with these evaluation projects:

esc_alg01
esc_alg02
ltsta_alg02
ltstb_alg01
ltstb_alg02
ltstc_alg02

In the other words, these directories contain reference annotation files and
algorithm's annotation files of the following evaluations:

1. The algorithm [4] was developed using the ESC DB [3] (evaluation project
   esc_alg01.evl) and then tested using the LTST DB (protocol B) [2] (evaluation
   project ltstb_alg01.evl)
2. The algorithm [5] was developed using the LTST DB (protocols A, B, and C) [2]
   (evaluation projects ltsta_alg02.evl, ltstb_alg02.evl, and ltstc_alg02.evl,
   respectively) and then tested using the ESC DB [3] (evaluation project
   esc_alg02.evl)

The files in the <database>_<algorithm> subdirectories are following:

<record>.hea
<record>.ref
<record>.cmr
<record>.dev
<record>.alg

These files and the <database>_<algorithm>.evl file are input files to the tool.

The

<record>.hea, <record>.ref and <record>.cmr

are reference annotation files of the database (<database>),

while

<record>.alg and <record>.dev

are annotation files of the algorithm (<algorithm>). Their content is described
below.

The subdirectory esc in the EVAL_ST directory contains reference annotation
files of the ESC DB [3] only (<record>.hea, <record>.ref, and <record>.cmr).

The subdirectory ltsta in the EVAL_ST directory contains reference annotation
files of the LTST DB, protocol A, [2] only (<record>.hea, <record>.ref, and
<record>.cmr).

The subdirectory ltstb in the EVAL_ST directory contains reference annotation
files of the LTST DB, protocol B, [2] only (<record>.hea, <record>.ref, and
<record>.cmr).

The subdirectory ltstc in the EVAL_ST directory contains reference annotation
files of the LTST DB, protocol C, [2] only (<record>.hea, <record>.ref, and
<record>.cmr).

The subdirectory bin in the EVAL_ST directory contains precompiled versions of
the EVAL_ST tool (eval_st - command-line oriented interface, and, geval_st -
graphic user interface in Motif graphic environment)

The subdirectory man in the EVAL_ST directory contains article [1] in the
PostScript and PDF format, this README file, and help files (eval_st.hlp and
geval_st.hlp).

The subdirectory src in the EVAL_ST directory contains source files eval_st.c
and geval_st.c, COPYING file, makefiles, this README file, and help files.

--------------------------------------------------------------------------------

WHAT ARE INPUT FILES TO THE TOOL?

<database>_<algorithm>.evl  (ASCII file)

         Evaluation project file which contains names of the records of the 
         <database> and the path to records' location in the <database>_<algorithm>
         directory when the <algorithm> is used. This file is in the working
         EVAL_ST directory.

         A row of this file is:
         <database>_<algorithm>/<record>,     i.e.,
         <path_to_the_record_files>/<record>

         Example:
         ltstb_alg02/s20011

         The files connected to each record (<record>) are following:

         <record>.hea
         <record>.ref
         <record>.cmr
         <record>.dev
         <record>.alg

         and are in the <database>_<algorithm> subdirectory. Their content is
         described next.

<record>.hea    (ASCII file, WFDB format)

         The header file of the record.

<record>.ref    (ASCII file)

         The reference annotation stream for the record <record>. This file
         contains the reference annotations for ischemic or non-ischemic
         heart-rate related ST episodes as annotated by developers of the
         database in each lead separately.
         Each row in this file is of the form:
         1. column - index of annotation [samples]
         2. column - type of annotation: -1 beginnig, -2 extrema, and -3 end
                     of the ST episode
         3. column - value 0 (at the beginning and end of an episode, if this
                     episode is ischemic)
         3. column - value 1 (at the beginning and end of an episode, if this
                     episode is non-ischemic heart-rate related)
         3. column - reference ST segment deviation measurement [uV] (at the
                     reference extrema of an episode, if this episode is ischemic
                     or non-ischemic heart-rate related)
         4. column - lead number in which the annotation was set (0, 1, or 2)

<record>.cmr    (ASCII file)
         Combined reference annotation stream for the record <record>.
         This file contains the reference annotations (ischemic or heart-rate 
         related) combined in the sense of logical OR function while ischemic 
         episode are more important during the combining process. It means 
         that having an ischemic episode in one lead and simultaneously a 
         heart-rate related episode in another lead, combined reference 
         annotation stream will contain the ischemic episode.
         Each row in this file is of the form:
         1. column - index of annotation [samples]
         2. column - type of annotation: -1 beginning, -2 extrema, and -3 end
                     of the ST episode
         3. column - value 0 (at the beginning and end of an episode, if this
                     episode is ischemic)
         3. column - value 1 (at the beginning and end of an episode, if this
                     episode is non-ischemic heart-rate related)
         3. column - reference ST segment deviation measurement [uV] (at the
                     reference extrema of an episode, if this episode is ischemic
                     or non-ischemic heart-rate related)
         4. column - lead number. For the algorithm's annotation stream this
                     value is always 0, since this is combined annotated
                     stream.

<record>.dev    (ASCII file)

         The ST segment deviation measurements measured by the algorithm 
         at the reference extrema annotations. This file is similar to the 
         <record>.ref file, while the reference ST segment deviation 
         measurements are here replaced by algorithm's measurements. 
         Each row in this file is of the form:
         1. column - index of annotation [samples]
         2. column - type of annotation: -1 beginning, -2 extrema, and -3 end
                     of episode
         3. column - value 0 (at the beginning and end of an episode, if this
                     episode is ischemic)
         3. column - value 1 (at the beginning and end of an episode, if this
                     episode is non-ischemic heart-rate related)
         3. column - algorithm's ST segment deviation measurement [uV] (at the
                     reference extrema of an episode, if this episode is ischemic
                     or non-ischemic heart-rate related)
         4. column - lead number in which the annotation was set (0, 1, or 2)

<record>.alg    (ASCII file)

         The algorithm's annotation stream for the record <record>.
         Each row in this file is of the form:
         1. column - index of annotation [in samples]
         2. column - type of annotation: -1 beginning, -2 extrema, and -3 end
                     of the ST episode
         3. column - value 0 (at the beginning and end of an episode, if this
                     episode is ischemic)
         3. column - value 1 (at the beginning and end of an episode, if this
                     episode is non-ischemic heart-rate related)
         3. column - ST segment deviation measurement [uV] (at the extrema of
                     algorithm's ST episode - within ST deviation detection
                     function)
         4. column - lead number. For the algorithm's annotation stream this
                     value is always 0 since this is combined annotated stream.

--------------------------------------------------------------------------------

WHAT ARE OUTPUT FILES OF THE TOOL?

All the output files appear in the working EVAL_ST directory.

<database>_<algorithm>_[ih|st].raw
 
         Aggregate evaluation statistics for the algorithm <algorithm> using the
         database <database>, while ischemic and non-ischemic heart-rate related
         episodes are differentiated (_ih.) or are considered as episodes of
         single type (_st.). This file contains numbers of annotated and detected
         episodes, percentage of annotated and detected episode durations,
         sensitivity (Se) and positive predictivity (+P) matrices, number of
         correctly detected, falsely detected and missed episodes, and statistics
         on episode detection Se/+P and episode duration detection Se/+P.

<database>_<algorithm>_[ih|st].rbr

         Evaluation statistics for the algorithm <algorithm> for each record of
         the database <database>, while ischemic and non-ischemic heart-rate
         related episodes are differentiated (_ih.) or are considered as episodes
         of single type (_st.). This file contains (for each record) a numbers
         of annotated and detected episodes, percentage of annotated and detected
         episode durations, Se and +P matrices, number of correctly detected,
         falsely detected and missed episodes, and statistics on episode
         detection Se/+P and episode duration detection Se/+P.

<database>_<algorithm>_[ih|st].ebe 

         Evaluation statistics of the ST segment deviation measurements at the
         episode extrema, while ischemic and non-ischemic heart-rate related
         episodes are differentiated (_ih.) or are considered as episodes of
         single type (_st.). File contains following ST segment deviation
         measurement statistics: mean [uV], standard deviation [uV], correlation
         coefficient, regression line of measurements, the percentage of
         measurements for which the absolute error differs by more than 100 uV,
         and the value of error which 95 % of the measurements do not exceed. 
         Additional information is: episode number, record, lead, time of the 
         extrema [h:m:s], algorithm measurement [uV], reference measurement 
         [uV], difference between algorithm and reference measurement [uV] 
         and error [%].

 <database>_<algorithm>_[ih|st].bts

         Evaluation statistics for the bootstrap evaluation. File contains
         aggregate statistics for Se and +P using the bootstrap method for the
         algorithm <algorithm> and database <database>, while ischemic and
         non-ischemic heart-rate related episodes are differentiated (_ih.) or
         are considered as episodes of single type (_st.). The aggregate
         statistics included are 5%, 16%, 84% and 95% confidence limits for
         the performance statistics, minimum and maximum performance statistics,
         median, mean, and difference between the mean performance statistics
         and the 5% confidence limits.

<database>_<algorithm>_[ih|st].<record> 

         Evaluation statistics of the algorithm  <algorithm> for the record
         <record> of the database <database>, while ischemic and non-ischemic
         heart-rate related episodes are distinguished (_ih.) or are considered
         as episodes of single type (_st.). File contains numbers of annotated
         and detected episodes, percentage of annotated and detected episode
         durations, Se and +P matrices, numbers of correctly detected, falsely
         detected and missed episodes, and statistics on episode detection Se/+P
         and episode duration detection Se/+P. In addition, the file contains
         also detailed matching report for each episode (reference and
         algorithm's).

<database>_<algorithm>_[ih|st].smrYYY

         Summary statistics, while ischemic and non-ischemic heart-rate related
         episodes are differentiated (_ih.) or are considered as episodes of
         single type (_st.). The YYY denotes the number of demanded summary
         statistics (000-999). The file contains summary statistics of the
         performed evaluations. These can include aggregate performance
         statistics, bootstrap statistics and ST segment deviation measurement
         statistics.

--------------------------------------------------------------------------------

HOW TO ADD YOUR OWN EVALUATION PROJECT?

In the EVAL_ST directory add your own <database>_<algorithm>.evl file and your
own <database>_<algorithm> subdirectory.

The <database>_<algorithm>.evl file should contain the paths to the record files
in the <database>_<algorithm> subdirectory.

The files in the <database>_<algorithm> subdirectory should be reference
annotation files (<record>.hea, <record>.ref and <record>.cmr) of the selected
database and annotation files of the evaluated algorithm (<record>.alg). If you
plan to evaluate also the ability to measure ST deviations, add the algorithm's
<record>.dev file.

For all these files, strictly follow the file format described above.

If your database is LTST DB, simply copy the reference annotation files from
the ltsta (protocol A), ltstb (protocol B), or ltstc (protocol C) subdirectory
(according to the selected annotation protocol) to your <database>_<algorithm>
subdirectory.

If your database is ESC DB, simply copy the reference annotation files from
the esc subdirectory to your <database>_<algorithm> subdirectory.

--------------------------------------------------------------------------------

HOW TO USE THE eval_st TOOL?

Usage of the eval_st (EVAL_ST) command-line oriented user interface to
evaluate transient ST segment episode detectors:

To write help text:

   eval_st -h

To evaluate an algorithm using a single record of a database:

   eval_st -r <database>_<algorithm>.evl <record> -[s|i]

           <database>_<algorithm>.evl is the evaluation project file,
           where <database> is database used and <algorithm> is the
           algorithm evaluated

           One of the options -s or -i has to be used but only one at
           one time.

   -r      Performs evaluation of the algorithm <algorithm> using the
           selected record <record> of the database <database>

   -s      All the episodes are considered as episodes of single type

   -i      Ischemic and non-ischemic heart-rate related episodes are 
           differentiated

   Examples:

       eval_st -r ltstb_alg02.evl s20011 -s

       Generates performance evaluation statistics for the record s20011
       while all the episodes are considered as episodes of single type

       eval_st -r ltstb_alg02.evl s20011 -i

       Generates performance statistics using the record s20011 for the
       ischemic episodes

To evaluate an algorithm using an entire database:

   eval_st -a <database>_<algorithm>.evl -[s|i] [-e|-b [NR_TRIALS]]

           <database>_<algorithm>.evl is the evaluation project file,
           where <database> is database used and <algorithm> is the
           algorithm evaluated

           One of the options -s or -i has to be used but only one at
           one time. Only one of the options -e or -b can be used in one
           time.

   -a      Performs aggregate statistics and record-by-record statistics
           for the algorithm <algorithm> using the selected database
           <database>

   -s      All the episodes are considered as episodes of single type

   -i      Ischemic and non-ischemic heart-rate related episodes are 
           differentiated

   -b [NR_TRIALS] Performs the bootstrap evaluation using NR_TRIALS
           bootstrap trials (default is 10000)

   -e      Performs evaluation of ST segment deviation measurements

   Examples:

       eval_st -a ltstb_alg02.evl -s

       Generates aggregate statistics and record-by-record statistics
       while all the episodes are considered as episodes of single type

       eval_st -a ltstb_alg02.evl -i

       Generates aggregate statistics and record-by-record statistics
       for ischemic episodes

       eval_st -a ltstb_alg02.evl -s -b

       Generates aggregate statistics and record-by-record statistics,
       and performs the bootstrap evaluation while all the episodes are
       considered as episodes of single type

       eval_st -a ltstb_alg02.evl -s -e

       Generates aggregates statistics and record-by-record statistics,
       and performs evaluation of ST segment deviation measurements

--------------------------------------------------------------------------------

HOW TO USE THE geval_st TOOL?

To run it, type:

./geval_st &

Usage of the geval_st (EVAL_ST) graphic user interface to evaluate
transient ST segment episode detectors:

Menu
                      Options                     Mnemonic  Accelerator
         Description

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

File
                      Open                        Alt+F O   Ctrl+O
    
         Opens evaluation project file <database>_<algorithm>.evl
         which contain the list of records given the algorithm
         <algorithm> and database for testing<database>

                      Examine                     Alt+F E   Ctrl+E
 
         Opens secondary window with information about statistics
         already perfromed given the selected evaluation project

                      Quit                        Alt+F Q   Ctrl+Q

         Quits the interface

Evaluation_mode
                      Ischemic/Heart-rate ...     Alt+E I   Ctrl+I
     
         Sets the evaluation mode when the ischemic and non-ischemic
         heart-rate related episodes are differentiated

                      ST segment changes          Alt+E S   Ctrl+S

         Sets the evaluation mode when all the episodes are considered
         as episodes of single type

Single_record
                      Record                      Alt+S R   Ctrl+R    
   
         Opens the dialog to select a singe record for evaluation
         and then performs evaluation for the selected record

Aggregate_statistics
                      Evaluate                    Alt+A E   Ctrl+A

         Performs aggregate statistics for the selected evaluation
         project

                      Extrema                     Alt+A X   Ctrl+X
       
         Performs evaluation of ST segment deviation measurements
         for the selected evaluation project

Bootstrap
                      Number_of_bootstrap_trials  Alt+B N   Ctrl+N
     
         Opens the dialog to set the nuber of bootstrap trials to 
         be used

                      Evaluate                    Alt+B E   Ctrl+B      
 
         Performs the bootstrap evaluation for the selected evaluation
         project

Summary
                      Compare                     Alt+M C   Ctrl+C
    
         Opens a "summary" secondary window and displays the relevant
         statistics of the last three performance evaluation projects

                      Differences                 Alt+M D   Ctrl+D
    
         Comparison of algorithms

Help                  Help                        Alt+H H   Ctrl+H

         This message

--------------------------------------------------------------------------------

REFERENCES

[1] Jager, F., Smrdel A., and Mark, R. G. An Open Source Tool to Evaluate
    Performance of Transient ST Segment Episode Detection Algorithms.
    Computers in Cardiology, 2004.
[2] Jager, F., Taddei, A., Moody, G. B., Emdin, M., Antolic, G., Dorn, R.,
    Smrdel, A., Marchesi, C., and Mark, R. G. Long-term ST database: a 
    reference for the development and evaluation of automated ischaemia 
    detectors and for the study of the dynamics of myocardial ischaemia.
    Med. & Biol. Eng. & Comput. 2003;  41:172-182.
[3] Taddei, A., Distante, G., Emdin, M., Pisani, P., Moody, G. B., 
    Zeelenberg, C., and Marchesi, C. The European ST-T database: standard for 
    evaluating systems for the analysis of ST-T changes in ambulatory 
    electrocardiography. Eur. Heart J. 1992; 13:1164-1172.
[4] Jager, F., Moody, G. B., and Mark, R. G. Detection of transient ST 
    segment episodes during ambulatory ECG monitoring.
    Comput. Biomed. Res. 1998; 31:305-322.
[5] Smrdel, A., and Jager, F. Automated detection of transient ST-segment 
    episodes in 24h electrocardiograms. 
    Med. & Biol. Eng. & Comput. 2004; 42:303-311.

--------------------------------------------------------------------------------