Challenge Open Access

You Snooze You Win - The PhysioNet Computing in Cardiology Challenge 2018

Published: Feb. 21, 2018. Version: 1.0.0


Please include the standard citation for PhysioNet: (show more options)
Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P. C., Mark, R., ... & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation [Online]. 101 (23), pp. e215–e220.

 

Introduction

At the end of last year, American scientists Jeffrey Hall, Michael Rosbash and Michael Young received a Nobel Prize in Physiology “for their discoveries of molecular mechanisms controlling the circadian rhythm"— the mechanism that regulates sleep (Osborn, 2017). The precise reasons why humans sleep (and even how much sleep we need) remains a topic of scientific inquiry. Contemporary theorists indicate that sleep may be responsible for learning and/or the clearing of neural waste products (Ogilvie and Patel, 2017).

While the precise reasons why we sleep are not perfectly understood, there is consensus on the importance of sleep for our overall health, and well-being. Inadequate sleep is associated with a wide range of negative outcomes including: impaired memory and learning, obesity, irritability, cardiovascular dysfunction, hypotension, diminished immune function (Harvard Medical School, 2006), depression (Nutt et al, 2008), and quality of life (Lee, 2009). Further studies even suggest causal links between quality of sleep, and important outcomes including mental health.

It follows that improving the quality of sleep could be used to improve a range of societal health outcomes, more generally. Of course, the treatment of sleep disorders is necessarily preceded by the diagnosis of sleep disorders. Traditionally, such diagnoses are developed in sleep laboratory settings, where polysomnography, audio, and videography of sleeping subject may be carefully inspected by sleep experts to identify potential sleep disorders.

One of the more well-studied sleep disorders is Obstructive Sleep Apnea Hypopnea Syndrome (or simply, apnea). Apneas are characterized by a complete collapse of the airway, leading to awakening, and consequent disturbances of sleep. While apneas are arguably the best understood of sleep disturbances, they are not the only cause of disturbance. Sleep arousals can also be spontaneous, result from teeth grinding, partial airway obstructions, or even snoring. In this year's PhysioNet Challenge we will use a variety of physiological signals, collected during polysomnographic sleep studies, to detect these other sources of arousal (non-apnea) during sleep.

Challenge Data

Data for this challenge were contributed by the Massachusetts General Hospital’s (MGH) Computational Clinical Neurophysiology Laboratory (CCNL), and the Clinical Data Animation Laboratory (CDAC). The dataset includes 1,985 subjects which were monitored at an MGH sleep laboratory for the diagnosis of sleep disorders. The data were partitioned into balanced training (n = 994), and test sets (n = 989).

The sleep stages of the subjects were annotated by clinical staff at the MGH according to the American Academy of Sleep Medicine (AASM) manual for the scoring of sleep. More specifically, the following six sleep stages were annotated in 30 second contiguous intervals: wakefulness, stage 1, stage 2, stage 3, rapid eye movement (REM), and undefined.

Certified sleep technologists at the MGH also annotated waveforms for the presence of arousals that interrupted the sleep of the subjects. The annotated arousals were classified as either: spontaneous arousals, respiratory effort related arousals (RERA), bruxisms, hypoventilations, hypopneas, apneas (central, obstructive and mixed), vocalizations, snores, periodic leg movements, Cheyne-Stokes breathing or partial airway obstructions.

The subjects had a variety of physiological signals recorded as they slept through the night including: electroencephalography (EEG), electrooculography (EOG), electromyography (EMG), electrocardiology (EKG), and oxygen saturation (SaO2). Excluding SaO2, all signals were sampled to 200 Hz and were measured in microvolts. For analytic convenience, SaO2 was resampled to 200 Hz, and is measured as a percentage.

Objective of the Challenge

The goal of the challenge is use information from the available signals to correctly classify target arousal regions. For the purpose of the Challenge, target arousals are defined as regions where either of the following conditions were met:

  • From 2 seconds before a RERA arousal begins, up to 10 seconds after it ends or,
  • From 2 seconds before a non-RERA, non-apnea arousal begins, up to 2 seconds after it ends.

Please note that regions falling within 10 seconds before or after a subject wakes up, has an apnea arousal, or a hypopnea arousal will not be scored for the Challenge.

We have pre-computed the target arousals for you. They are contained in a sample-wise vector (described below in “Accessing the Data”), marked by “1”. Regions that will not be scored are marked by a “-1”, and regions that will be penalized if marked by your algorithm are marked by “0”. You do not need to recompute these scores.

Accessing the Data

The Challenge data repository contains two directories (training and test) which are each approximately 135 GB in size. Each directory contains one subdirectory per subject (e.g. training/tr03-0005). Each subdirectory contains signal, header, and arousal files; for example:

  1. tr03-0005.mat: a Matlab V4 file containing the signal data.
  2. tr03-0005.hea: record header file - a text file which describes the format of the signal data.
  3. tr03-0005.arousal: arousal and sleep stage annotations, in WFDB annotation format.
  4. tr03-0005-arousal.mat: a Matlab V7 structure containing a sample-wise vector with three distinct values (+1, 0, -1) where:
    • +1: Designates arousal regions
    • 0: Designates non-arousal regions
    • -1: Designates regions that will not be scored

Table 1 lists functions that can be used to import the data into Python, Matlab, and C programs.

 

Table 1: Functions that can be used to import Challenge data.
File type Python Matlab C / C++
Signal (.mat) and header (.hea) files wfdb.rdrecord rdmat isigopen
Arousal annotation files (.arousal) wfdb.rdann rdann annopen
Arousal files (.mat) scipy.io.loadmat load libmatio

 

Submitting your Entry

Participants should use the provided signal and arousal data to develop a model that classifies test-set subjects. More specifically, for each subject in /test, participants must generate a .vec text file that describes the probability of arousal at each sample, such as:

0.001
0.000
0.024
0.051

The names of the generated annotation files should match the name of the test subject. For instance, test/te09-0094.mat should have a corresponding file named annotations/te09-0094.vec.

Entries must be submitted as a zip file containing:

  • All of the code and data files needed to train and run your algorithm
  • An AUTHORS.txt file containing the list of authors
  • A LICENSE.txt file containing the license for your code
  • The .vec files described above

To upload your entry, create a PhysioNet account (if you don't have one), and go to challenge.physionet.org. Entries must be uploaded prior to the deadline in order to be eligible.

Scoring

Your final algorithm will only be graded for its binary classification performance on target arousal and non-arousal regions (designated by +1 and 0 in teNN-NNNN-arousals.mat), measured by the area under the precision-recall curve. The area is defined as follows:

 

R j = n u m b e r o f a r o u s a l s a m p l e s w i t h p r e d i c t e d p r o b a b i l i t y ( j / 1000 ) o r g r e a t e r t o t a l n u m b e r o f a r o u s a l s a m p l e s R_j = \frac{number\,of\,arousal\,samples\,with\,predicted\,probability\,(j/1000)\,or\,greater}{total\,number\,of\,arousal\,samples}

 

P j = n u m b e r o f a r o u s a l s a m p l e s w i t h p r e d i c t e d p r o b a b i l i t y ( j / 1000 ) o r g r e a t e r t o t a l n u m b e r o f s a m p l e s w i t h p r e d i c t e d p r o b a b i l i t y ( j / 1000 ) o r g r e a t e r P_j = \frac{number\,of\,arousal\,samples\,with\,predicted\,probability\,(j/1000)\,or\,greater}{total\,number\,of\,samples\,with\,predicted\,probability\,(j/1000)\,or\,greater}

 

A U P R C = j P j ( R j R j + 1 ) AUPRC = \sum_{j} P_j(R_j -R_{j+1})

Note that this is the gross AUPRC (i.e., for each possible value of j, the precision and recall are calculated for the entire test database), which is not the same as averaging the AUPRC for each record.

Python (score2018.py) and Matlab/Octave (score2018.m) implementations of the scoring algorithm are available in the challenge files.

Sample Submission

Two simple example algorithms are provided and may be used as a template for your own submission:

Rules and Deadlines

Entrants may have an overall total of up to three submitted entries over both the unofficial and official phases of the competition (see Table 2). Following submission, entrants will receive an email confirming their submission and reporting how well their arousal annotations match those of the held-out test set.

All deadlines occur at noon GMT (UTC) on the dates mentioned below. If you do not know the difference between GMT and your local time, find out what it is before the deadline!

 

Table 2: Rules and deadlines.
  Start at noon GMT on Entry limit End at noon GMT on
Unofficial Phase 15 February 1 13 April
[Hiatus] 13 April 0 22 April
Official Phase 23 April 2 1 September
* Wildcard submissions due     15 July

 

All official entries must be received no later than noon GMT on Saturday, 1 September 2018. In the interest of fairness to all participants, late entries will not be accepted or scored. Entries that cannot be scored (because of missing components, improper formatting, or excessive run time) are not counted against the entry limits.

To be eligible for the open-source award, you must do all of the following:

  1. Submit at least one open-source entry that can be scored before the Phase I deadline (noon GMT on Monday, 9 April 2018).
  2. Submit at least one entry during the second phase (between noon GMT on Monday, 16 April 2018 and noon GMT on Saturday, 1 September 2018). Only your final entry will count for ranking.
  3. Entering an Abstract to CinC: Submit an acceptable abstract (about 299 words) on your work on the Challenge to Computing in Cardiology no later than 15 April 2018. Include the overall score for your Phase I entry in your abstract. Please select “PhysioNet/CinC Challenge” as the topic of your abstract, so it can be identified easily by the abstract review committee. You will be notified if your abstract has been accepted by email from CinC during the first week in June.
  4. Wildcard submissions: For teams who did not submit an abstract in time, or whose abstracts were not accepted, the team who submits the highest-scoring entry before 15 July 2018 will have another chance to compete, if they submit a high-quality abstract and present their work at the CinC conference. We will contact the winners in July with more information.
  5. Submit a full (4-page) paper on your work on the Challenge to CinC no later than the deadline of conference paper submission.
  6. Attend CinC 2018 (23-26 September 2018) in Maastricht and present your work there.

Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after CinC 2018 has taken place, so the competitors are able to discuss the results in a single forum. We expect a special issue from the journal Physiological Measurement to follow the conference and encourage all entrants (and those who missed the opportunity to compete or attend CinC 2018) to submit extended analysis and articles to that issue, taking into account the publications and discussions at CinC 2018.

Attending the Conference

If your abstract is accepted, you must log in to the conference site and agree that you will attend. Then, you must submit a full article describing your results and mark it as a preprint (for others to read) by September 15th. (Don't forget that the competition deadline is noon GMT on the 1st September - this deadline will *not* be extended.)

After agreeing to attend, you must register for the conference, pay the conference fee (prices go up after July ends), and secure a visa if you need one. See the Computing in Cardiology site for more information.

If you need a visa, we strongly suggest you register this week and begin the process. Visas can take months to issue and attendance is mandatory - you cannot receive a prize if you do not attend because defending your work is part of the Challenge. The conference (not PhysioNet) will supply you with a letter for your visa. Please see the CinC 2018 site for details on how to obtain that letter and who to contact. If you have any questions about this process, or are concerned about paying the conference fee before securing a visa, please contact the conference organizers, not PhysioNet.

If your abstract is rejected, then you have one more chance! This year we are introducing a 'wildcard' submission. On July the 15th, the top scoring entry that has not so far been accepted to CinC will be offered the opportunity to submit another (or a new) abstract to the conference system (containing full results). If the team can submit a quality abstract (with performance results) and register for the conference then it's members will be eligible for a prize (assuming they also attend the conference and present a poster). Don't forget, your abstract was probably rejected because it didn't contain any useful results (even on training data) and/or did not describe your methods well. So please pay attention to the abstract when submitting - it won't be automatic. We strongly believe that if you are unable to explain what you did and why, then the code is of very limited value.

We hope this is a suitable encouragement for teams that are either late to the Challenge or failed to secure a place at the conference to continue with their efforts in the competition. It would be a shame not to see potentially great works at the conference.

Look out for future announcements via the community discussion forum.

After the Challenge

As is customary, we hope to run a special issue in Physiological Measurement with a closing date of 31 January 2019. We will therefore encourage competitors (and non-competitors) to submit updates and further reworks based on the Challenge after the award ceremony at the Computing in Cardiology Conference in Maastricht in September.

Obtaining complimentary MATLAB licenses

The MathWorks has kindly decided to sponsor Physionet’s 2018 Challenge providing licenses. The MathWorks is offering to all teams that wish to use MATLAB, complimentary licenses. User can apply for a license and learn more about MATLAB support through The Mathworks’ PhysioNet Challenge link. If you have questions or need technical support, please contact The MathWorks at studentcompetitions@mathworks.com.

Challenge Results

Official results, as well as a paper describing the Challenge, are now available. Top scores were achieved by

  1. Matthew Howe-Patterson, Bahareh Pourbabaee, and Frederic Benard (0.54)
  2. Guðni Fannar Kristjansson, Heiðar Már Þráinsson, Hanna Ragnarsdóttir, Bragi Marinósson, Eysteinn Gunnlaugsson, Eysteinn Finnsson, Sigurður Ægir Jónsson, Halla Helgadóttir, and Jón Skírnir Ágústsson (0.45)
  3. Runnan He, Kuanquan Wang, Yang Liu, Na Zhao, Yongfeng Yuan, Qince Li, and Henggui Zhang (0.43)

An unofficial entry from Hongyang Li and Yuanfang Guan (who unfortunately missed the deadline to submit an abstract) achieved a score of 0.55.

Papers

The following paper is an introduction to the challenge topic, with a summary of the challenge results and a discussion of their implications. Please cite this publication when referencing the Challenge.

Mohammad M Ghassemi, Benjamin E Moody, Li-wei H Lehman, Christopher Song, Qiao Li, Haoqi Sun, Roger G Mark, M Brandon Westover, Gari D Clifford. You Snooze, You Win: the PhysioNet/Computing in Cardiology Challenge 2018.

Over 20 papers were presented at Computers in Cardiology 2018. These papers have been made available under the terms of the Creative Commons Attribution License 3.0 (CCAL). See this page for details. We wish to thank all of the authors for their contributions.


Share
Access

Access Policy:
Anyone can access the files, as long as they conform to the terms of the specified license.

License (for files):
Open Data Commons Attribution License v1.0

Corresponding Author
You must be logged in to view the contact information.

Files

Total uncompressed size: 45.8 MB.

Access the files

Folder Navigation: <base>/training
Name Size Modified
Parent Directory
tr03-0005
tr03-0029
tr03-0052
tr03-0061
tr03-0078
tr03-0079
tr03-0083
tr03-0086
tr03-0087
tr03-0092
tr03-0100
tr03-0103
tr03-0134
tr03-0135
tr03-0141
tr03-0146
tr03-0152
tr03-0166
tr03-0167
tr03-0179
tr03-0184
tr03-0187
tr03-0198
tr03-0212
tr03-0216
tr03-0241
tr03-0251
tr03-0256
tr03-0257
tr03-0267
tr03-0287
tr03-0289
tr03-0299
tr03-0300
tr03-0304
tr03-0314
tr03-0322
tr03-0333
tr03-0373
tr03-0392
tr03-0394
tr03-0396
tr03-0401
tr03-0402
tr03-0411
tr03-0413
tr03-0418
tr03-0426
tr03-0428
tr03-0434
tr03-0473
tr03-0485
tr03-0487
tr03-0494
tr03-0523
tr03-0532
tr03-0536
tr03-0541
tr03-0560
tr03-0567
tr03-0584
tr03-0592
tr03-0618
tr03-0623
tr03-0645
tr03-0672
tr03-0674
tr03-0678
tr03-0686
tr03-0697
tr03-0708
tr03-0713
tr03-0715
tr03-0726
tr03-0733
tr03-0743
tr03-0759
tr03-0764
tr03-0773
tr03-0780
tr03-0785
tr03-0792
tr03-0793
tr03-0796
tr03-0802
tr03-0816
tr03-0862
tr03-0863
tr03-0876
tr03-0879
tr03-0885
tr03-0900
tr03-0904
tr03-0906
tr03-0907
tr03-0909
tr03-0919
tr03-0921
tr03-0933
tr03-0940
tr03-0955
tr03-0982
tr03-1004
tr03-1010
tr03-1013
tr03-1023
tr03-1050
tr03-1056
tr03-1075
tr03-1082
tr03-1100
tr03-1112
tr03-1115
tr03-1119
tr03-1129
tr03-1135
tr03-1143
tr03-1155
tr03-1160
tr03-1181
tr03-1183
tr03-1185
tr03-1197
tr03-1226
tr03-1231
tr03-1292
tr03-1294
tr03-1302
tr03-1319
tr03-1333
tr03-1346
tr03-1362
tr03-1367
tr03-1371
tr03-1373
tr03-1374
tr03-1387
tr03-1389
tr03-1397
tr04-0008
tr04-0014
tr04-0015
tr04-0020
tr04-0029
tr04-0030
tr04-0039
tr04-0041
tr04-0075
tr04-0079
tr04-0117
tr04-0120
tr04-0121
tr04-0133
tr04-0144
tr04-0187
tr04-0196
tr04-0208
tr04-0209
tr04-0210
tr04-0214
tr04-0216
tr04-0223
tr04-0227
tr04-0231
tr04-0239
tr04-0247
tr04-0255
tr04-0261
tr04-0265
tr04-0269
tr04-0275
tr04-0279
tr04-0287
tr04-0332
tr04-0342
tr04-0358
tr04-0362
tr04-0374
tr04-0393
tr04-0400
tr04-0402
tr04-0404
tr04-0417
tr04-0425
tr04-0468
tr04-0471
tr04-0474
tr04-0503
tr04-0536
tr04-0551
tr04-0568
tr04-0569
tr04-0570
tr04-0579
tr04-0583
tr04-0587
tr04-0603
tr04-0611
tr04-0613
tr04-0631
tr04-0649
tr04-0653
tr04-0658
tr04-0671
tr04-0675
tr04-0679
tr04-0695
tr04-0699
tr04-0702
tr04-0703
tr04-0710
tr04-0726
tr04-0741
tr04-0746
tr04-0762
tr04-0785
tr04-0805
tr04-0806
tr04-0808
tr04-0825
tr04-0829
tr04-0864
tr04-0878
tr04-0882
tr04-0887
tr04-0900
tr04-0908
tr04-0909
tr04-0913
tr04-0915
tr04-0931
tr04-0933
tr04-0939
tr04-0959
tr04-0960
tr04-0963
tr04-0966
tr04-0983
tr04-1006
tr04-1013
tr04-1021
tr04-1023
tr04-1032
tr04-1034
tr04-1064
tr04-1078
tr04-1087
tr04-1096
tr04-1097
tr04-1105
tr04-1117
tr05-0011
tr05-0028
tr05-0048
tr05-0067
tr05-0069
tr05-0074
tr05-0114
tr05-0119
tr05-0174
tr05-0193
tr05-0206
tr05-0226
tr05-0243
tr05-0253
tr05-0256
tr05-0260
tr05-0266
tr05-0284
tr05-0292
tr05-0301
tr05-0312
tr05-0315
tr05-0326
tr05-0327
tr05-0332
tr05-0334
tr05-0348
tr05-0356
tr05-0379
tr05-0400
tr05-0403
tr05-0404
tr05-0405
tr05-0411
tr05-0418
tr05-0443
tr05-0449
tr05-0456
tr05-0459
tr05-0471
tr05-0497
tr05-0501
tr05-0531
tr05-0549
tr05-0550
tr05-0570
tr05-0572
tr05-0574
tr05-0596
tr05-0610
tr05-0613
tr05-0627
tr05-0629
tr05-0635
tr05-0646
tr05-0647
tr05-0652
tr05-0655
tr05-0664
tr05-0685
tr05-0686
tr05-0701
tr05-0707
tr05-0715
tr05-0724
tr05-0738
tr05-0739
tr05-0744
tr05-0782
tr05-0784
tr05-0800
tr05-0812
tr05-0827
tr05-0838
tr05-0849
tr05-0857
tr05-0860
tr05-0864
tr05-0880
tr05-0900
tr05-0905
tr05-0910
tr05-0924
tr05-0932
tr05-0949
tr05-0994
tr05-1034
tr05-1042
tr05-1060
tr05-1097
tr05-1112
tr05-1118
tr05-1121
tr05-1128
tr05-1129
tr05-1135
tr05-1146
tr05-1149
tr05-1163
tr05-1167
tr05-1175
tr05-1176
tr05-1183
tr05-1190
tr05-1197
tr05-1215
tr05-1222
tr05-1225
tr05-1229
tr05-1233
tr05-1234
tr05-1238
tr05-1243
tr05-1247
tr05-1258
tr05-1268
tr05-1270
tr05-1277
tr05-1302
tr05-1313
tr05-1317
tr05-1324
tr05-1356
tr05-1371
tr05-1372
tr05-1375
tr05-1377
tr05-1385
tr05-1386
tr05-1404
tr05-1406
tr05-1419
tr05-1434
tr05-1452
tr05-1453
tr05-1457
tr05-1464
tr05-1475
tr05-1489
tr05-1497
tr05-1510
tr05-1513
tr05-1521
tr05-1528
tr05-1547
tr05-1551
tr05-1554
tr05-1558
tr05-1570
tr05-1572
tr05-1575
tr05-1591
tr05-1604
tr05-1624
tr05-1649
tr05-1653
tr05-1659
tr05-1675
tr05-1681
tr05-1684
tr05-1687
tr05-1688
tr05-1689
tr05-1716
tr05-1720
tr05-1728
tr06-0014
tr06-0017
tr06-0041
tr06-0050
tr06-0062
tr06-0084
tr06-0103
tr06-0107
tr06-0108
tr06-0117
tr06-0122
tr06-0126
tr06-0154
tr06-0166
tr06-0168
tr06-0177
tr06-0182
tr06-0193
tr06-0202
tr06-0204
tr06-0221
tr06-0232
tr06-0234
tr06-0242
tr06-0249
tr06-0262
tr06-0282
tr06-0302
tr06-0307
tr06-0313
tr06-0315
tr06-0317
tr06-0318
tr06-0332
tr06-0333
tr06-0347
tr06-0362
tr06-0363
tr06-0379
tr06-0390
tr06-0397
tr06-0404
tr06-0418
tr06-0429
tr06-0447
tr06-0449
tr06-0452
tr06-0475
tr06-0488
tr06-0503
tr06-0515
tr06-0517
tr06-0532
tr06-0536
tr06-0538
tr06-0544
tr06-0547
tr06-0556
tr06-0567
tr06-0570
tr06-0576
tr06-0579
tr06-0584
tr06-0609
tr06-0627
tr06-0644
tr06-0677
tr06-0694
tr06-0696
tr06-0705
tr06-0709
tr06-0721
tr06-0750
tr06-0764
tr06-0769
tr06-0771
tr06-0773
tr06-0780
tr06-0785
tr06-0791
tr06-0797
tr06-0802
tr06-0812
tr06-0815
tr06-0825
tr06-0827
tr06-0843
tr06-0850
tr06-0862
tr06-0865
tr06-0876
tr06-0880
tr06-0883
tr06-1010
tr06-1117
tr06-1123
tr07-0016
tr07-0023
tr07-0027
tr07-0030
tr07-0034
tr07-0036
tr07-0040
tr07-0043
tr07-0056
tr07-0084
tr07-0100
tr07-0102
tr07-0105
tr07-0123
tr07-0125
tr07-0126
tr07-0127
tr07-0153
tr07-0162
tr07-0168
tr07-0178
tr07-0179
tr07-0191
tr07-0197
tr07-0212
tr07-0214
tr07-0230
tr07-0231
tr07-0232
tr07-0235
tr07-0236
tr07-0246
tr07-0247
tr07-0257
tr07-0262
tr07-0281
tr07-0283
tr07-0291
tr07-0293
tr07-0325
tr07-0343
tr07-0359
tr07-0380
tr07-0394
tr07-0401
tr07-0409
tr07-0416
tr07-0418
tr07-0451
tr07-0455
tr07-0458
tr07-0466
tr07-0489
tr07-0490
tr07-0494
tr07-0509
tr07-0517
tr07-0540
tr07-0542
tr07-0549
tr07-0564
tr07-0566
tr07-0568
tr07-0575
tr07-0579
tr07-0585
tr07-0586
tr07-0593
tr07-0594
tr07-0601
tr07-0602
tr07-0605
tr07-0610
tr07-0615
tr07-0625
tr07-0642
tr07-0650
tr07-0662
tr07-0665
tr07-0675
tr07-0681
tr07-0687
tr07-0691
tr07-0703
tr07-0709
tr07-0718
tr07-0752
tr07-0761
tr07-0770
tr07-0771
tr07-0782
tr07-0786
tr07-0796
tr07-0804
tr07-0820
tr07-0828
tr07-0830
tr07-0841
tr07-0848
tr07-0861
tr07-0865
tr07-0874
tr07-0891
tr07-0894
tr07-0895
tr07-0902
tr07-0906
tr07-0909
tr07-0913
tr07-0918
tr08-0010
tr08-0012
tr08-0021
tr08-0030
tr08-0105
tr08-0111
tr08-0113
tr08-0137
tr08-0139
tr08-0140
tr08-0156
tr08-0157
tr08-0159
tr08-0168
tr08-0183
tr08-0192
tr08-0214
tr08-0245
tr08-0256
tr08-0266
tr08-0277
tr08-0280
tr08-0287
tr08-0295
tr08-0303
tr08-0310
tr08-0314
tr08-0315
tr08-0336
tr08-0347
tr08-0348
tr08-0351
tr08-0353
tr08-0356
tr09-0040
tr09-0044
tr09-0049
tr09-0050
tr09-0051
tr09-0065
tr09-0068
tr09-0070
tr09-0082
tr09-0086
tr09-0122
tr09-0129
tr09-0159
tr09-0175
tr09-0186
tr09-0205
tr09-0217
tr09-0220
tr09-0237
tr09-0246
tr09-0296
tr09-0328
tr09-0331
tr09-0334
tr09-0340
tr09-0423
tr09-0436
tr09-0453
tr09-0469
tr09-0471
tr09-0473
tr09-0478
tr09-0489
tr09-0507
tr09-0511
tr09-0520
tr09-0527
tr09-0533
tr09-0541
tr09-0553
tr09-0558
tr09-0567
tr09-0568
tr09-0575
tr09-0580
tr09-0593
tr09-0600
tr09-0636
tr09-0647
tr10-0006
tr10-0036
tr10-0044
tr10-0047
tr10-0059
tr10-0077
tr10-0084
tr10-0094
tr10-0108
tr10-0114
tr10-0141
tr10-0169
tr10-0191
tr10-0247
tr10-0251
tr10-0263
tr10-0306
tr10-0336
tr10-0350
tr10-0363
tr10-0368
tr10-0392
tr10-0413
tr10-0423
tr10-0442
tr10-0447
tr10-0477
tr10-0509
tr10-0511
tr10-0512
tr10-0536
tr10-0554
tr10-0591
tr10-0592
tr10-0598
tr10-0621
tr10-0626
tr10-0628
tr10-0635
tr10-0704
tr10-0707
tr10-0728
tr10-0731
tr10-0740
tr10-0752
tr10-0762
tr10-0768
tr10-0771
tr10-0808
tr10-0832
tr10-0853
tr10-0864
tr10-0869
tr10-0872
tr11-0006
tr11-0016
tr11-0024
tr11-0025
tr11-0029
tr11-0042
tr11-0050
tr11-0059
tr11-0064
tr11-0072
tr11-0074
tr11-0080
tr11-0101
tr11-0120
tr11-0125
tr11-0200
tr11-0324
tr11-0326
tr11-0334
tr11-0335
tr11-0338
tr11-0339
tr11-0347
tr11-0350
tr11-0354
tr11-0377
tr11-0403
tr11-0417
tr11-0432
tr11-0436
tr11-0452
tr11-0457
tr11-0459
tr11-0471
tr11-0479
tr11-0481
tr11-0503
tr11-0509
tr11-0510
tr11-0516
tr11-0537
tr11-0563
tr11-0565
tr11-0573
tr11-0577
tr11-0584
tr11-0587
tr11-0590
tr11-0592
tr11-0612
tr11-0640
tr11-0644
tr11-0655
tr11-0659
tr11-0676
tr11-0708
tr11-0734
tr11-0736
tr11-0767
tr11-0786
tr11-0792
tr12-0003
tr12-0009
tr12-0015
tr12-0051
tr12-0058
tr12-0061
tr12-0064
tr12-0095
tr12-0097
tr12-0106
tr12-0108
tr12-0115
tr12-0120
tr12-0121
tr12-0122
tr12-0135
tr12-0169
tr12-0170
tr12-0173
tr12-0185
tr12-0209
tr12-0229
tr12-0241
tr12-0243
tr12-0244
tr12-0247
tr12-0248
tr12-0253
tr12-0255
tr12-0261
tr12-0277
tr12-0279
tr12-0299
tr12-0308
tr12-0310
tr12-0319
tr12-0321
tr12-0339
tr12-0348
tr12-0363
tr12-0364
tr12-0368
tr12-0375
tr12-0395
tr12-0404
tr12-0408
tr12-0410
tr12-0412
tr12-0414
tr12-0425
tr12-0426
tr12-0438
tr12-0439
tr12-0440
tr12-0441
tr12-0448
tr12-0463
tr12-0476
tr12-0477
tr12-0481
tr12-0490
tr12-0492
tr12-0493
tr12-0497
tr12-0503
tr12-0509
tr12-0512
tr12-0515
tr12-0519
tr12-0520
tr12-0522
tr12-0530
tr12-0532
tr12-0549
tr12-0559
tr12-0560
tr12-0573
tr12-0607
tr12-0619
tr12-0646
tr12-0672
tr12-0681
tr12-0682
tr12-0684
tr12-0685
tr12-0687
tr12-0697
tr12-0804
tr13-0056
tr13-0058
tr13-0069
tr13-0076
tr13-0080
tr13-0084
tr13-0093
tr13-0101
tr13-0128
tr13-0129
tr13-0140
tr13-0145
tr13-0164
tr13-0170
tr13-0187
tr13-0202
tr13-0204
tr13-0208
tr13-0212
tr13-0220
tr13-0226
tr13-0250
tr13-0255
tr13-0364
tr13-0367
tr13-0372
tr13-0374
tr13-0379
tr13-0383
tr13-0387
tr13-0425
tr13-0438
tr13-0447
tr13-0448
tr13-0464
tr13-0475
tr13-0492
tr13-0493
tr13-0505
tr13-0508
tr13-0517
tr13-0518
tr13-0525
tr13-0538
tr13-0544
tr13-0547
tr13-0549
tr13-0551
tr13-0566
tr13-0567
tr13-0576
tr13-0589
tr13-0607
tr13-0615
tr13-0621
tr13-0627
tr13-0638
tr13-0646
tr13-0653
tr13-0681
tr13-0685
tr13-0686
tr13-0801
tr13-0819
tr14-0003
tr14-0011
tr14-0016
tr14-0037
tr14-0064
tr14-0102
tr14-0110
tr14-0112
tr14-0167
tr14-0182
tr14-0185
tr14-0193
tr14-0206
tr14-0231
tr14-0240
tr14-0258
tr14-0268
tr14-0272
tr14-0276
tr14-0278
tr14-0291
ANNOTATORS (download) 34 B 2018-02-20
MD5SUMS (download) 236.9 KB 2018-05-07
RECORDS (download) 10.7 KB 2018-02-20
SHA1SUMS (download) 268.0 KB 2018-05-07
SHA256SUMS (download) 361.2 KB 2018-05-07