[Error] "No such file '../../data/multitask/train_listfile.csv' "

Hi,

First of all, great work! We are looking forward to contributing our own benchmarks and tests!

Regarding the error I’m receiving, it appears that I am successfully implementing the scripts throughout the README.

However, when I try to run one of the models such as: python -u main.py --network lstm_2layer --dim 512 --mode train --batch_size 8 --log_every 30

I receive the following error: IOError: [Errno 2] No such file or directory: '../../data/multitask/train_listfile.csv'

I checked to make sure that I am in the mimic3models/phenotyping/ directory. Also, as the error appears to state I do not have the listed file inside of the mimic3-benchmarks-master/data directory.

Here is the I/O from the terminal if that helps: `bwimbp:mimic3-benchmarks-master bwi$ source activate py27 (py27) bwimbp:mimic3-benchmarks-master bwi$ export PYTHONPATH=$PYTHONPATH:/Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master (py27) bwimbp:mimic3-benchmarks-master bwi

$ python scripts/extract_subjects.py /Users/bwi/Documents/AD-LSTM-Benchmark/MIMIC/physionet.org/works/MIMICIIIClinicalDatabaseDemo/files/version_1_4 data/root/ 'START: 136 129 100 int64 REMOVE ICU TRANSFERS: 126 120 94 REMOVE MULTIPLE STAYS PER ADMIT: 115 115 90 /Users/bwi/anaconda/envs/py27/lib/python2.7/site-packages/pandas/core/indexing.py:179: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy self._setitem_with_indexer(indexer, value) REMOVE PATIENTS AGE < 18: 114 114 89 SUBJECT 89 of 89…DONE! SUBJECT 89 of 89…DONE! processing CHARTEVENTS: ROW 100000 of 263201376…last write (104) 667 rows for processing CHARTEVENTS: ROW 200000 of 263201376…last write (298) 25438 rows foprocessing CHARTEVENTS: ROW 300000 of 263201376…last write (72(py27) bwimbp:mimic3-benchmarks-master bwi$ python scripts/validate_events.py data/root/ Namespace(subjects_root_path=‘data/root/’) processed 1 / 89
(‘n_events’, 747793, ‘emptyhadm’, 13184, ‘noicustay’, 45336, ‘recovered’, 45336, ‘couldnotrecover’, 0, ‘icustaymissinginstays’, 1200, ‘nohadminstay’, 86481) (py27) bwimbp:mimic3-benchmarks-master bwi$ python scripts/extract_episodes_from_subjects.py data/root/ Subject 10006: reading…got 1 stays, 19 diagnoses, 1164 events…cleaning and converting to time series…/Users/bwi/anaconda/envs/py27/lib/python2.7/site-packages/pandas/core/indexing.py:179: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy self._setitem_with_indexer(indexer, value) /Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master/mimic3benchmark/preprocessing.py:157: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy v.ix[idx] = np.nan /Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master/mimic3benchmark/preprocessing.py:149: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy v.ix[idx] = np.nan extracting separate episodes… 206504 DONE! Subject 10011: reading…got 1 stays, 6 diagnoses, 12486 events…cleaning and converting to time series…extracting separate episodes… 232110 DONE! Subject 10013: reading…got 1 stays, 8 diagnoses, 2348 events…cleaning and converting to time series…extracting separate episodes… 264446 DONE! Subject 10017: reading…got 1 stays, 14 diagnoses, 2063 events…cleaning and converting to time series…extracting separate episodes… 204881 DONE! Subject 10019: reading…got 1 stays, 12 diagnoses, 2717 events…cleaning and converting to time series…extracting separate episodes… 228977 DONE! Subject 10026: reading…got 1 stays, 7 diagnoses, 3909 events…cleaning and converting to time series…extracting separate episodes… 277021 DONE! Subject 10027: reading…got 1 stays, 12 diagnoses, 10688 events…cleaning and converting to time series…extracting separate episodes… 286020 DONE! Subject 10029: reading…got 1 stays, 21 diagnoses, 3381 events…cleaning and converting to time series…extracting separate episodes… 226055 DONE! Subject 10032: reading…got 1 stays, 8 diagnoses, 3215 events…cleaning and converting to time series…extracting separate episodes… 267090 DONE! Subject 10033: reading…got 1 stays, 12 diagnoses, 1221 events…cleaning and converting to time series…extracting separate episodes… 254543 DONE! Subject 10035: reading…got 1 stays, 3 diagnoses, 2019 events…cleaning and converting to time series…extracting separate episodes… 296804 DONE! Subject 10036: reading…got 1 stays, 12 diagnoses, 1735 events…cleaning and converting to time series…extracting separate episodes… 227834 DONE! Subject 10038: reading…got 1 stays, 20 diagnoses, 3873 events…cleaning and converting to time series…extracting separate episodes… 235482 DONE! Subject 10040: reading…got 1 stays, 9 diagnoses, 3780 events…cleaning and converting to time series…extracting separate episodes… 272047 DONE! Subject 10042: reading…got 1 stays, 9 diagnoses, 10130 events…cleaning and converting to time series…/Users/bwi/anaconda/envs/py27/lib/python2.7/site-packages/pandas/core/generic.py:2999: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy self[name] = value extracting separate episodes… 258147 DONE! Subject 10043: reading…got 1 stays, 8 diagnoses, 2555 events…cleaning and converting to time series…extracting separate episodes… 266122 DONE! Subject 10044: reading…got 1 stays, 9 diagnoses, 4902 events…cleaning and converting to time series…extracting separate episodes… 270154 DONE! Subject 10046: reading…got 1 stays, 3 diagnoses, 1979 events…cleaning and converting to time series…extracting separate episodes… 213289 DONE! Subject 10056: reading…got 1 stays, 8 diagnoses, 1305 events…cleaning and converting to time series…extracting separate episodes… 285789 DONE! Subject 10059: reading…got 2 stays, 21 diagnoses, 14133 events…cleaning and converting to time series…extracting separate episodes… 215460/Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master/mimic3benchmark/subject.py:41: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy events[‘HOURS’] = (events.CHARTTIME - dt).apply(lambda s: s / np.timedelta64(1, ‘s’)) / 60./60 248755 DONE! Subject 10061: reading…got 1 stays, 9 diagnoses, 25884 events…cleaning and converting to time series…extracting separate episodes… 223177 DONE! Subject 10064: reading…got 1 stays, 4 diagnoses, 505 events…cleaning and converting to time series…extracting separate episodes… 231809 DONE! Subject 10067: reading…got 1 stays, 8 diagnoses, 73 events…cleaning and converting to time series…extracting separate episodes… 236674 DONE! Subject 10069: reading…got 1 stays, 8 diagnoses, 16658 events…cleaning and converting to time series…extracting separate episodes… 290490 DONE! Subject 10074: reading…got 1 stays, 11 diagnoses, 851 events…cleaning and converting to time series…extracting separate episodes… 224021 DONE! Subject 10076: reading…got 1 stays, 15 diagnoses, 11258 events…cleaning and converting to time series…extracting separate episodes… 201006 DONE! Subject 10088: reading…got 3 stays, 55 diagnoses, 10799 events…cleaning and converting to time series…extracting separate episodes… 256345 249695 277403 DONE! Subject 10089: reading…got 1 stays, 5 diagnoses, 3066 events…cleaning and converting to time series…extracting separate episodes… 246080 DONE! Subject 10090: reading…got 1 stays, 7 diagnoses, 1949 events…cleaning and converting to time series…extracting separate episodes… 295741 DONE! Subject 10093: reading…got 1 stays, 10 diagnoses, 1539 events…cleaning and converting to time series…extracting separate episodes… 251573 DONE! Subject 10094: reading…got 2 stays, 19 diagnoses, 10284 events…cleaning and converting to time series…extracting separate episodes… 243600 273347 DONE! Subject 10098: reading…got 1 stays, 5 diagnoses, 1400 events…cleaning and converting to time series…extracting separate episodes… 262670 DONE! Subject 10101: reading…got 1 stays, 7 diagnoses, 1086 events…cleaning and converting to time series…extracting separate episodes… 293280 DONE! Subject 10102: reading…got 1 stays, 12 diagnoses, 3135 events…cleaning and converting to time series…extracting separate episodes… 223870 DONE! Subject 10104: reading…got 1 stays, 8 diagnoses, 934 events…cleaning and converting to time series…extracting separate episodes… 204201 DONE! Subject 10106: reading…got 1 stays, 5 diagnoses, 2091 events…cleaning and converting to time series…extracting separate episodes… 217960 DONE! Subject 10111: reading…got 1 stays, 9 diagnoses, 10494 events…cleaning and converting to time series…extracting separate episodes… 263934 DONE! Subject 10112: reading…got 1 stays, 9 diagnoses, 4694 events…cleaning and converting to time series…extracting separate episodes… 224063 DONE! Subject 10114: reading…got 1 stays, 9 diagnoses, 3502 events…cleaning and converting to time series…extracting separate episodes… 234989 DONE! Subject 10117: reading…got 1 stays, 5 diagnoses, 982 events…cleaning and converting to time series…extracting separate episodes… 214861 DONE! Subject 10119: reading…got 2 stays, 14 diagnoses, 17327 events…cleaning and converting to time series…extracting separate episodes… 247686 205589 DONE! Subject 10124: reading…got 1 stays, 16 diagnoses, 4053 events…cleaning and converting to time series…extracting separate episodes… 261764 DONE! Subject 10126: reading…got 1 stays, 13 diagnoses, 88206 events…cleaning and converting to time series…extracting separate episodes… 249805 DONE! Subject 10130: reading…got 1 stays, 8 diagnoses, 2124 events…cleaning and converting to time series…extracting separate episodes… 241562 DONE! Subject 10132: reading…got 1 stays, 9 diagnoses, 1181 events…cleaning and converting to time series…extracting separate episodes… 292910 DONE! Subject 40124: reading…got 2 stays, 14 diagnoses, 7570 events…cleaning and converting to time series…extracting separate episodes… 279554 269173 DONE! Subject 40177: reading…got 1 stays, 19 diagnoses, 1445 events…cleaning and converting to time series…extracting separate episodes… 285750 DONE! Subject 40204: reading…got 1 stays, 14 diagnoses, 855 events…cleaning and converting to time series…extracting separate episodes… 285369 DONE! Subject 40277: reading…got 1 stays, 10 diagnoses, 1539 events…cleaning and converting to time series…extracting separate episodes… 219013 DONE! Subject 40286: reading…got 1 stays, 10 diagnoses, 2006 events…cleaning and converting to time series…extracting separate episodes… 238399 DONE! Subject 40310: reading…got 1 stays, 17 diagnoses, 8757 events…cleaning and converting to time series…extracting separate episodes… 204132 DONE! Subject 40456: reading…got 1 stays, 9 diagnoses, 2240 events…cleaning and converting to time series…extracting separate episodes… 242790 DONE! Subject 40503: reading…got 1 stays, 14 diagnoses, 1068 events…cleaning and converting to time series…extracting separate episodes… 293429 DONE! Subject 40595: reading…got 1 stays, 24 diagnoses, 12066 events…cleaning and converting to time series…extracting separate episodes… 276601 DONE! Subject 40601: reading…got 1 stays, 11 diagnoses, 2564 events…cleaning and converting to time series…extracting separate episodes… 279529 DONE! Subject 40612: reading…got 1 stays, 11 diagnoses, 4008 events…cleaning and converting to time series…extracting separate episodes… 231005 DONE! Subject 40655: reading…got 1 stays, 18 diagnoses, 1163 events…cleaning and converting to time series…extracting separate episodes… 220016 DONE! Subject 40687: reading…got 1 stays, 14 diagnoses, 1698 events…cleaning and converting to time series…extracting separate episodes… 279183 DONE! Subject 41795: reading…got 2 stays, 38 diagnoses, 17160 events…cleaning and converting to time series…extracting separate episodes… 216185 293178 DONE! Subject 41914: reading…got 1 stays, 31 diagnoses, 17864 events…cleaning and converting to time series…extracting separate episodes… 256338 DONE! Subject 41976: reading…got 14 stays, 251 diagnoses, 41135 events…cleaning and converting to time series…extracting separate episodes… 285272 205170 253931 234541 265505 285353 263095 213315 209797 242680 280943 291067 216493 267267 DONE! Subject 41983: reading…got 1 stays, 16 diagnoses, 2427 events…cleaning and converting to time series…extracting separate episodes… 283875 DONE! Subject 42033: reading…got 1 stays, 7 diagnoses, 1005 events…cleaning and converting to time series…extracting separate episodes… 256542 DONE! Subject 42066: reading…got 1 stays, 10 diagnoses, 5326 events…cleaning and converting to time series…extracting separate episodes… 244243 DONE! Subject 42075: reading…got 1 stays, 17 diagnoses, 18326 events…cleaning and converting to time series…extracting separate episodes… 298685 DONE! Subject 42135: reading…got 2 stays, 34 diagnoses, 30076 events…cleaning and converting to time series…extracting separate episodes… 210164 281609 DONE! Subject 42199: reading…got 1 stays, 15 diagnoses, 6311 events…cleaning and converting to time series…extracting separate episodes… 274509 DONE! Subject 42231: reading…got 1 stays, 13 diagnoses, 1177 events…cleaning and converting to time series…extracting separate episodes… 254635 DONE! Subject 42275: reading…got 1 stays, 16 diagnoses, 880 events…cleaning and converting to time series…extracting separate episodes… 290478 DONE! Subject 42292: reading…got 1 stays, 18 diagnoses, 1302 events…cleaning and converting to time series…extracting separate episodes… 277238 DONE! Subject 42302: reading…got 1 stays, 23 diagnoses, 1005 events…cleaning and converting to time series…extracting separate episodes… 251281 DONE! Subject 42321: reading…got 1 stays, 18 diagnoses, 2645 events…cleaning and converting to time series…extracting separate episodes… 201204 DONE! Subject 42346: reading…got 2 stays, 40 diagnoses, 2136 events…cleaning and converting to time series…extracting separate episodes… 223285 279721 DONE! Subject 42367: reading…got 1 stays, 19 diagnoses, 38700 events…cleaning and converting to time series…extracting separate episodes… 250305 DONE! Subject 42412: reading…got 1 stays, 21 diagnoses, 1841 events…cleaning and converting to time series…extracting separate episodes… 241992 DONE! Subject 42430: reading…got 1 stays, 10 diagnoses, 2919 events…cleaning and converting to time series…extracting separate episodes… 210474 DONE! Subject 42458: reading…got 1 stays, 6 diagnoses, 897 events…cleaning and converting to time series…extracting separate episodes… 219307 DONE! Subject 43779: reading…got 1 stays, 11 diagnoses, 1004 events…cleaning and converting to time series…extracting separate episodes… 229194 DONE! Subject 43798: reading…got 1 stays, 15 diagnoses, 9854 events…cleaning and converting to time series…extracting separate episodes… 243229 DONE! Subject 43827: reading…got 1 stays, 14 diagnoses, 1889 events…cleaning and converting to time series…extracting separate episodes… 243238 DONE! Subject 43879: reading…got 1 stays, 8 diagnoses, 1246 events…cleaning and converting to time series…extracting separate episodes… 264258 DONE! Subject 43881: reading…got 2 stays, 34 diagnoses, 3360 events…cleaning and converting to time series…extracting separate episodes… 214180 243123 DONE! Subject 43909: reading…got 1 stays, 9 diagnoses, 232 events…cleaning and converting to time series…extracting separate episodes… 297782 DONE! Subject 43927: reading…got 1 stays, 11 diagnoses, 1865 events…cleaning and converting to time series…extracting separate episodes… 290513 DONE! Subject 44083: reading…got 3 stays, 14 diagnoses, 8196 events…cleaning and converting to time series…extracting separate episodes… 265615 282640 286428 DONE! Subject 44154: reading…got 1 stays, 8 diagnoses, 1246 events…cleaning and converting to time series…extracting separate episodes… 217724 DONE! Subject 44212: reading…got 1 stays, 23 diagnoses, 55533 events…cleaning and converting to time series…extracting separate episodes… 239396 DONE! Subject 44222: reading…got 1 stays, 11 diagnoses, 1130 events…cleaning and converting to time series…extracting separate episodes… 238186 DONE! Subject 44228: reading…got 1 stays, 11 diagnoses, 5257 events…cleaning and converting to time series…extracting separate episodes… 217992 DONE! (py27) bwimbp:mimic3-benchmarks-master bwi$ cd mimic3models (py27) bwimbp:mimic3models bwi$ cd phenotyping (py27) bwimbp:phenotyping bwi$ python -u main.py --network lstm_2layer --dim 512 --mode train --batch_size 8 --log_every 30 Namespace(batch_norm=True, batch_size=8, dim=512, dropout=0.0, epochs=100, l1=0, l2=0, load_state=‘’, log_every=30, mode=‘train’, network=‘lstm_2layer’, prefix=‘’, save_every=1, shuffle=True, small_part=False, timestep=‘0.8’) Traceback (most recent call last): File “main.py”, line 48, in <module> listfile=‘…/…/data/phenotyping/train_listfile.csv’) File “/Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master/mimic3benchmark/readers.py”, line 242, in init Reader.init(self, dataset_dir, listfile) File “/Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master/mimic3benchmark/readers.py”, line 13, in init with open(listfile_path, “r”) as lfile: IOError: [Errno 2] No such file or directory: ‘…/…/data/phenotyping/train_listfile.csv’ (py27) bwimbp:phenotyping bwi$ cd … (py27) bwimbp:mimic3models bwi$ cd mult* (py27) bwimbp:multitask bwi$ python -u main.py --network lstm --dim 1024 --mode train --batch_size 8 --log_every 30 --ihm_C 0.02 Namespace(batch_norm=True, batch_size=8, decomp_C=1.0, dim=1024, dropout=0.0, epochs=100, ihm_C=0.02, imputation=‘previous’, l1=0, l2=0, load_state=‘’, log_every=30, los_C=1.0, mode=‘train’, nbins=10, network=‘lstm’, partition=‘custom’, ph_C=1.0, prefix=‘’, save_every=1, shuffle=True, small_part=False, timestep=‘1.0’) Traceback (most recent call last): File “main.py”, line 58, in <module> listfile=‘…/…/data/multitask/train_listfile.csv’) File “/Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master/mimic3benchmark/readers.py”, line 304, in init Reader.init(self, dataset_dir, listfile) File “/Users/bwi/Documents/AD-LSTM-Benchmark/mimic3-benchmarks-master/mimic3benchmark/readers.py”, line 13, in init with open(listfile_path, “r”) as lfile: IOError: [Errno 2] No such file or directory: ‘…/…/data/multitask/train_listfile.csv’ (py27) bwimbp:multitask bwi$ `

Thank you for any pointers in the right direction!

My best, Michael

[Bowiemb]

@Harhro94 @Hrant-Khachatrian I will echo, @guiguiguiguigui. Thank you!

I am now in the middle of training the phenotype lstm on the full dataset that I was approved for the other day! I’m assuming this may take a while…

Btw: I would recommend adding to the ReadMe that you should download the development versions Theano and Lasagne. Those versions are the only ones that I could get to work, and it may save someone in the future some time.

See “Bleeding-Edge Installation”: http://deeplearning.net/software/theano/install_macos.html

Thanks again!

My best, Michael

[Hrant-Khachatrian]

Dear @Bowiemb and @guiguiguiguigui ,

For some reason one of the required steps disappeared from the readme. We are very sorry for this.

Please check the 7th step one more time. Few other instructions on working with baseline models are also recovered.

Thank you for your interest. Let us know if anything goes wrong. We’ll try to respond as soon as possible.