This directory includes datasets used in ICML 2014 Deep Supervised and Convolutional Generative Stochastic Network paper. "As described in the paper two datasets are used. Both are based on protein structures from cullpdb servers. The difference is that the first one is divided to training/validation/test set, while the second one is filtered to remove redundancy with CB513 dataset (for the purpose of testing performance on CB513 dataset)." cullpdb+profile_6133.npy.gz is the one with training/validation/test set division; cullpdb+profile_6133_filtered.npy.gz is the one after filtering for redundancy with cb513. this is used for evaluation on cb513. "cb513+profile_split1.npy.gz is the CB513 features I used. Note that one of the sequences in CB513 is longer than 700 amino acids, and it is splited to two overlapping sequences and these are the last two samples (i.e. there are 514 rows instead of 513)." It is currently in numpy format as a (N protein x k features) matrix. You can reshape it to (N protein x 700 amino acids x 57 features) first. The 57 features are: "[0,22): amino acid residues, with the order of 'A', 'C', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'M', 'L', 'N', 'Q', 'P', 'S', 'R', 'T', 'W', 'V', 'Y', 'X','NoSeq'" "[22,31): Secondary structure labels, with the sequence of 'L', 'B', 'E', 'G', 'I', 'H', 'S', 'T','NoSeq'" "[31,33): N- and C- terminals;" "[33,35): relative and absolute solvent accessibility, used only for training. (absolute accessibility is thresholded at 15; relative accessibility is normalized by the largest accessibility value in a protein and thresholded at 0.15; original solvent accessibility is computed by DSSP)" "[35,57): sequence profile. Note the order of amino acid residues is ACDEFGHIKLMNPQRSTVWXY and it is different from the order for amino acid residues" The last feature of both amino acid residues and secondary structure labels just mark end of the protein sequence. "[22,31) and [33,35) are hidden during testing." "The dataset division for the first ""cullpdb+profile_6133.npy.gz"" dataset is" "[0,5600) training" "[5605,5877) test " "[5877,6133) validation" " For the filtered dataset ""cullpdb+profile_6133_filtered.npy.gz"", all proteins can be used for training and test on CB513 dataset." import numpy >>> d=numpy.load("cullpdb+profile_6133_filtered.npy.gz") >>> d.ndim 2 >>> d[:,:].shape (5534, 39900) (/ 39900 700.0) 57.0 d[1,0:56] array([ 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 1. , 1. , 0.03455623, 0.00364771, 0.05215356, 0.2404891 , 0.00343602, 0.01551975, 0.08166025, 0.00463257, 0.99330717, 0.00769087, 0.03167623, 0.67699581, 0.02231344, 0.99712843, 0.95968956, 0.07175755, 0.04269665, 0.00776757, 0.00630555, 0.26894143, 0.01567329]) >>> d[1,(1*57):(1*57+56)] array([ 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 1. , 0.91606164, 0.00275934, 0.09975048, 0.99423403, 0.00150118, 0.00792324, 0.01679065, 0.00230584, 0.85569692, 0.00273196, 0.00618146, 0.04352164, 0.92890567, 0.25730947, 0.85320967, 0.15577585, 0.08393843, 0.00532502, 0.00164232, 0.11920292, 0.00411088]) for i in range(700): if d[1,(i*57+32)]==1: print i "[0,22): amino acid residues, with the order of 'A', 'C', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'M', 'L', 'N', 'Q', 'P', 'S', 'R', 'T', 'W', 'V', 'Y', 'X','NoSeq'" "[22,31): Secondary structure labels, with the sequence of 'L', 'B', 'E', 'G', 'I', 'H', 'S', 'T','NoSeq'" file = open("cullpdb+profile_6133_filtered.txt","w") for p in range(5534): w=1 for i in range(700): s='' if w==1: for j in range(33): x='0' if d[p,(i*57)+j]>0: x='1' s+=x s+='\n' file.write(s) if d[p,(i*57+32)]==1: w=0 file.close() reczko@max:~/bak/doc/sci/biology/protein/structure-prediction/ss/princeton-data-jzthree$ tar tf cullpdb.tar.gz pdbs/ pdbs/12as.pdb pdbs/16vp.pdb # convert to clstm format reczko@max:~/bak/doc/sci/biology/protein/structure-prediction/ss/princeton-data-jzthree$ awk -f cullpdb2letters1.awk cullpdb+profile_6133_filtered.txt > cullpdb-all.txt wc cullpdb-all.txt 5534 11068 2377704 cullpdb-all.txt reczko@max:~/bak/doc/sci/biology/protein/structure-prediction/ss/princeton-data-jzthree$ head -4500 cullpdb-all.txt > cullpdb-tra.txt https://rostlab.org/papers/2008_rev_assignment2/paper.html#VORONO_TESSELLATION_FOR_GEOMETRICAL_RESIDUE_PARTITIONING Converting secondary structure states to three classes Several secondary structure assignment methods are presently available, but DSSP continues to be the most widely used method followed by STRIDE. In fact, most prediction methods are based on DSSP assignments. Typically, the 8 DSSP states are converted into three classes using the following convention: [GHI] -> h, [EB] -> e, [TS' '] -> c, which reads: 310-, a, p-helices are grouped into one helix class; Extended b-sheets and b-bridges are grouped into one sheet class; and the remaining secondary structure states turn, bend and Ňnot assignedÓ are grouped into one coil class. Usually, 310-helices and b-bridges constitute short secondary structure segments that have some structural similarity to a-helix and b-strand, respectively. However, they do have different sequence characteristics. Prediction methods, in general, are more precise in the core of regular secondary structure segments than at the termini (Rost, 1994; Cuff, 1999). Thus, 310-helices and b-bridges are more difficult to predict than a-helices and b-strands. Therefore an alternative conversion that has been used more recently yields a seemingly higher level of prediction accuracy: [H] -> h, [E] -> e, [GIBTS' '] -> c. [EB]->e: 2201 5534 0.397723 of all ss have odd #of E only E 1368 5534 0.247199 of all ss have odd #of E