B 3RcM@sdZddlZddlZddlZddlmZddlmZddlmZm Z ddl m Z ddl m Z mZddlZeZdZd Zd d Zd'd d ZddZd(ddZd)ddZddZd*ddZddZedkreejdkredeje_eeeeed deeZ ee Z!e!d krxe"e!D]bZ#ej$%de#d e!feee e#d dZ&ej'rxej(rxeee&Z)ne&gZ)xe)D]Z*e*+qWej,reej-Z.e.d krej,Z,e)d/e,nPxNe"e.D]BZ#e0d ej-e#Z1e)e#Z2d!ej,3d"de1fZ,e2/e,qWej4r,ej5r,ed#ej5ej4feej-d krTed$x8e"ee)D](Z#e6e)e#e_2ej7ej5ej4d%qbWq,WnVe!d kreee dd dZ&ej'rej(reee&Z)ne&gZ)xe)D]Z*e*+qWej,rpeej-Z.e.d kr ej,Z,e)d/e,nPxNe"e.D]BZ#e0d ej-e#Z1e)e#Z2d!ej,3d"de1fZ,e2/e,q*Wej4rej5red#ej5ej4fee)d kred$xBe"ee)D](Z#e6e)e#e_2ej7ej5ej4d%qWned&dS)+a command line utility for growing composite models **Usage** _GrowComposite [optional args] filename_ **Command Line Arguments** - -n *count*: number of new models to build - -C *pickle file name*: name of file containing composite upon which to build. - --inNote *note*: note to be used in loading composite models from the database for growing - --balTable *table name*: table from which to take the original data set (for balancing) - --balWeight *weight*: (between 0 and 1) weighting factor for the new data (for balancing). OR, *weight* can be a list of weights - --balCnt *count*: number of individual models in the balanced composite (for balancing) - --balH: use only the holdout set from the original data set in the balancing (for balancing) - --balT: use only the training set from the original data set in the balancing (for balancing) - -S: shuffle the original data set (for balancing) - -r: randomize the activities of the original data set (for balancing) - -N *note*: note to be attached to the grown composite when it's saved in the database - --outNote *note*: equivalent to -N - -o *filename*: name of an output file to hold the pickled composite after it has been grown. If multiple balance weights are used, the weights will be added to the filenames. - -L *limit*: provide an (integer) limit on individual model complexity - -d *database name*: instead of reading the data from a QDAT file, pull it from a database. In this case, the _filename_ argument provides the name of the database table containing the data set. - -p *tablename*: store persistence data in the database in table *tablename* - -l: locks the random number generator to give consistent sets of training and hold-out data. This is primarily intended for testing purposes. - -g: be less greedy when training the models. - -G *number*: force trees to be rooted at descriptor *number*. - -D: show a detailed breakdown of the composite model performance across the training and, when appropriate, hold-out sets. - -t *threshold value*: use high-confidence predictions for the final analysis of the hold-out data. - -q *list string*: Add QuantTrees to the composite and use the list specified in *list string* as the number of target quantization bounds for each descriptor. Don't forget to include 0's at the beginning and end of *list string* for the name and value fields. For example, if there are 4 descriptors and you want 2 quant bounds apiece, you would use _-q "[0,2,2,2,2,0]"_. Two special cases: 1) If you would like to ignore a descriptor in the model building, use '-1' for its number of quant bounds. 2) If you have integer valued data that should not be quantized further, enter 0 for that descriptor. - -V: print the version number and exit N) DbConnect) CompositeRun)ScreenCompositeBuildComposite)AdjustComposite) DataUtils SplitDataz0.5.0cCstrtjd|dS)z emits messages to _sys.stdout_ override this in modules which import this one to redirect output **Arguments** - msg: the string to be displayed z%s N)_verbosesysstdoutwrite)msgr5lib/python3.7/site-packages/rdkit/ML/GrowComposite.pymessagets rc)Cst|_|dkr||j}|jdkr0|d|_|jdkrFt|}n6|j gkr`||_| }ntj |j||j |j |jd}|j}t||jdkrtj|d|dn|jdkrtj|d|d|}|} t| } td| td t| dd |} |j} ttd| d} |jrdd lm}m}|j gkrRdd lm}|j }ndd lm!}|j"}|j#}|j}|r|$|%|j&| | dg| |||j'|j(|j)d||j |j*|j+|t, dn0ddl-m}|j#}|j&| | dg| |j'|dd|.|/|0\}}}t12|}t12|}|%|_3x&tt|D]}||4|j3q|j9rt,rVtdt?@t|A||| d|jB}|\} }!}"}#}$}"}%t|}&| |!}'t<|!|'t=_>|#t=_C|$t=_DtE|%t=_F|'|&}(|(dkrt<|(|&t=_G|S)a does the actual work of building a composite model **Arguments** - details: a _CompositeRun.CompositeRun_ object containing details (options, parameters, etc.) about the run - composite: the composite model to grow - progressCallback: (optional) a function which is called with a single argument (the number of models built so far) after each model is built. - saveIt: (optional) if this is nonzero, the resulting model will be pickled and dumped to the filename specified in _details.outName_ - setDescNames: (optional) if nonzero, the composite's _SetInputOrder()_ method will be called using the results of the data set's _GetVarNames()_ method; it is assumed that the details object has a _descNames attribute which is passed to the composites _SetDescriptorNames()_ method. Otherwise (the default), _SetDescriptorNames()_ gets the results of _GetVarNames()_. - data: (optional) the data set to be used. If this is not provided, the data set described in details will be used. **Returns** the enlarged composite model Nz.pkl)Z quantNameuserZpasswordr )shuffle runDetailsrzTraining with %d examplesz %d descriptors) CrossValidate PruneTree)BuildQuantTree)ID3) buildDriverprunernTriespruneIt lessGreedyneedsQuantizationZ treeBuilderZ nQuantBoundsstartAtZmaxDepthprogressCallbacksilent)r)rrr z># Overall Average Error: %%% 5.2f, Average Deviation: %%% 6.2fgY@)verbosezTesting all examplesz)%d examples (%% %5.2f) were misclassifiedz Entire data set:)HtimeZasctimeZrundate tableNamestripoutNamedbNamerZBuildQuantDataSetqBounds GetDataSetZ DBToQuantDataZ qTableNameZdbUserZ dbPassword _randomSeedInitRandomNumbersshuffleActivitiesRandomizeActivitiesrandomActivities GetNamedDatalenrZGetNVars nPossibleValslistrangeuseTreesZrdkit.ML.DecTreerrrZ QuantTreeBootrZID3BootZCrossValidationDriverZ SetInputOrder GetVarNamesZGrownModelsrrr! limitDepthr Zrdkit.ML.NeuralZ AverageErrorsZ SortModelsZ GetAllDatanumpyZarrayZ _varNamesZ NameModelsumZsqrtZ bayesModelZTrain detailedResrZtestallfloat _runDetailsZ overall_errorrZShowVoteResultsZGetNPts thresholdZoverall_correct_confZoverall_incorrect_confreprZoverall_result_matrixZoverall_fraction_dropped))details compositer"ZsaveIt setDescNamesdatafNameZseed namedExamples trainExamplesZ nExamplesZnVarsr4ZattrsrrrZbuilderrZdriverrZ modelListZcountsZavgErrsiZ weightedErrsZ averageErrZdevsZavgDevZ badExamplesZwrongZresTupZnGoodZnBad_ZavgGoodZavgBadZvoteTabZnPtsZnClassZnRejrrrGrowIts                      (  rKcCsg}|jr^|jr^t|j|j}|jdd|jd}xH|D] }|d}|tt|q8Wn|j r||t t |j d|S)NZMODELzwhere note='%s')Zfieldswhererrb) persistTblNameinNoterr*ZGetDataappendpickleloadsstrcomposFileNameloadopen)rBresZconnZmdlsrowZrawDrrr GetCompositess  rYc s|jr|jt|kr|Std|dkrhtd|j}|j}||_|j}|j|_|}||_||_|dkrt|S|j |_ |j |_ t |j |jdkrt j|d|dn|jdkrt j|d|d||js|jrtjt|j dd\}}fdd |Dfd d |D} |jd krnt j|j|jd dd \}} fdd |D}| fdd | D7} ||jr| } n} dd |D} d}|dkrtd|}|dkr|S|j |_ |j |_ t |j |jdkrt j|d|dn|jdkrt j|d|d|} dd |D}d}g}|j}t|ttfs\|f}x:|D]2}td||t j!|| | ||j| |dqbW|S)a balances the composite using the parameters provided in details **Arguments** - details a _CompositeRun.RunDetails_ object - composite: the composite model to be balanced - data1: (optional) if provided, this should be the data set used to construct the original models - data2: (optional) if provided, this should be the data set used to construct the new individual models zBalancing CompositeNz Reading First Data Setr )rrr)r#csg|] }|qSrr).0x)rGrr Isz$BalanceComposite..csg|] }|qSrr)rZr[)rGrrr\Jsgr%)Z indicesOnlycsg|] }|qSrr)rZr[)rHrrr\Nscsg|] }|qSrr)rZr[)rHrrr\OscSsg|] }|qSr)upper)rZr[rrrr\Vsz Reading Second Data SetcSsg|] }|qSr)r])rZr[rrrr\isz Balancing with Weight: %.4f)Znames1Znames2)"balCntr3rbalTabler(r'r*balDbr,Z _splitFracZ splitFracr-Z randomSeedrr.r/r0r1r2 balDoHoldout balDoTrainrZ SplitIndicesZ filterFracZ FilterDataZ filterValr8 balWeight isinstancetupler5rPrBalanceComposite)rBrCZdata1Zdata2rFZtmpr*ZtrainIdxZtestIdxZ testExamplesZtempZdataSet1Zcols1ZdataSet2Zcols2rWZweightsweightr)rGrHrrfs             rfcCs,tdt|r(tdtdtjdS)z prints the version number z#This is GrowComposite.py version %szcommand line was: N)print__VERSION_STRINGjoinr argv) includeArgsrrr ShowVersionys rncCstttddS)zM provides a list of arguments for when this is used from the command line r%N)ri__doc__r exitrrrrUsagesrqcCs|dkr t}t|S)a initializes a details object with default values **Arguments** - details: (optional) a _CompositeRun.CompositeRun_ object. If this is not provided, the global _runDetails will be used. **Returns** the initialized _CompositeRun_ object. N)r?r SetDefaults)rrrrrrsrrc Csddl}|tjdddddddd d d d g\}}d |_d |_d |_d|_d|_d|_d|_ d |_ x|D]\}}|dkrt ||_ ql|dkr||_ql|dkr||_ql|dkrt ||_t|jttfs|jf|_ql|dkrt ||_ql|dkrd|_ql|dkrd|_ ql|dkr"||_ ql|dkr4||_ql|dksH|dkrP||_ql|dkrb||_ql|dkrt||_ql|dkrd|_ql|dkrd|_ql|dkrtql|dkrd|_ql|d krd|_ql|d!krt ||_ql|d"kr||_ql|d#krd|_ql|d$krt||_ql|d%kr0d|_ql|d&krFt ||_ql|d'kr~t |}t|ttfspt d(||_!||_"ql|d)krt |}t#|t#gt#d*gkst d+||_$||_%ql|d,krt&t'dqlt(d-|tj)d.tqlW|d|_*|j s|j|_ dS)/z parses command line arguments and updates _runDetails_ **Arguments** - runDetails: a _CompositeRun.CompositeRun_ object. rNr z)P:o:n:p:b:sf:F:v:hlgd:rSTt:Q:q:DVG:L:C:N:zinNote=zoutNote=z balTable=z balWeight=zbalCnt=ZbalHZbalTzbalDb=r)g?z-nz-Cz --balTablez --balWeightz--balCntz--balHz--balTz--balDbz--inNotez-Nz --outNotez-oz-pz-rz-Sz-hz-lz-gz-Gz-dz-Tz-tz-Dz-Lz-qz4bad argument type for -q, specify a list as a stringz-Qrz4bad argument type for -Q, specify a list as a stringz-Vz bad argument:)file)+getoptr rlrOrTr_rcr^rarbr`intr9evalrdrer5Znoter)rNr1r/rqZ lockRandomrr!r*r7r>r@r=r:AssertionErrorZ qBoundCountr+typeZactivityBoundsZactivityBoundsValsrnrpristderrr')rrtargsZextraargvalr+rrr ParseArgss                               r}__main__rrh)rmzT--------------------------------- Doing %d of %d --------------------------------- )rDdz %s.%d.pklz.pklzUpdating results table %s:%szDWARNING: updating results table with models having different weights)ZdbtablezNo models found)Nr rN)NN)r)N)8ror r&r;Zrdkit.Dbase.DbConnectionrZrdkit.MLrrrZrdkit.ML.CompositerZ rdkit.ML.DatarrrQr?rjr rrKrYrfrnrqrrr}__name__r3rlrkcmdZ initModelsr9r6rIryr rCr_r^Z compositesZmdlZClearModelExamplesr)rcZnWeightsZPicklerurgZmodelsplitrNr*dumpsZStorerrrr^s     ^  b       "