B §dÈŸã @sÚdZddlmZmZmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z dZ dZyddlmZddlmZWn4ek r°ddlmZddlmZdZ dZYnXedkrèe j ej ej e¡¡¡ddlmZmZnd dlmZmZe ¡d krddl Z ddl!Z!d d „Zd dddœZ"e ¡dkrNde ¡krNde"d<dZ#dZ$e$j%de#e"e ¡dZ&ej ej 'eddddd¡¡Z(ej 'dddde&¡Z)dd„Z*Gdd „d e+ƒZ,d!d"„Z-e.d#krÖe  /e-ƒ¡dS)$a™ Manage conda environment for CCTBX programs This is a minimal wrapper for conda functionality. It is meant to create the minimum environment for building CCTBX programs and is aimed towards developers that do not want to manage their own conda environments. Advanced users should use conda natively so that they have full control. This is called by bootstrap.py to set up a developer's conda environment. If no conda installation is found, one will be downloaded. It will be installed in the "mc3" directory at the same level as "modules." The minimum version for conda is 4.4. This is when conda moved towards building packages with a set of common compilers. é)Úabsolute_importÚdivisionÚprint_functionNFT)Úurljoin)Úurlopen)Ú check_outputÚcalléÚWindowsc Os„d|dksd|dkrlddg|}d}t ¡2}tj|f|žd|i|—Ž}| d¡| ¡}WdQRX|St|f|ž|Ž}|SdS)Nz conda.exerÚactivateÚcmdz/cÚÚstdout)ÚtempfileZ TemporaryFileÚ subprocessZ check_callÚseekÚreadr)Ú command_listÚargsÚkwargsÚoutputÚfÚ returncode©rú~/mnt/filia/a/genomebrowser/www/genomebrowser/fleming/tools/molprobity/modules/cctbx_project/libtbx/auto_build/install_conda.pyr3s   rzosx-64zlinux-64zwin-64)ÚDarwinÚLinuxr rZarm64z osx-arm64ÚPYTHON_VERSIONz${builder}_py{version}_{platform}.txtÚcctbx)ÚbuilderÚversionÚplatformz..Z cctbx_projectÚlibtbxÚ auto_buildÚ conda_envsc CsptreZdZdZej ddejde e e   ¡d¡Z eee e e eej ddejde e e   ¡d¡ej ddddd d œe   ¡¡eeeeej d dejd e e e   ¡d¡d œ Zeded <ed<dd dgZedddddejfdd„Zd)dd„Zd*dd„Zdd„Zd+dd„Zdd„Zd,dd „Zd-d"d#„Zd.d$d%„Zd/d'd(„ZdS)0Ú conda_managerau Class for managing conda environments Attributes ---------- env_locations: dict Dictionary for determining which environment to use for each builder. The keys are the builder names and must match those in bootstrap.py The values are the relative path from "modules" to the file. Builders for programs outside of cctbx_project should place their environment files inside their repository. For example, the Phenix environment file should be in the phenix repository. Methods ------- __init__(root_dir, conda_base, conda_env, verbose, log) Constructor that does the basic check for a conda installation get_conda_exe(prefix, check_file) Returns the platform-dependent conda executable get_conda_python(conda_env, check_file) Returns the platform-dependent conda python update_environments() Returns a list of paths that are conda environments based on the environments.txt file in ${HOME}/.conda install_miniconda(prefix) Downloads and installs the latest version of miniconda3 update_conda() Updates the current version of conda create_environment(builder, filename, copy) Uses the known conda installtion to create an environment write_conda_setpaths(prefix, build_dir, conda_env, check_file) Writes an additional script that activates the conda environment before calling the normal setpaths script. Úphenixr$)rr r!Údialsz .conda-envsz macos.txtz linux.txtz windows.txt)rrr Z phasertng) Z cctbxliterr/Zphenix_voyagerZphenix_releaseZ xfellegacyz dials-oldr0ZexternalZ molprobityZqrefineZphaserZvoyagerÚxfelÚlabelitNTéFcsöt ¡ˆ_|ˆ_dˆ_|dk r,tj |¡ˆ_dˆ_|dk rHtj |¡ˆ_|ˆ_|ˆ_ |ˆ_ |ˆ_ dˆ_ ˆjdk r‚ˆj ˆjddˆ_ tj tj d¡dd¡ˆ_ˆ ¡ˆ_tj ¡ˆ_dˆjd<d ˆ_ˆjdkrTtj d ¡}|dk rTˆjd krú| d ¡}n | d ¡}xN|D]F} tj tj | d¡¡}ˆj |dd} tj | ¡r |ˆ_| ˆ_ Pq Wˆjdkrœx:ˆjD]0} ˆj | dd} tj | ¡rh| ˆ_| ˆ_ PqhW‡fdd„} ˆjdkrìˆjdk rì| ˆjˆj ƒ\}} tj | ¡rì|ˆ_| ˆ_ ˆjdkrvˆjdkrvtj ˆjd¡} tj | ¡r8tdˆj d| ˆ_n,tdˆj dtdˆj dˆjˆjdˆ_ˆj ˆjddˆ_ ˆ ¡ˆ_ˆjdk rÎtj ˆj ¡s¼| ˆjˆj ƒ\ˆ_ˆ_ ˆ ¡ˆ_tj ˆj ¡sÖtˆjƒ‚t  t!ˆj ddgˆjd¡}ˆj|dkg}x"ˆjD]}| "||dk¡q Wd|kr@d}t# $|t%¡|d d¡}t&|dƒ}t&|d ƒ}|d!ks„|d!krŒ|d!krŒtd"ƒ‚td#j'ˆjd$ˆj dˆj rÎt!ˆj dgˆjd}t|ˆj dˆjdk ròtd%j'ˆjd&ˆj ddS)'a‘ Constructor that performs a basic check for the conda installation. If an installation is not found, the latest version can be downloaded and installed. Parameters ---------- root_dir: str Required argument for specifying the root level directory for CCTBX builds. This is where the "modules" and "build" directories reside. If a new conda installation is required, it will be placed here under the "mc3" directory. The conda environment for building will also be placed in this directory conda_base: str Required argument for specifying location of a conda installation. If this is None, miniconda will be installed unless conda_env is set to a valid path. conda_env: str Optional argument for specifying location of a conda environment. Since this is assumed to be a working conda environment, a new installation of conda will not be created. This is for developers that want to manage their own environments. check_file: bool Flag for checking if a file exists. A RuntimeError is raised if a this flag is set and a file does not exist. Used in get_conda_exe and get_conda_python. max_retries: int When downloading conda packages, there may be network issues that prevent the environment from being constructed. This parameter controls the number of retry attempts for constructing the conda environment. The first retry is attempted 1 minute after the initial failure. The second retry is attempted 2 minutes, etc. verbose: bool Flag for showing conda output log: file For storing log output NF)Ú check_fileú~z.condazenvironments.txtr Z PYTHONPATHzØ The conda executable cannot be found. Please make sure the correct directory for the base conda installation was provided. The directory can be found by running "conda info" and looking the "base environment" value.ÚPATHr ú;ú:z..cs„|dkr d}|dkrd}xftj |¡s~tj tj |d¡¡}ˆj|dd}tj |¡r^||fS|tj tj |d¡¡krdSqWdS)Nr z..F)r4)r r )ÚosÚpathÚisfileÚabspathÚjoinÚ get_conda_exe)Znew_conda_baseZ new_conda_exe)ÚselfrrÚ walk_up_check%s z-conda_manager.__init__..walk_up_checkÚmc3z Using default conda installation)Úfilez+Location of conda installation not providedz$Proceeding with a fresh installation)ÚprefixÚinfoz--json)ÚenvZ root_prefixÚenvszø There is a mismatch between the conda settings in your home directory and what "conda info" is reporting. This is not a fatal error, but if an error is encountered, please check that your conda installation and environments exist and are working. Z conda_versionÚ.rr éz£ CCTBX programs require conda version 4.4 and greater to make use of the common compilers provided by conda. Please update your version with "conda update conda". z!Base conda installation: {base})Úbasez Build environment: {conda_env})Ú conda_env)(r!ÚsystemÚroot_dirÚ conda_baser9r:ÚnormpathrJr4Ú max_retriesÚverboseÚlogÚ conda_exer>r=Ú expanduserÚenvironment_fileÚupdate_environmentsÚ environmentsÚenvironÚcopyrEÚconda_exe_not_foundÚgetÚsplitr<r;ÚisdirÚprintÚinstall_minicondaÚ RuntimeErrorÚjsonÚloadsrÚappendÚwarningsÚwarnÚRuntimeWarningÚintÚformat)r?rLrMrJr4rOrPrQÚpathsr:rRZ environmentr@Ú install_dirZ conda_infoZconsistency_checkrEÚmessageZ split_versionZ major_versionZ minor_versionrr)r?rÚ__init__¾sª'                           zconda_manager.__init__cCsf|dkr|j}|dkr|j}|jdkr8tj |dd¡}ntj |dd¡}|rbtj |¡sbt|jƒ‚|S)a3 Find the conda executable. This is platform-dependent Parameters ---------- prefix: str The path to the base conda environment check_file: bool Used to override the check_file attribute Returns ------- conda_exe: str The path to the conda executable Nr ÚScriptsz conda.exeÚbinÚconda) rMr4rKr9r:r=r;r_rY)r?rCr4rRrrrr>zs   zconda_manager.get_conda_execCs˜|dkr|j}|dkr|j}|dkr*|j}|jdkrDtj |d¡}n0|jdkrdtj |dddd¡}ntj |d d¡}|r”tj |¡s”td j |d ƒ‚|S) a, Find the conda python. This is platform-dependent Parameters ---------- conda_env: str The path to the conda environment check_file: bool Used to override the check_file attribute Returns ------- conda_python: str The path to the conda python Nr z python.exerz python.appZContentsZMacOSÚpythonrmz‹ Python could not be located in {conda_env}. Please make sure {conda_env} is a valid conda environment and that Python is installed. )rJ) rJrMr4rKr9r:r=r;r_rg)r?rJr4Z conda_pythonrrrÚget_conda_pythonœs"   zconda_manager.get_conda_pythonc Csþtƒ}yVt|jƒ}| ¡}WdQRXx2|D]*}| ¡}tj |¡r,| tj  |¡¡q,WWnt k rpYnX|j dk rútj  |j d¡tj  tj  d¡dd¡g}xT|D]L}tj |¡rªt |¡}x0|D](}tj  ||¡}tj |¡rÊ| |¡qÊWqªW|S)z¢ Read and check for existence of environment directories Returns ------- environments: list List of paths that exist on the filesystem NrFr5z.conda)Úsetr'rTÚ readlinesÚstripr9r:r\ÚaddrNÚIOErrorrMr=rSÚlistdir) r?rVrrhrEZenv_dirsZenv_dirÚdirsZ_dirrrrrUÆs*          z!conda_manager.update_environmentsc Csr|dkr|j}ddddœ}dj||jd}t|jdkrLd j||jd}|jdkr`|d 7}n|d 7}d }t||ƒ}tj ||¡}tj |¡sÈt d j|d|j dt ||ƒt dj|d|j dnt dj|d|j dtj |d¡}|jdkrdj|d}d|d|g}ndddj|dg}d|g|}t dj|d|j dt ||j d} |jrnt | |j d|S)aT Download a miniconda installer and install. The default installation location is at the same directory level as the "modules" directory. Parameters ---------- prefix: str The installation directory for miniconda Returns ------- conda_base: str The location of the "base" conda environment NZMacOSXrr )rrr z#Miniconda3-latest-{platform}-x86_64)r!z osx-arm64z"Miniconda3-latest-{platform}-arm64z.exez.shz$https://repo.anaconda.com/miniconda/zDownloading {url})r+)rBzDownloaded file to {filename})r,zUsing local copy at {filename}rAzK/InstallationType=JustMe /RegisterPython=0 /AddToPath=0 /S /D={install_dir})riú"z-bz-pz"{install_dir}"z/bin/shz'Installing miniconda to "{install_dir}")rE)rLrgrKÚconda_platformrr9r:r=r;r]rQr-rrErP) r?rCZos_namesr,Zurl_baser+riÚflagsrrrrrr^èsH           zconda_manager.install_minicondacCsR|jdddddddg}yt||jd}Wntk rDtd ƒYn Xt|ƒd S) z¿ Update the version of conda, if possible. The defaults channel is used because that is the default for a normal miniconda installation. Parameters ---------- None Úupdatez-nrIz-cÚdefaultsz-yrn)rEa ******************************************************************************* There was a failure in updating your base conda installaion. To update manually, try running conda update -n base -c defaults conda If you are using conda from a different channel, replace "defaults" with that channel ******************************************************************************* N)rRrrEÚ Exceptionr])r?rrrrrÚ update_conda*s  zconda_manager.update_condac CsÊt}|js|rt}xlt|jƒD]^}|d7}y|||jd}Wn<tk rxtdj||jt   ¡dƒt   |d¡YqXPqW||jkr’t dƒ‚tdj||d|j d | ¡|_||jkrÆt d ƒ‚d S) a­ Internal convenience function for retrying creation/update of a conda environment. Parameters ---------- command_list: list Command list for check_output text: str Text to be printed. Usually "installion into" or "update of" prefix: str Directory of conda environment verbose: bool Used to override self.verbose Returns ------- Nothing r )rEa$ ******************************************************************************* There was a failure in constructing the conda environment. Attempt {retry} of {max_retries} will start {retry} minute(s) from {t}. ******************************************************************************* )ÚretryrOÚté<z‰ The conda environment could not be constructed. Please check that there is a working network connection for downloading conda packages. zCompleted {text}: {prefix})ÚtextrC)rBzU The newly installed environment cannot be found in ${HOME}/.conda/environments.txt. N)rrPrÚrangerOrEr}r]rgÚtimeZasctimeZsleepr_rQrUrV)r?rr‚rCrPZ run_commandrrrrrÚ_retry_commandGs*     zconda_manager._retry_commandrcCsò||jkr,tdj|d t|j ¡ƒ¡dƒ‚|jdkr>tdƒ‚|dkrR|dkrRd}|dkr˜tj |j d |j|¡}|dk r¤|d krŠtd ƒ‚|  d |¡}n tj  |¡}tj  |¡sÂtd j||dƒ‚d}|  d¡sÚ|  d¡rÞd}|jdkrþd}tj |j |¡}ntj  |j¡}d} d} tj |¡r,t |¡} tj |¡rDt |¡} | dk rx| dk rx| j| jkrxtd|jddS||jkrœd} |r’d} ddg} n d} ddg} |j| d|d|g} |rÌ|  dd ¡|jd!krôtj |jd"d#¡d$d%g| } |r |s |  d&¡|r |s |  d'¡|d(kr:|s:|  d)¡||jkrŒtd*d+„|pVd,Dƒƒ}d-|d.|d|d.|ddf}|  |¡td/j| d.||d0|jd|j| | d|dd1|jd!krî|  d¡rîtd2tj |d3¡d4dS)5a; Create the environment based on the builder and file. The environment name is "conda_base". Parameters ---------- builder: str The builder from bootstrap.py. The default environment is defined by the env_locations class variable filename: str If filename is not None, the argument overrides the file defined in the env_locations dictionary. The filename should be a relative path to the "modules" directory. python: str If set, the specific Python version of the environment for the builder is used instead of the default. Current options are '27' and '36' for Python 2.7 and 3.6, respectively. copy: bool If set to True, the --copy flag is passed to conda offline: bool If set to True, the --offline flag is passed to conda zS The builder, {builder}, is not recognized. The available builders are, {builders} z, )rZbuildersNz&A conda installation is not available.r0)Ú27Ú36z dials-oldÚmodules)r†Ú37Ú38Ú39Ú310zAOnly Python 2.7, 3.7, 3.8, 3.9, and 3.10 are currently supported.rzŸThe file, {filename}, is not available. Please contact the developers to make sure that the requested version of Python is supported for the {builder} builder.)r,rFZymlZyamlTrMzDThe environment is newer than the environment file. Skipping update.)rBZinstallr{ÚUpdatingz update ofÚcreateÚ Installingzinstallation intoz--prefixz--filer rEr rlr rIz&&z--copyz --offline)r0z dials-oldr1r2z-ycss|]}t|ƒVqdS)N)rf)Ú.0Úirrrú ész3conda_manager.create_environment..r‡z#"conda-forge::python>=%s.%s,<%s.%s"rz/{text} {builder} environment with: {filename})r‚rr,)rPzchttps://download.microsoft.com/download/5/D/8/5D8C65CB-C849-4025-8E95-C3966CAFD8AE/vcredist_x64.exezvcredist_x64.exe)r+r,)Ú env_locationsr_rgr=ÚsortedÚkeysrMr9r:rLÚreplacer<r;ÚendswithrJÚexistsÚstatÚst_mtimer]rQrVrRÚinsertrKrbÚenv_without_pythonÚtupler…r-)r?rr,rorXÚofflineZ yaml_formatÚnamerCZ file_statsZ env_statsÚcommandÚ text_messagesrZpython_versionZpython_requirementrrrÚcreate_environments”                      z conda_manager.create_environmentc Csâg}|r| d¡|r*| d¡| d¡tj |jd¡}d}ddg}||jkr\d}d d g}|j|d d d d|g|}td|jdtdj |dd|jdx|D]}td||jdq¢W|j ||d|ddtd|jd|S)a* Create a separate environment for development tools. Packages from the conda-forge channel are used. Package list: svn git git-lfs Parameters ---------- svn: bool If set to true, svn is installed. This is useful for Xcode 11.4 and later where svn is no longer available. git: bool If set to true, git is installed with git-lfs. This is needed for repositories that use git-lfs. Returns ------- dev_dir: str The directory with the environment ÚsvnÚgitzgit-lfsZdev_envrŽrzinstallation intor{rz update ofz-yz-cz conda-forgez--prefixzO-------------------------------------------------------------------------------)rBz0{text} extra development environment containing:r)r‚z -r T)rP) rbr9r:r=rLrVrRr]rQrgr…) r?r£r¤Z package_listrCr r¡rÚpackagerrrÚcreate_dev_environments,        z$conda_manager.create_dev_environmentrnc Cs>|dkr|j}|dkr t d¡}|dkr0tdƒ‚d}|dksHtj |¡sPt|ƒ‚tjdkrztj tj  |d¡¡s®t|ƒ‚n4tj tj  |d¡¡r¦tj tj  |d¡¡s®t|ƒ‚|dkr¼|j }|dkrÖd d l m }|ƒ}tj  |¡}d d „}tjdkrôd }tj  tj  |jdd¡¡} tj  tj  |d¡¡} tj  tj  ||d¡¡} t| dƒ} |  |j| || d¡WdQRX|r~|| ƒd} d}tj  tj  |d¡¡} tj  tj  ||d¡¡} t| dƒ} |  |j| || d¡WdQRX|r:|| ƒnFxBdD]8} d}tj  tj  |jddd| ¡¡} | dkr>d}d}d}n d }d!}d"}tj  tj  |d#| ¡¡} tj  tj  ||d$| ¡¡} t| dƒ"} |  |j| ||||| d%¡WdQRX|rÂ|| ƒd&}tj  tj  |d'| ¡¡} tj  tj  ||d(| ¡¡} t| dƒ} |  |j| d)¡WdQRX|rü|| ƒqüWdS)*aI Write a script similar to setpaths.sh/csh/bat that activates the environment first. This is useful for developers that want to use other software installed in the conda environment without having to manually activate it. Parameters ---------- prefix: str The prefix for the output file. The output file will be constructed as _