B ̵aKB @sfdZddlmZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZmZmZmZm Z dddddZ!d2dddddddddZ"GdddZ#Gddde#Z$Gd d!d!e#Z%eej&d"d#d3d&d'dd(d)dd*d+d,d-d.Z'eej&d"d#d4dddd/d0d1Z(dS)5z parquet compat ) annotationsN)AnyAnyStr)catch_warnings)FilePathOrBufferStorageOptions)import_optional_dependency)AbstractMethodError)doc) DataFrame MultiIndex get_option)generic)Version) IOHandles get_handle is_fsspec_urlis_urlstringify_pathstrBaseImpl)enginereturnc Cs|dkrtd}|dkrzttg}d}xF|D]>}y|Stk rf}z|dt|7}Wdd}~XYq*Xq*Wtd||dkrtS|dkrtStd dS) zreturn our implementationautozio.parquet.enginez - NzUnable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support. Trying to import the above resulted in these errors:pyarrow fastparquetz.engine must be one of 'pyarrow', 'fastparquet')r PyArrowImplFastParquetImpl ImportErrorr ValueError)rZengine_classesZ error_msgsZ engine_classerrr"0lib/python3.7/site-packages/pandas/io/parquet.py get_engine%s" & r$rbFrrrboolz.tuple[FilePathOrBuffer, IOHandles | None, Any])pathfsstorage_optionsmodeis_dirrcCst|}t|r:|dkr:td}|jj|f|p0i\}}n|rVt|rN|dkrVtdd}|s|st|trt j |st ||d|d}d}|j }|||fS)zFile handling for PyArrow.Nfsspecr%z8storage_options passed with buffer, or non-supported URLF)is_textr))rrrZcoreZ url_to_fsrr isinstancerosr'isdirrhandle)r'r(r)r*r+path_or_handler,handlesr"r"r#_get_path_or_handleGs"  r4c@s6eZdZeddddZddddZd dd ZdS) rr )dfcCsxt|tstdt|jtr>tdd|jjDsRtdn|jjdkrRtdtdd|jj D}|sttddS) Nz+to_parquet only supports IO with DataFramescss|]}|jdkVqdS)>emptystringN) inferred_type).0xr"r"r# wsz.BaseImpl.validate_dataframe..z parquet must have string column names for all values in each level of the MultiIndex >r6r7z%parquet must have string column namescss |]}|dk rt|tVqdS)N)r.r)r9namer"r"r#r;sz!Index level names must be strings) r.r r columnsr allZlevelsr8indexnames)r5Z valid_namesr"r"r#validate_dataframens   zBaseImpl.validate_dataframecKs t|dS)N)r )selfr5r' compressionkwargsr"r"r#writeszBaseImpl.writeNcKs t|dS)N)r )rBr'r=rDr"r"r#readsz BaseImpl.read)N)__name__ __module__ __qualname__ staticmethodrArErFr"r"r"r#rmsc@s>eZdZddZdddddd d d d d Zdd dddZdS)rcCs&tdddddl}ddl}||_dS)Nrz(pyarrow is required for parquet support.)extrar)rZpyarrow.parquetZpandas.core.arrays._arrow_utilsapi)rBrpandasr"r"r#__init__s  zPyArrowImpl.__init__snappyNr zFilePathOrBuffer[AnyStr]z str | Nonez bool | Nonerzlist[str] | None)r5r'rCr?r)partition_colsc Ks||d|ddi}|dk r*||d<|jjj|f|} t||dd|d|dk d\} } |d<zH|dk r|jjj| | f||d|n|jjj| | fd|i|Wd| dk r| XdS)NZschemaZpreserve_index filesystemwb)r)r*r+)rCrPrC) rApoprLZTableZ from_pandasr4parquetZwrite_to_datasetZ write_tableclose) rBr5r'rCr?r)rPrDZfrom_pandas_kwargstabler2r3r"r"r#rEs.   zPyArrowImpl.writeF)r)c KsDd|d<i}|rddl}|j||j||j||j| |j | |j | |j||j||j||j|i }|j|d<td} | dkrd|d<t||dd|d d \} } |d<z>|jjj| fd |i|jf|} | dkr(| jdd d } | S| dk r>| XdS)NTZuse_pandas_metadatarZ types_mapperzmode.data_managerZarrayZ split_blocksrQr%)r)r*r=F)copy)rMrLZint8Z Int8DtypeZint16Z Int16DtypeZint32Z Int32DtypeZint64Z Int64DtypeZuint8Z UInt8DtypeZuint16Z UInt16DtypeZuint32Z UInt32DtypeZuint64Z UInt64DtypeZbool_Z BooleanDtyper7Z StringDtypegetr r4rSrTZ read_table to_pandasZ _as_managerrU) rBr'r=use_nullable_dtypesr)rDZto_pandas_kwargsZpdmappingZmanagerr2r3resultr"r"r#rFs@    zPyArrowImpl.read)rONNN)NFN)rGrHrIrNrErFr"r"r"r#rs(rc@s6eZdZddZd ddddd Zddd d d ZdS)rcCstddd}||_dS)Nrz,fastparquet is required for parquet support.)rK)rrL)rBrr"r"r#rNs zFastParquetImpl.__init__rONr r)r5r)c s||d|kr$|dk r$tdnd|kr6|d}|dk rFd|d<t|}t|rrtdfdd|d<n r~td td d $|jj||f|||d |WdQRXdS) N partition_onzYCannot use both partition_on and partition_cols. Use partition_cols for partitioning dataZhiveZ file_schemer,csj|dfpiS)NrR)open)r'_)r,r)r"r##sz'FastParquetImpl.write.. open_withz?storage_options passed with file object or non-fsspec file pathT)record)rCZ write_indexr]) rAr rSrrrrrLrE)rBr5r'rCr?rPr)rDr")r,r)r#rEs.   zFastParquetImpl.write)r)c  si}|dd}t|jjtdkr,d|d<|r8tdt|}d}t|rtdt|jjtdkrj|dfpxij |d <qćfd d |d <n,t |t rt j |st|ddd }|j}|jj|f|}|jfd|i|} |dk r|| S)NrZFz0.7.1Z pandas_nullszNThe 'use_nullable_dtypes' argument is not supported for the fastparquet enginer,z0.6.1r%r(csj|dfpiS)Nr%)r^)r'r_)r,r)r"r#r`Lsz&FastParquetImpl.read..ra)r-r)r=)rSrrL __version__r rrrr^r(r.rr/r'r0rr1Z ParquetFilerYrU) rBr'r=r)rDZparquet_kwargsrZr3Z parquet_filer\r")r,r)r#rF5s0 zFastParquetImpl.read)rONNN)NN)rGrHrIrNrErFr"r"r"r#rs  ,rr))r)rrOr zFilePathOrBuffer | Nonez str | Nonez bool | Nonezlist[str] | Nonez bytes | None)r5r'rrCr?r)rPrc Ksrt|tr|g}t|}|dkr(tn|} |j|| f||||d||dkrjt| tjsbt| SdSdS)a Write a DataFrame to the parquet format. Parameters ---------- df : DataFrame path : str or file-like object, default None If a string, it will be used as Root Directory path when writing a partitioned dataset. By file-like object, we refer to objects with a write() method, such as a file handle (e.g. via builtin open function) or io.BytesIO. The engine fastparquet does not accept file-like objects. If path is None, a bytes object is returned. .. versionchanged:: 1.2.0 engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. compression : {{'snappy', 'gzip', 'brotli', None}}, default 'snappy' Name of the compression to use. Use ``None`` for no compression. index : bool, default None If ``True``, include the dataframe's index(es) in the file output. If ``False``, they will not be written to the file. If ``None``, similar to ``True`` the dataframe's index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn't require much space and is faster. Other indexes will be included as columns in the file output. partition_cols : str or list, optional, default None Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. {storage_options} .. versionadded:: 1.2.0 kwargs Additional keyword arguments passed to the engine Returns ------- bytes if no path argument is provided else None N)rCr?rPr))r.rr$ioBytesIOrEAssertionErrorgetvalue) r5r'rrCr?r)rPrDimplZ path_or_bufr"r"r# to_parquetas 9 ri)rr)rZcKs"t|}|j|f|||d|S)a Load a parquet object from the file path, returning a DataFrame. Parameters ---------- path : str, path object or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.parquet``. A file URL can also be a path to a directory that contains multiple partitioned parquet files. Both pyarrow and fastparquet support paths to directories as well as file URLs. A directory path could be: ``file://localhost/path/to/tables`` or ``s3://bucket/partition_dir`` If you want to pass in a path object, pandas accepts any ``os.PathLike``. By file-like object, we refer to objects with a ``read()`` method, such as a file handle (e.g. via builtin ``open`` function) or ``StringIO``. engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. columns : list, default=None If not None, only these columns will be read from the file. {storage_options} .. versionadded:: 1.3.0 use_nullable_dtypes : bool, default False If True, use dtypes that use ``pd.NA`` as missing value indicator for the resulting DataFrame. (only applicable for the ``pyarrow`` engine) As new dtypes are added that support ``pd.NA`` in the future, the output with this option will change to use those dtypes. Note: this is an experimental option, and behaviour (e.g. additional support dtypes) may change without notice. .. versionadded:: 1.2.0 **kwargs Any additional kwargs are passed to the engine. Returns ------- DataFrame )r=r)rZ)r$rF)r'rr=r)rZrDrhr"r"r# read_parquets<rj)Nr%F)NrrONNN)rNNF))__doc__Z __future__rrdr/typingrrwarningsrZpandas._typingrrZpandas.compat._optionalrZ pandas.errorsr Zpandas.util._decoratorsr rMr r r Z pandas.corerZpandas.util.versionrZpandas.io.commonrrrrrr$r4rrrZ _shared_docsrirjr"r"r"r#s@        %!$ig H