o vhH+@sddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZdd lmZdd lmZdd lmZerPdd lmZdd lmZGdddeZdS)) annotations) TYPE_CHECKINGN)lib)import_optional_dependency) ParserError ParserWarning)find_stack_level) pandas_dtype) is_integer)arrow_table_to_pandas) ParserBase) ReadBuffer) DataFramecsTeZdZdZdfdd Zddd Zdd d ZdddZdddZdddZ Z S)ArrowParserWrapperz7 Wrapper for the pyarrow engine for read_csv() srcReadBuffer[bytes]returnNonec s$t|||_||_|dSN)super__init__kwdsr _parse_kwds)selfrr __class__d/var/www/html/hyperkenya/venv/lib/python3.10/site-packages/pandas/io/parsers/arrow_parser_wrapper.pyrs  zArrowParserWrapper.__init__cCsN|jd}|dur dn||_|jd}t|trtdt|jd|_dS)z? Validates keywords before passing to pyarrow. encodingNzutf-8 na_valuesz?The pyarrow engine doesn't support passing a dict for na_values)rgetr isinstancedict ValueErrorlistr)rrrrrrr&s   zArrowParserWrapper._parse_kwdscCsddddddd}|D]\}}||jvr'|j|dur'|j||j|<q |j}t|tr4|g}nd}||jd <d d |jD|_|jd }|durt|rZ||jd <n*|t j j krfd|jd <n|t j j krwddd}||jd <n |t j j krdd|jd <dd |jD|_d|jdv|jd<|jdurd|jvrdd|jdD|jd<|jdu|jdur|jn|jd|jd|_dS)z: Rename some arguments to pass to pyarrow include_columns null_values escape_charignore_empty_lines decimal_point quote_char)usecolsr escapecharskip_blank_linesdecimal quotecharNtimestamp_parserscS&i|]\}}|dur|dvr||qS)N) delimiterr*r'r(r.0 option_name option_valuerrr Sz;ArrowParserWrapper._get_pyarrow_options.. on_bad_linesinvalid_row_handlerrstrcSs.tjd|jd|jd|jttddS)Nz Expected z columns, but found z: ) stacklevelskip)warningswarnexpected_columnsactual_columnstextrr) invalid_rowrrrhandle_warninges z?ArrowParserWrapper._get_pyarrow_options..handle_warningcSsdS)Nr=r)_rrrpsz9ArrowParserWrapper._get_pyarrow_options..cSr1)N)r%r& true_values false_valuesr)r0rr3rrrr7rr8strings_can_be_nullcSsg|]}d|qS)fr)r4nrrr s z;ArrowParserWrapper._get_pyarrow_options..skiprows)autogenerate_column_names skip_rowsr)rr;)itemsrr pop date_formatr!r; parse_optionscallabler BadLineHandleMethodERRORWARNSKIPconvert_optionsheaderr read_options)rmapping pandas_name pyarrow_namerSr9rDrrr_get_pyarrow_options4s^           z'ArrowParserWrapper._get_pyarrow_optionsframerc stj}d}|jdur<|jdur|jdurt||_t|j|kr8ddt|t|jD}||j|_d}|j_|j\}|jdur|j}t|jD]S\}}t |rdj|||<n |jvrqt d|d|j dur|j |dur||j |fn j||j j|f\} } | dur|  | | <|j | =qTj|ddd|jdur|sdgtjjj_|j durt|j trۇfd d |j D|_ nt|j |_ z |j WSty} zt | d} ~ wwS) z Processes data read in based on kwargs. Parameters ---------- frame: DataFrame The DataFrame to process. Returns ------- DataFrame The processed DataFrame. TNcSsg|]}t|qSr)r;r4xrrrrMsz>ArrowParserWrapper._finalize_pandas_output..FzIndex z invalid)dropinplacecs$i|]\}}|jvr|t|qSr)columnsr )r4kvrarrr7s  z>ArrowParserWrapper._finalize_pandas_output..)lenrfr[namesrange_do_date_conversions index_colcopy enumerater r#dtyper astype set_indexindexr!r"rQr TypeError) rranum_colsmulti_index_namedcolumns_prefixrE index_to_setiitemkey new_dtypeerrir_finalize_pandas_outputsZ              z*ArrowParserWrapper._finalize_pandas_outputcCs8t|rtdd|Dstdt|rtddS)Ncs|]}t|tVqdSrr!r;rbrrr sz7ArrowParserWrapper._validate_usecols..zwThe pyarrow engine does not allow 'usecols' to be integer column positions. Pass a list of string column names instead.z=The pyarrow engine does not allow 'usecols' to be a callable.)r is_list_likeallr#rU)rr+rrr_validate_usecolssz$ArrowParserWrapper._validate_usecolsc Cstd}td}|z |jdi|j}Wn2tyI|jdd}|dur.|||jdt}t |rDt dd|DsHtdwz|j |j |j di|j|jdi|j|d }Wn|jyv}zt||d}~ww|jd }|tjur|j} |} t|jjD]\} } |j| r| | | | | } q|| }ttd d t t!||d d} Wdn1swY|"| S)z Reads the contents of a CSV file into a DataFrame and processes it according to the kwargs passed in the constructor. Returns ------- DataFrame The DataFrame created from the CSV file. pyarrowz pyarrow.csvr%Nr&csrrrrbrrrrs  z*ArrowParserWrapper.read..z9The 'pyarrow' engine requires all na_values to be strings)r\rTrZ dtype_backendignorezmake_block is deprecatedT)r null_to_int64r)#rr`ConvertOptionsrZrur rsetrrrread_csvr ReadOptionsr\ ParseOptionsrT ArrowInvalidrr no_defaultschemafloat64rptypesis_nullfield with_typecastr>catch_warningsfilterwarningsDeprecationWarningr r)rpa pyarrow_csvrZincludenullstabler~r new_schemanew_typerz arrow_typerarrrreadsf           zArrowParserWrapper.read)rrrr)rr)rarrr)rr) __name__ __module__ __qualname____doc__rrr`rrr __classcell__rrrrrs   [ J r) __future__rtypingrr> pandas._libsrpandas.compat._optionalr pandas.errorsrrpandas.util._exceptionsrpandas.core.dtypes.commonr pandas.core.dtypes.inferencer pandas.io._utilr pandas.io.parsers.base_parserr pandas._typingr pandasrrrrrrs