/usr/lib/rads/venv/lib/python3.13/site-packages/s3transfer/__pycache__
Edit: /usr/lib/rads/venv/lib/python3.13/site-packages/s3transfer/__pycache__/download.cpython-313.pyc (31441B)
j{ S SK r S SKrS SKrS SKJr S SKJr S SKJrJ r J
r
S SK J r S SK
JrJr S SKJrJrJrJrJrJrJrJrJr \R4 " \5 r " S S 5 r " S
S \ 5 r " S S
\ 5 r " S S\ 5 r " S S\ 5 r! " S S\5 r" " S S\5 r# " S S\#5 r$ " S S\5 r% " S S\5 r& " S S \5 r' " S S \5 r( " S S!\5 r) " S" S#5 r* " S$ S%5 r+g)& N) ClientErrorseekable)RetriesExceededErrorS3DownloadFailedErrorS3ValidationErrorIN_MEMORY_DOWNLOAD_TAG)SubmissionTaskTask) S3_RETRYABLE_DOWNLOAD_ERRORSCountCallbackInvokerDeferredOpenFileFunctionContainerStreamReaderProgresscalculate_num_partscalculate_range_parameter
get_callbacksinvoke_progress_callbacksc R \ rS rSrSrS r\S 5 rS rS r S r
S r S r S
r
S rg )
DownloadOutputManager* a Base manager class for handling various types of files for downloads
This class is typically used for the DownloadSubmissionTask class to help
determine the following:
* Provides the fileobj to write to downloads to
* Get a task to complete once everything downloaded has been written
The answers/implementations differ for the various types of file outputs
that may be accepted. All implementations must subclass and override
public methods from this class.
c ( Xl X l X0l g N)_osutil_transfer_coordinator _io_executor)selfosutiltransfer_coordinator io_executors F/usr/lib/rads/venv/lib/python3.13/site-packages/s3transfer/download.py__init__ DownloadOutputManager.__init__8 s %9" ' c [ S5 e)a5 Determines if the target for the download is compatible with manager
:param download_target: The target for which the upload will write
data to.
:param osutil: The os utility to be used for the transfer
:returns: True if the manager can handle the type of target specified
otherwise returns False.
z must implement is_compatible()NotImplementedErrorclsdownload_targetr s r"
is_compatible#DownloadOutputManager.is_compatible= s ""BCCr% c g)zGet the tag (if any) to associate all GetObjectTasks
:rtype: s3transfer.futures.TaskTag
:returns: The tag to associate all GetObjectTasks with
N r s r" get_download_task_tag+DownloadOutputManager.get_download_task_tagK s r% c [ S5 e)zGet file-like object to use for io writes in the io executor
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
returns: A file-like object to write to
z*must implement get_fileobj_for_io_writes()r' r transfer_futures r" get_fileobj_for_io_writes/DownloadOutputManager.get_fileobj_for_io_writesS s ""NOOr% c p U R R U R U R XU5 5 g)zQueue IO write for submission to the IO executor.
This method accepts an IO executor and information about the
downloaded data, and handles submitting this to the IO executor.
This method may defer submission to the IO executor if necessary.
N)r submitr get_io_write_taskr fileobjdataoffsets r" queue_file_io_task(DownloadOutputManager.queue_file_io_task] s0
"")) t 5 5gV L
r% c 2 [ U R UUUS.S9$ )a Get an IO write task for the requested set of data
This task can be ran immediately or be submitted to the IO executor
for it to run.
:type fileobj: file-like object
:param fileobj: The file-like object to write to
:type data: bytes
:param data: The data to write out
:type offset: integer
:param offset: The offset to write the data to in the file-like object
:returns: An IO task to be used to write data to a file-like object
)r< r= r> main_kwargs) IOWriteTaskr r; s r" r: 'DownloadOutputManager.get_io_write_taskj s( " & &"
r% c [ S5 e)aq Get the final io task to complete the download
This is needed because based on the architecture of the TransferManager
the final tasks will be sent to the IO executor, but the executor
needs a final task for it to signal that the transfer is done and
all done callbacks can be run.
:rtype: s3transfer.tasks.Task
:returns: A final task to completed in the io executor
z"must implement get_final_io_task()r' r0 s r" get_final_io_task'DownloadOutputManager.get_final_io_task s ""FGGr% c [ USU R R S9nU R R UR
5 U$ )Nwb)mode
open_function)r r openr add_failure_cleanupclose)r filenamefs r" _get_fileobj_from_filename0DownloadOutputManager._get_fileobj_from_filename s< 4t||/@/@
""66qww?r% )r r r N)__name__
__module__ __qualname____firstlineno____doc__r# classmethodr, r1 r6 r? r: rG rR __static_attributes__r/ r% r" r r * sB (
D DP
4 Hr% r c J ^ \ rS rSrU 4S jr\S 5 rS rS rS r Sr
U =r $ ) DownloadFilenameOutputManager c P > [ TU ] XU5 S U l S U l S U l g r )superr# _final_filename_temp_filename
_temp_fileobj)r r r r! __class__s r" r# &DownloadFilenameOutputManager.__init__ s*
{C # " !r% c " [ U[ 5 $ r )
isinstancestrr) s r" r, +DownloadFilenameOutputManager.is_compatible s /3//r% c UR R R nX l U R R U5 U l U R 5 U l U R $ r ) meta call_argsr< r` r get_temp_filenamera _get_temp_fileobjrb )r r5 r< s r" r6 7DownloadFilenameOutputManager.get_fileobj_for_io_writes sS !&&0088 & "ll <
[ TU ] XU5 Uc
[ 5 nX@l [ R
" 5 U l g r )r_ r#
DeferQueue _defer_queue threadingLock_io_submit_lockr r r r! defer_queuerc s r" r# )DownloadNonSeekableOutputManager.__init__ s6 {C $,K ' (~~ / r% c [ US5 $ )Nwrite)hasattrr) s r" r, .DownloadNonSeekableOutputManager.is_compatible s 00r% c [ $ r r r0 s r" r1 6DownloadNonSeekableOutputManager.get_download_task_tag s %%r% c B UR R R $ r r r4 s r" r6 :DownloadNonSeekableOutputManager.get_fileobj_for_io_writes s ##--555r% c ( [ U R S9$ r r r0 s r" rG 2DownloadNonSeekableOutputManager.get_final_io_task s '!%!;!;
r% c > U R U R R X25 nU H2 nUS n[ R SUS U5 [
TU ] XU5 M4 S S S 5 g ! , ( d f g = f)Nr= z%Queueing IO offset %s for fileobj: %sr> )r r request_writesloggerdebugr_ r? )r r< r= r> writesr rc s r" r? 3DownloadNonSeekableOutputManager.queue_file_io_task sk
!
!&&55fCF V} ;(O
*7&A "
!
!s AA++
A9c 0 [ U R UUS.S9$ )N)r< r= rB )IOStreamingWriteTaskr r; s r" r: 2DownloadNonSeekableOutputManager.get_io_write_task s# # & &"
r% )r r r )rT rU rV rW r# rY r, r1 r6 rG r? r: rZ rw rx s @r" r r s> EI0 1 1&6
B
r% r c J ^ \ rS rSr SU 4S jjr\S 5 rS rS rSr U =r
$ )$DownloadSpecialFilenameOutputManageri c 4 > [ TU ] XX45 S U l g r )r_ r# _fileobjr s r" r# -DownloadSpecialFilenameOutputManager.__init__ s +
r% c R [ U[ 5 =( a UR U5 $ r )rf rg is_special_filer) s r" r, 2DownloadSpecialFilenameOutputManager.is_compatible s& /3/
F4J4J 5
r% c UR R R nU R U5 U l U R $ r )rj rk r< rR r )r r5 rP s r" r6 >DownloadSpecialFilenameOutputManager.get_fileobj_for_io_writes s5 "''1199 77A
}} r% c D [ U R SSU R 0S9$ )NTr< )r rq rC ) IOCloseTaskr r r0 s r" rG 6DownloadSpecialFilenameOutputManager.get_final_io_task s' !%!;!;"DMM2
r% )r r ) rT rU rV rW r# rY r, r6 rG rZ rw rx s @r" r r s. EI
r% r c B \ rS rSrSrS r S S jrS rS rS r S r
S
r g) DownloadSubmissionTaski z/Task for submitting tasks to execute a downloadc [ [ [ [ /nUR R
R nU H nUR XB5 ( d M Us $ [ SU S[ U5 S35 e)a Retrieves a class for managing output for a download
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future for the request
:type osutil: s3transfer.utils.OSUtils
:param osutil: The os utility associated to the transfer
:rtype: class of DownloadOutputManager
:returns: The appropriate class to use for managing a specific type of
input for downloads.
zOutput z
of type: z is not supported.)
r r\ rz r rj rk r< r, RuntimeErrortype)r r5 r download_manager_resolver_chainr< download_manager_clss r" _get_download_output_manager_cls7DownloadSubmissionTask._get_download_output_manager_cls# sz
1 ) ) , +
' "&&0088$C #11'BB++ %D gYjg 7I J
r% Nc
UR R b UR R c UR " SUR R R
UR R R S.UR R R D6nUR R US 5 UR R UR S5 5 U R Xc5 " X0R U5 n UR R UR : a U R UUUUUU UU5 gU R UUUUUU UU5 g)a
:param client: The client associated with the transfer manager
:type config: s3transfer.manager.TransferConfig
:param config: The transfer config associated with the transfer
manager
:type osutil: s3transfer.utils.OSUtil
:param osutil: The os utility associated to the transfer manager
:type request_executor: s3transfer.futures.BoundedExecutor
:param request_executor: The request executor associated with the
transfer manager
:type io_executor: s3transfer.futures.BoundedExecutor
:param io_executor: The io executor associated with the
transfer manager
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future associated with the
transfer request that tasks are being submitted for
:type bandwidth_limiter: s3transfer.bandwidth.BandwidthLimiter
:param bandwidth_limiter: The bandwidth limiter to use when
downloading streams
NBucketKey
ContentLengthETagr/ )rj sizeetag head_objectrk bucketkey
extra_argsprovide_transfer_sizeprovide_object_etaggetr r multipart_threshold_submit_download_request _submit_ranged_download_request)
r clientconfigr request_executorr! r5 bandwidth_limiterresponsedownload_output_managers
r" _submit DownloadSubmissionTask._submit? sO J
% % - ##((0 )) &++55<<#((2266 "&&00;;H
6 6 )
4 4X\\&5I J"&"G"G #
,,k#; $ $v'A'A A ) ) ' !
0 0 ' !
r% c UR R n UR U5 n
[ US5 n UR 5 n UR 5 n
U R R U[ U R UU R U R U
U R U UR UUR US.
U
/S9U S9 g )Nprogress)
r r r r< r callbacks max_attemptsr io_chunksizer r rC done_callbackstag)rj rk r6 r r1 rG r r9 ImmediatelyWriteIOGetObjectTaskr r r num_download_attemptsr )r r r r r r! r r5 r rk r< progress_callbacksget_object_tag
final_tasks r" r /DownloadSubmissionTask._submit_download_request s $((22 *CC
+?J G 1FFH ->>@
"")) +%)%?%? $ ' . .$== &"+"6"6!3$*$@$@/F$*$7$7): !+|
% *
r% c UR R n [ US5 n
UR U5 n UR n [ UR R U 5 n
UR 5 n[ U R Xe5 5 n[ U
5 H n[ U UU
5 nSU0nUR R b UR R US' UR U R 5 UR 5 U R R# U[% U R UU R& U R( U UU
UR* UU - UUR, US. UR. /S9US9 M UR1 5 g )Nr RangeIfMatch) r r r r< r r r start_indexr r r r r )rj rk r r6 multipart_chunksizer r r1 r &_get_final_io_task_submission_callbackranger r updater incrementr r9
GetObjectTaskr r r r decrementfinalize)r r r r r r! r r5 r rk r r< part_size num_partsr finalize_download_invokerirange_parameterr s r" r 6DownloadSubmissionTask._submit_ranged_download_request s $((22 +?J G *CC
.. '(<(<(A(A9M 1FFH %9 7 7'
%
!
y!A 71i O J ##((4(7(<(<(A(A
9% i 2 2 3 % / / 1 & & - - )-)C)C"("+"2"2 (}}#*&0%7(.(D(D'(9}3J(.(;(;-> ! %>$G$G#H " #'
.
"H "**,r% c b UR 5 n[ U R R X#5 $ r )rG r r r9 )r download_managerr! r s r" r =DownloadSubmissionTask._get_final_io_task_submission_callback s0 &779
& & - -{
r% c B X!- nX#S-
:X a SnOXA- S-
nSU SU 3nU$ )N zbytes=-r/ )r r
part_indexr start_range end_range range_params r" _calculate_range_param-DownloadSubmissionTask._calculate_range_param s> , Q &I#/!3I {m1YK8 r% r/ r ) rT rU rV rW rX r r r r r r rZ r/ r% r" r r s. 9
H Qf0
dH-T
r% r c . \ rS rSr SS jrS rS rSrg)r i! Nc
Sn [ U5 H n
U
nUR " SX#S.UD6nU R UR S5 UR S5 5 [ US U5 nU ( a U R UU R 5 n[ UU 5 nU HG nU R R 5 ( d$ U R UUUU5 U[ U5 -
nMF g g [% U 5 e! [ aI nUR R S0 5 R S5 nUS:X a [ S U S
U S 35 ee SnAf[ a3 n[ R! S UU
US
S9 Un [# XjW-
5 SnAGMm SnAff = f)a^ Downloads an object and places content into io queue
:param client: The client to use when calling GetObject
:param bucket: The bucket to download from
:param key: The key to download from
:param fileobj: The file handle to write content to
:param exta_args: Any extra arguments to include in GetObject request
:param callbacks: List of progress callbacks to invoke on download
:param max_attempts: The number of retries to do when downloading
:param download_output_manager: The download output manager associated
with the current download.
:param io_chunksize: The size of each io chunk to read from the
download stream and queue in the io queue.
:param start_index: The location in the file to start writing the
content of the key to.
:param bandwidth_limiter: The bandwidth limiter to use when throttling
the downloading of data in streams.
Nr r ContentRangeBodyErrorCodePreconditionFailedzContents of stored object "z
" in bucket "z " did not match expected ETag.zCRetrying exception caught (%s), retrying request, (attempt %s / %s)T)exc_infor/ )r
get_object_validate_content_ranger r get_bandwith_limited_streamr DownloadChunkIteratordone
_handle_iolenr r r r
r r r r )r r r r r< r r r r r r r last_exceptionr
current_indexr streaming_bodychunkschunke
error_codes r" _mainGetObjectTask._main" s @ |$A<
+
!,, ! .8 ,, NN7+ LL 0 "6 V$i" %)EE *D,F,F # /~|L #E 55::<< 3 # ! ) &U3
$ G %| #>225
ZZ^^GR 8 < ^ \ rS rSrSr SU 4S jjrS rSrU =r$ )r i zA NOOP task to serve as an indicator that the download is complete
Note that the default for is_final is set to True because this should
always be the last task.
c ( > [ TU ] UUUUUS9 g )N)r rC pending_main_kwargsr rq )r_ r# )r r rC rA r rq rc s r" r# !CompleteDownloadNOOPTask.__init__ s& !5# 3)
r% c g r r/ r0 s r" r CompleteDownloadNOOPTask._main s r% r/ )NNNT) rT rU rV rW rX r# r rZ rw rx s @r" r r s%
r% r c * \ rS rSrS rS rS r\rSrg)r i c * Xl X l SU l g)zIterator to chunk out a downloaded S3 stream
:param body: A readable file-like object
:param chunksize: The amount to read each time
r N_body
_chunksize
_num_reads)r body chunksizes r" r# DownloadChunkIterator.__init__ s
#r% c U $ r r/ r0 s r" __iter__ DownloadChunkIterator.__iter__ s r% c U R R U R 5 nU =R S-
sl U( a U$ U R S:X a U$ [ 5 e)Nr )rH readrI rJ
StopIteration)r r s r" __next__ DownloadChunkIterator.__next__ sI
0 1 L
__
!
Lo r% rG N) rT rU rV rW r# rO rT nextrZ r/ r% r" r r s Dr% r c $ \ rS rSrSrS rS rSrg)r i a IO queue that defers write requests until they are queued sequentially.
This class is used to track IO data for a *single* fileobj.
You can send data to this queue, and it will defer any IO write requests
until it has the next contiguous block available (starting at 0).
c . / U l 0 U l SU l g )Nr )_writes_pending_offsets _next_offsetr0 s r" r# DeferQueue.__init__ s " r% c $ U[ U5 - U R :: a / $ / nXR : a U R U-
nX$S nU R nXR ; a8 U R U n[ U5 [ U5 :: a / $ X R U' O/[ R " U R
U5 X R U' U R
( a U R
S U R :X a [ R " U R
5 nU R U nUR XgS.5 U R U U =R [ U5 -
sl U R
( a U R
S U R :X a M U$ )a Request any available writes given new incoming data.
You call this method by providing new data along with the
offset associated with the data. If that new data unlocks
any contiguous writes that can now be submitted, this
method will return all applicable writes.
This is done with 1 method call so you don't have to
make two method calls (put(), get()) which acquires a lock
each method call.
Nr )r> r= )r r[ rZ heapqheappushrY heappopappend)r r> r= r
seen_bytes queued_datanext_write_offset
next_writes r" r DeferQueue.request_writes% sW CI !2!2 2
I %% % **V3J $D&&F ** * //7K4yC ,,
15%%f- NN4<< 0,0 ! !& )llt||A$2C2C C %
dll ; ../@AJ MM%6K L%%&78 Z 0 llt||A$2C2C C
r% )r[ rZ rY N)rT rU rV rW rX r# r rZ r/ r% r" r r s
:r% r ),r^ loggingr botocore.exceptionsr s3transfer.compatr s3transfer.exceptionsr r r s3transfer.futuresr
s3transfer.tasksr r s3transfer.utilsr
r r r r r r r r getLoggerrT r r r\ rz r r r r r rD r rr r r r r r/ r% r" ro s
+ &
6 1
8 $n nb%$9 %P
$9
",
'< ,
^
+K
:~^ ~BxD xv
m
$
4
9t 9 $
t
6