Your IP : 216.73.217.13
U
�Ad�o � @ s. d dl Z d dlZd dlZd dlZd dlZd dlmZmZmZ d dl m
Z
d dl m Z d dl
mZ d dlmZ d dlmZ d dlmZmZmZmZmZmZmZ d d lm Z m Z m Z d dl Z d d
l m Z d d l!m"Z# d d l!m$Z$ e#�%e&�Z'd
Z(dd� Z)dd� Z*ed d�dd�Z+G dd� d�Z,G dd� de,�Z-G dd� d�Z.G d d � d e/�Z0d d � Z1d:e2e.d$�d%d&�Z3d;ed'ef e4e5ej6ee7 ed(�d)d*�Z8d<ed'ef ee4 e7e5ed-�d.d/�Z9dde'j:ddd ddd#d!d+f eee e5ee eeee5ge5f e4e2e7d0�d1d2�Z;G d3d4� d4�Z<d=d5d6�Z=efe j>fd7�d8d9�Z?dS )>� N)�ThreadPoolExecutor� TimeoutError� as_completed)� parsedate)�ENOENT)�partial)� NOT_FOUND)�count)�Any�Callable�Iterator�List�Optional�Tuple�Union)�quote�urlparse�
urlunparse)�
exceptions)�log)�version�REDACTEDc C s< t t| dd��}|d s4|d r4|d |d< d|d< t|�S )NZhttp)�scheme� � � )�listr r )�urlZ
parsed_url� r �6/usr/lib/python3/dist-packages/cloudinit/url_helper.py� _cleanurl$ s
r c G s$ dd� }| }|D ]}|||�}q|S )Nc S sN t t| ��}|d }|r*|�d�s*|d7 }|tt|�dd�7 }||d< t|�S )Nr �/z/:)Zsafe)r r �endswithr �strr )r �add_onZ
url_parsed�pathr r r �combine_single/ s z#combine_url.<locals>.combine_singler )�baseZadd_onsr& r r$ r r r � combine_url. s
r( )� FileResponse� UrlResponse��returnc
K s� | � � } | �d�rd| } | �� �d�r�|�d�r>t�d| � | td�d� }z"t|d��}|�� }W 5 Q R X W nH t k
r� } z*|j
}|j
t kr�t }t
||d| d�|�W 5 d}~X Y nX t||d �S t| f|�S dS )
a0 Wrapper function around readurl to allow passing a file path as url.
When url is not a local file path, passthrough any kwargs to readurl.
In the case of parameter passthrough to readurl, default values for some
parameters. See: call-signature of readurl in this module for param docs.
r! z file://%szfile://�dataz'Unable to post data to file resource %sN�rb)�cause�code�headersr )�contents)�lstrip�
startswith�lower�get�LOG�warning�len�open�read�IOError�errnor r �UrlErrorr) �readurl)r �kwargsZ file_path�fpr2 �er0 r r r �read_file_or_url> s"
$ rC c @ s& e Zd Zd dd�Zdd� Zdd� ZdS )
�StringResponse�� c C s || _ i | _|| _d | _d S �N)r0 r1 r2 r )�selfr2 r0 r r r �__init__^ s zStringResponse.__init__c O s | j dkrdS dS )NrE FT�r0 �rG �argsr@ r r r �okd s
zStringResponse.okc C s | j �d�S )Nzutf-8)r2 �decode�rG r r r �__str__i s zStringResponse.__str__N)rE )�__name__�
__module__� __qualname__rH rL rO r r r r rD ] s
rD c @ s e Zd Zddd�ZdS )r) rE c C s t j| ||d� || _d S )NrI )rD rH r )rG r% r2 r0 r r r rH n s zFileResponse.__init__N)rE �rP rQ rR rH r r r r r) m s r) c @ s� e Zd Zejd�dd�Zeed�dd��Zedd� �Z dd
d �Z
ed d
� �Z edd� �Z dd� Z
dee eee d�dd�ZdS )r* )�responsec C s
|| _ d S rF )� _response)rG rT r r r rH t s zUrlResponse.__init__r+ c C s | j jd krdS | j jS )N� )rU �contentrN r r r r2 w s zUrlResponse.contentsc C s | j jS rF )rU r rN r r r r } s zUrlResponse.urlFc C s2 d}|r d}d| j kr"|k r*n ndS dS d S )Ni, i� rE TFrI )rG Z redirects_ok�upperr r r rL � s zUrlResponse.okc C s | j jS rF )rU r1 rN r r r r1 � s zUrlResponse.headersc C s | j jS rF )rU � status_coderN r r r r0 � s zUrlResponse.codec C s | j jS rF )rU �textrN r r r rO � s zUrlResponse.__str__r )�
chunk_size�decode_unicoder, c c s | j �||�E dH dS )at Iterates over the response data.
When stream=True is set on the request, this avoids reading the content
at once into memory for large responses.
:param chunk_size: Number of bytes it should read into memory.
:param decode_unicode: If True, content will be decoded using the best
available encoding based on the response.
N)rU � iter_content)rG r[ r\ r r r r] � s zUrlResponse.iter_contentN)F)r F)rP rQ rR �requestsZResponserH �property�bytesr2 r rL r1 r0 rO r �int�boolr r] r r r r r* s s$
� �r* c @ s e Zd Zddd�ZdS )r> Nc C s< t �| t|�� || _|| _|| _| jd kr2i | _|| _d S rF )r< rH r# r/ r0 r1 r )rG r/ r0 r1 r r r r rH � s
zUrlError.__init__)NNNrS r r r r r> � s r> c C s� i }t | �j}|dkr~|r~d|kr8|d r8|d |d< nd|d< d|krfd|krf|d |d g|d< nd|kr~t|d �|d< |S )NZhttpsZca_certsZverifyTZ cert_fileZkey_fileZcert)r r r# )r � ssl_detailsZssl_argsr r r r �
_get_ssl_args� s
�
rd r TFr )�streamr, c s* t | �} | |d�}|�t| |�� |
|d< |s:|r6dnd}||d< |dk rpt|t�r^||d< ntt|�d�|d< |dkr|g }d }|r�tt|�d d �}d
d t� � i}� r�|�� � |� |ṡ fd d
�}|}|r�||d< |dkr�d}g }|
r�t
� nt d|�D �] }|| �|d< i }|� � D ]p\}�|dk�r0�q|dk�r�|�r��fdd�|D �}|�r�t
���||< |D ]}t|| |< �qjn�||< �qz�|�r�t�d||
�r�dn|| |� | dk�r�t�� } | �}|jf |�} W 5 Q R X | �r�| �� t�d| | jt| j�|d � t| �W S tjk
�r } z�t| tj ��rtt | d��rtt | j d��rt|� t | | j j| j j!| d�� n,|� t | | d�� t| tj"��r�W Y �t �q | �r�| |�
� |d ��s�W Y �N �q |
�r�|dk�s�|d |k �r
|dk�r
|�r t�d|� t#�$|� W 5 d} ~ X Y q�X q�|d �dS )a� Wrapper around requests.Session to read the url and retry if necessary
:param url: Mandatory url to request.
:param data: Optional form data to post the URL. Will set request_method
to 'POST' if present.
:param timeout: Timeout in seconds to wait for a response. May be a tuple
if specifying (connection timeout, read timeout).
:param retries: Number of times to retry on exception if exception_cb is
None or exception_cb returns True for the exception caught. Default is
to fail with 0 retries on exception.
:param sec_between: Default 1: amount of seconds passed to time.sleep
between retries. None or -1 means don't sleep.
:param headers: Optional dict of headers to send during request
:param headers_cb: Optional callable returning a dict of values to send as
headers during request
:param headers_redact: Optional list of header names to redact from the log
:param ssl_details: Optional dict providing key_file, ca_certs, and
cert_file keys for use on in ssl connections.
:param check_status: Optional boolean set True to raise when HTTPError
occurs. Default: True.
:param allow_redirects: Optional boolean passed straight to Session.request
as 'allow_redirects'. Default: True.
:param exception_cb: Optional callable which accepts the params
msg and exception and returns a boolean True if retries are permitted.
:param session: Optional exiting requests.Session instance to reuse.
:param infinite: Bool, set True to retry indefinitely. Default: False.
:param log_req_resp: Set False to turn off verbose debug messages.
:param request_method: String passed as 'method' to Session.request.
Typically GET, or POST. Default: POST if data is provided, GET
otherwise.
:param stream: if False, the response content will be immediately
downloaded.
)r re �allow_redirectsZPOSTZGET�methodN�timeoutr r z
User-Agentz
Cloud-Init/%sc s � S rF r �r )r1 r r �_cb s zreadurl.<locals>._cbr- ���r1 c s g | ]}� � |�r|�qS r )r6 )�.0�k)�vr r �
<listcomp>' s
zreadurl.<locals>.<listcomp>z'[%s/%s] open '%s' with %s configuration�infinitez(Read from %s (%s, %sb) after %s attemptsrT rY �r0 r1 r ri �1Please wait %s seconds while we wait to try again)%r �updaterd �
isinstance�tuple�max�floatra r Zversion_stringr �range�items�copy�deepcopyr r7 �debugr^ ZSessionZrequestZraise_for_statusrY r9 rW r* r ZRequestExceptionZ HTTPError�hasattrrT �appendr> r1 ZSSLError�time�sleep) r r- rh ZretriesZ sec_betweenr1 �
headers_cb�headers_redactrc � check_statusrf � exception_cbZsessionrp Z log_req_resp�request_methodre Zreq_argsZ manual_triesZ def_headersrj Zexcps�iZfiltered_req_argsrm Zmatched_headers�keyZsess�rrB r )r1 rn r r? � s� 4�
�
�
�
�� ����
��� r? .)�func�addrrh �event�delayr, c C s |r|j |d�rdS | ||�S )z Execute func with optional delay�rh N��wait�r� r� rh r� r� r r r �_run_func_with_delayu s r� �333333�?�
)r� � addresses�
stagger_delayrh r, c
s d}d}d}g }t �� �tt|�d�� z�z�� ����fdd�t|�D �}t|�d�D ]N} || }| �� }
|
r||
}|�|� qT| � � }|rT��
� ||f W W �hS qT|r�t � d||� |�nt �
d |� td
��W n0 tk
�r t � d d �|�d �|�� Y nX W 5 � jdd� X ||fS )
a execute multiple callbacks in parallel
Run blocking func against two different addresses staggered with a
delay. The first call to return successfully is returned from this
function and remaining unfinished calls are cancelled if they have not
yet started
N)Z max_workersFr� c
s, i | ]$\}}� j t�|��|� d �|�qS )r� )Zsubmitr� )rl r� r� �Zexecutorr� Zis_doner� rh r r �
<dictcomp>� s ��z dual_stack.<locals>.<dictcomp>r� z<Exception(s) %s during request to %s, raising last exceptionzEmpty result for address %szNo result returnedzJTimed out waiting for addresses: %s, exception(s) raised while waiting: %s� )� threading�Eventr r9 Zshutdown� enumerater � exceptionr~ �result�setr7 r8 �error�
ValueErrorr �join) r� r� r� rh Z
return_resultZreturned_addressZlast_exceptionr ZfuturesZfutureZreturn_exceptionr r� r �
dual_stack� sJ
�
� �r� )� status_cbr� �
sleep_timer� �
sleep_time_cbr� �connect_synchronously� async_delayc s t t d�dd�} dd� � dd� ���fdd �����fd
d ������� �
fd d
�}
� ���
fdd�}t�� }|
rv|
n|}|p�| }d�d}||��}|||||�}|r�|\}}||jfS � �|�r�q��d �t�d|� t�|� t |� t�� �p�d}q�t�d�
� dS )a�
urls: a list of urls to try
max_wait: roughly the maximum time to wait before giving up
The max time is *actually* len(urls)*timeout as each url will
be tried once and given the timeout provided.
a number <= 0 will always result in only one try
timeout: the timeout provided to urlopen
status_cb: call method with string message when a url is not available
headers_cb: call method with single argument of url to get headers
for request.
headers_redact: a list of header names to redact from the log
exception_cb: call method with 2 arguments 'msg' (per status_cb) and
'exception', the exception that occurred.
sleep_time_cb: call method with 2 arguments (response, loop_n) that
generates the next sleep time.
request_method: indicate the type of HTTP request, GET, PUT, or POST
connect_synchronously: if false, enables executing requests in parallel
async_delay: delay before parallel metadata requests, see RFC 6555
returns: tuple of (url, response contents), on failure, (False, None)
the idea of this routine is to wait for the EC2 metadata service to
come up. On both Eucalyptus and EC2 we have seen the case where
the instance hit the MD before the MD service was up. EC2 seems
to have permanently fixed this, though.
In openstack, the metadata service might be painfully slow, and
unable to avoid hitting a timeout of even up to 10 seconds or more
(LP: #894279) for a simple GET.
Offset those needs with the need to not hang forever (and block boot)
on a system where cloud-init is configured to look for EC2 Metadata
service but is not going to find one. It is possible that the instance
data host (169.254.169.254) may be firewalled off Entirely for a system,
meaning that the connection will block forever unless a timeout is set.
A value of None for max_wait will retry indefinitely.
)� loop_numberr, c S s t |d �d S )N� r )ra )�_r� r r r �default_sleep_time s z(wait_for_url.<locals>.default_sleep_timec S s$ | dkr dS | dkp"t � � | | kS )z4Check if time is up based on start time and max waitNFr )r )�max_wait�
start_timer r r �timeup s z wait_for_url.<locals>.timeupc S sf | j s*d| j }tt|�| j| j|d�}n4| �� sVd| j }tt|�| j| j|d�}nd}d}||fS )z?Map requests response code/contents to internal "UrlError" typezempty response [%s]rq zbad status code [%s]r N)r2 r0 r> r� r1 rL )rT r �reason�url_excr r r �handle_url_response s&
�
�z)wait_for_url.<locals>.handle_url_responsec
s� d}d}z,| |�\}}� ||�\}}|s2||fW S W nZ t k
rb } zd| }| }W 5 d} ~ X Y n. tk
r� } zd| }| }W 5 d} ~ X Y nX tt�� | �}
�r�d� nd} d||
| |f } || � |r�|| |d� dS ) z:Execute request, handle response, optionally log exceptionr Nzrequest error [%s]zunexpected error [%s]z%ssZ unlimitedz Calling '%s' failed [%s/%s]: %s)�msgr� )r> � Exceptionra r )
Z
url_reader_cb�urlsr� �exc_cb�log_cbr� r rT r� rB Z
time_takenZ max_wait_strZ
status_msg)r� r� r r �read_url_handle_exceptions. s0 �z0wait_for_url.<locals>.read_url_handle_exceptionsc s$ t | � d kri n� | ��|d�d�S )NF)r1 r� rh r� r� )r? )r rh )r� r� r� r r � read_url_cbN s �z!wait_for_url.<locals>.read_url_cbc s� �� fdd�}�D ]l}t � � }�dkrb��| �r6 dS �dk rb� rb|� | � krbt| � | �� �||| ||�}|r| S qdS )z|iterate over list of urls, request each one and handle responses
and thrown exceptions individually per url
c s | � | ��fS rF r ri )r� rh r r �url_reader_serial] s z@wait_for_url.<locals>.read_url_serial.<locals>.url_reader_serialr N)r ra )r� rh r� r� r� r Znow�out)�loop_nr� r� r� r� r� r� r �read_url_serialX s,
��� �z%wait_for_url.<locals>.read_url_serialc s, t t�� |d�}�|�| ||�}|r(|S dS )z�pass list of urls to dual_stack which sends requests in parallel
handle response and exceptions of the first endpoint to respond
)r� rh N)r r� )r� rh r� r� Zurl_reader_parallelr� )r� r� r� r� r r �read_url_parallels s � �z'wait_for_url.<locals>.read_url_parallelr Nr rr z$Timed out, no response from urls: %s)FN)ra r r2 r7 r| r� r� )r� r� rh r� r� r� r� r� r� r� r� r� r� r� r� r� Z do_read_urlZcalculate_sleep_timerT r Zaddressr ) r� r� r� r� r� r� r� r� r� r� r� r � wait_for_url� s: 4
�
�
r� c @ s^ e Zd Zddd�Zdd� Zdd� Zd d
� Zd d � Zd
d� Zdd� Z dd� Z
dd� Z dd� Z dS )�OauthUrlHelperN�/run/oauth_skew.jsonc C st || _ |p d| _|| _|| _|| _d| _d| _| j| j| j f}t|�sNd| _nt|�s^t d��| �
� }|pli | _ d S )Nr Tr� FzBall or none of token_key, token_secret, or consumer_key can be set) � consumer_key�consumer_secret� token_key� token_secret�skew_data_file� _do_oauth�skew_change_limit�any�allr� �read_skew_file� skew_data)rG r� r� r� r� r� Zrequired�oldr r r rH � s
�zOauthUrlHelper.__init__c
C sD | j r@tj�| j �r@t| j dd��}t�|�W 5 Q R � S Q R X d S )Nr� ��mode)r� �osr% �isfiler: �json�load)rG rA r r r r� � s z OauthUrlHelper.read_skew_filec C sT | j s
d S | �� }|d kr i }|||< t| j dd��}|�t�|�� W 5 Q R X d S )N�wr� )r� r� r: �writer� �dumps)rG �host�valueZcurrA r r r �update_skew_file� s z OauthUrlHelper.update_skew_filec
C s� t |t�r |jdks"|jdks"d S d|jkr>t�d|j� d S |jd }zt�t|��}W n6 t k
r� } zt�d||� W Y �d S d }~X Y nX t
|t�� �}t |j �j
}| j�|d�}t|| �| jkr�| �||� t�d||� || j|< d S )Ni� i� �datez$Missing header 'date' in %s responsez#Failed to convert datetime '%s': %sr z$Setting oauth clockskew for %s to %d)rt r> r0 r1 r7 r8 r �mktimer r� ra r r �netlocr� r6 �absr� r� ) rG r� r� r� Z remote_timerB Zskewr� Zold_skewr r r r� � s0 ���
zOauthUrlHelper.exception_cbc C sZ | j s
i S d }t|�j}| jr>|| jkr>tt�� �| j| }t|| j| j| j | j
|d�S )N)r r� r� r� r� � timestamp) r� r r� r� ra r �
oauth_headersr� r� r� r� )rG r r� r� r r r r� � s
�zOauthUrlHelper.headers_cbc C s6 t | j|�d��|d< t | j|�d��|d< |||�S )Nr� r� )r � _headers_cbr6 �
_exception_cb)rG Z wrapped_funcrK r@ r r r �_wrapped s � �zOauthUrlHelper._wrappedc O s | � t||�S rF )r� r� rJ r r r r� s zOauthUrlHelper.wait_for_urlc O s | � t||�S rF )r� r? rJ r r r r? s zOauthUrlHelper.readurlc
C s* d }z|r|||�}W 5 | � ||� X |S rF )r� )rG Zextra_exception_cbr� r� Zretr r r r� s z OauthUrlHelper._exception_cbc C s$ i }|r||�}|� | �|�� |S rF )rs r� )rG Zextra_headers_cbr r1 r r r r� s
zOauthUrlHelper._headers_cb)NNNNr� )
rP rQ rR rH r� r� r� r� r� r� r? r� r� r r r r r� � s �
r� c
C s| zdd l m} W n, tk
r< } ztd�|�W 5 d }~X Y nX |rLt|�}nd }|j|||||j|d�}|�| �\} }
} |
S )Nr z oauth support is not available)Z
client_secretZresource_owner_keyZresource_owner_secretZsignature_methodr� )Zoauthlib.oauth1�oauth1� ImportError�NotImplementedErrorr# ZClientZSIGNATURE_PLAINTEXTZsign) r r� r� r� r� r� r� rB ZclientZ_uriZsigned_headersZ_bodyr r r r� & s"
�r� )� retry_codes�retry_instancesc C s6 t |t�sdS |j|kr dS |jr2t |j|�r2dS dS )a Configurable retry exception callback for readurl().
:param retry_codes: Codes to retry on. Defaults to 404.
:param retry_instances: Exception types to retry on. Defaults to
requests.Timeout.
:returns: False to raise the exception from readurl(), True to retry.
FT)rt r> r0 r/ )r� �excr� r� r r r �retry_on_url_exc? s
r� )NNr r NNNNTTNNFTr F)N)r� r� )N)@rz r� r� r� r Zconcurrent.futuresr r r Z email.utilsr r= r � functoolsr Z http.clientr � itertoolsr �typingr
r r r
r r r Z urllib.parser r r r^ r Z cloudinitr Zloggingr Z getLoggerrP r7 r r r( rC rD r) r* r< r> rd rb r? r# ra r� rw r� r� r| r� r� r� ZTimeoutr� r r r r �<module> s� $
1 �� : �
� �
� U�� R| �
�