swiftly.client

Contains tools for connecting to Swift services.

For convenience, the following names are imported from submodules:

StandardClient	swiftly.client.standardclient.StandardClient
DirectClient	swiftly.client.directclient.DirectClient
ClientManager	swiftly.client.manager.ClientManager
generate_temp_url	swiftly.client.utils.generate_temp_url()
get_trans_id_time	swiftly.client.utils.get_trans_id_time()

Copyright 2011-2013 Gregory Holt Portions Copyright (c) 2010-2012 OpenStack Foundation

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

client

Contains the base Client class for accessing Swift services.

class swiftly.client.client.Client

The base class for accessing Swift services.

For concrete examples, see swiftly.client.standardclient.StandardClient and swiftly.client.directclient.DirectClient.

To form a new concrete subclass, you would need to implement request() and get_account_hash() minimally and optionally reset() and auth().

auth()

Just performs any authentication steps without making an actual request to the Swift system.

delete_account(headers=None, yes_i_mean_delete_the_account=False, query=None, cdn=False, body=None)

Sends a DELETE request to the account and returns the results.

With query['bulk-delete'] = '' this might mean a bulk delete request where the body of the request is new-line separated, url-encoded list of names to delete. Be careful with this! One wrong move and you might mark your account for deletion of you have the access to do so!

For a plain DELETE to the account, on clusters that support it and, assuming you have permissions to do so, the account will be marked as deleted and immediately begin removing the objects from the cluster in the backgound.

THERE IS NO GOING BACK!

Parameters:

• headers – Additional headers to send with the request.
• yes_i_mean_delete_the_account – Set to True to verify you really mean to delete the entire account. This is required unless body and 'bulk-delete' in query.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – Some account DELETE requests, like the bulk delete request, take a body.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

delete_container(container, headers=None, query=None, cdn=False, body=None)

DELETEs the container and returns the results.

Parameters:

• container – The name of the container.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – Some container DELETE requests might take a body in the future.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

delete_object(container, obj, headers=None, query=None, cdn=False, body=None)

DELETEs the object and returns the results.

Parameters:

• container – The name of the container.
• obj – The name of the object.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – Some object DELETE requests might take a body in the future.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

get_account(headers=None, prefix=None, delimiter=None, marker=None, end_marker=None, limit=None, query=None, cdn=False, decode_json=True)

GETs the account and returns the results. This is done to list the containers for the account. Some useful headers are also returned:

x-account-bytes-used	Object storage used for the account, in bytes.
x-account-container-count	The number of containers in the account.
x-account-object-count	The number of objects in the account.

Also, any user headers beginning with x-account-meta- are returned.

These values can be delayed depending the Swift cluster.

Parameters:

• headers – Additional headers to send with the request.
• prefix – The prefix container names must match to be listed.
• delimiter –

  The delimiter for the listing. Delimiters indicate how far to progress through container names before “rolling them up”. For instance, a delimiter=’.’ query on an account with the containers:

  one.one
  one.two
  two
  three.one
  

  would return the JSON value of:

  [{'subdir': 'one.'},
   {'count': 0, 'bytes': 0, 'name': 'two'},
   {'subdir': 'three.'}]
  

  Using this with prefix can allow you to traverse a psuedo hierarchy.

• marker – Only container names after this marker will be returned. Swift returns a limited number of containers per request (often 10,000). To get the next batch of names, you issue another query with the marker set to the last name you received. You can continue to issue requests until you receive no more names.
• end_marker – Only container names before this marker will be returned.
• limit – Limits the size of the list returned per request. The default and maximum depends on the Swift cluster (usually 10,000).
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• decode_json – If set False, the usual decoding of the JSON response will be skipped and the raw contents will be returned instead.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the decoded JSON response or the raw str for the HTTP body.

get_account_hash()

Returns the account identifier for the Swift account being accessed.

get_container(container, headers=None, prefix=None, delimiter=None, marker=None, end_marker=None, limit=None, query=None, cdn=False, decode_json=True)

GETs the container and returns the results. This is done to list the objects for the container. Some useful headers are also returned:

x-container-bytes-used	Object storage used for the container, in bytes.
x-container-object-count	The number of objects in the container.

Also, any user headers beginning with x-container-meta- are returned.

These values can be delayed depending the Swift cluster.

Parameters:

• container – The name of the container.
• headers – Additional headers to send with the request.
• prefix – The prefix object names must match to be listed.
• delimiter –

  The delimiter for the listing. Delimiters indicate how far to progress through object names before “rolling them up”. For instance, a delimiter=’/’ query on an container with the objects:

  one/one
  one/two
  two
  three/one
  

  would return the JSON value of:

  [{'subdir': 'one/'},
   {'count': 0, 'bytes': 0, 'name': 'two'},
   {'subdir': 'three/'}]
  

  Using this with prefix can allow you to traverse a psuedo hierarchy.

• marker – Only object names after this marker will be returned. Swift returns a limited number of objects per request (often 10,000). To get the next batch of names, you issue another query with the marker set to the last name you received. You can continue to issue requests until you receive no more names.
• end_marker – Only object names before this marker will be returned.
• limit – Limits the size of the list returned per request. The default and maximum depends on the Swift cluster (usually 10,000).
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• decode_json – If set False, the usual decoding of the JSON response will be skipped and the raw contents will be returned instead.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the decoded JSON response or the raw str for the HTTP body.

get_object(container, obj, headers=None, stream=True, query=None, cdn=False)

GETs the object and returns the results.

Parameters:

• container – The name of the container.
• obj – The name of the object.
• headers – Additional headers to send with the request.
• stream – Indicates whether to stream the contents or preread them fully and return them as a str. Default: True to stream the contents. When streaming, contents will have the standard file-like-object read function, which accepts an optional size parameter to limit how much data is read per call. When streaming is on, be certain to fully read the contents before issuing another request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:if stream was True, contents is a file-like-object of the contents of the HTTP body. If stream was False, contents is just a simple str of the HTTP body.

head_account(headers=None, query=None, cdn=False)

HEADs the account and returns the results. Useful headers returned are:

x-account-bytes-used	Object storage used for the account, in bytes.
x-account-container-count	The number of containers in the account.
x-account-object-count	The number of objects in the account.

Also, any user headers beginning with x-account-meta- are returned.

These values can be delayed depending the Swift cluster.

Parameters:

• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

head_container(container, headers=None, query=None, cdn=False)

HEADs the container and returns the results. Useful headers returned are:

x-container-bytes-used	Object storage used for the container, in bytes.
x-container-object-count	The number of objects in the container.

Also, any user headers beginning with x-container-meta- are returned.

These values can be delayed depending the Swift cluster.

Parameters:

• container – The name of the container.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

head_object(container, obj, headers=None, query=None, cdn=False)

HEADs the object and returns the results.

Parameters:

• container – The name of the container.
• obj – The name of the object.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

no_content_methods = None

These HTTP methods do not allow contents

post_account(headers=None, query=None, cdn=False, body=None)

POSTs the account and returns the results. This is usually done to set X-Account-Meta-xxx headers. Note that any existing X-Account-Meta-xxx headers will remain untouched. To remove an X-Account-Meta-xxx header, send the header with an empty string as its value.

Parameters:

• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – No known Swift POSTs take a body; but the option is there for the future.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

post_container(container, headers=None, query=None, cdn=False, body=None)

POSTs the container and returns the results. This is usually done to set X-Container-Meta-xxx headers. Note that any existing X-Container-Meta-xxx headers will remain untouched. To remove an X-Container-Meta-xxx header, send the header with an empty string as its value.

Parameters:

• container – The name of the container.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – No known Swift POSTs take a body; but the option is there for the future.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

post_object(container, obj, headers=None, query=None, cdn=False, body=None)

POSTs the object and returns the results. This is used to update the object’s header values. Note that all headers must be sent with the POST, unlike the account and container POSTs. With account and container POSTs, existing headers are untouched. But with object POSTs, any existing headers are removed. The full list of supported headers depends on the Swift cluster, but usually include Content-Type, Content-Encoding, and any X-Object-Meta-xxx headers.

Parameters:

• container – The name of the container.
• obj – The name of the object.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – No known Swift POSTs take a body; but the option is there for the future.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

put_account(headers=None, query=None, cdn=False, body=None)

PUTs the account and returns the results. This is usually done with the extract-archive bulk upload request and has no other use I know of (but the call is left open in case there ever is).

Parameters:

• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – Some account PUT requests, like the extract-archive bulk upload request, take a body.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

put_container(container, headers=None, query=None, cdn=False, body=None)

PUTs the container and returns the results. This is usually done to create new containers and can also be used to set X-Container-Meta-xxx headers. Note that if the container already exists, any existing X-Container-Meta-xxx headers will remain untouched. To remove an X-Container-Meta-xxx header, send the header with an empty string as its value.

Parameters:

• container – The name of the container.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.
• body – Some container PUT requests, like the extract-archive bulk upload request, take a body.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

put_object(container, obj, contents, headers=None, query=None, cdn=False)

PUTs the object and returns the results. This is used to create or overwrite objects. X-Object-Meta-xxx can optionally be sent to be stored with the object. Content-Type, Content-Encoding and other standard HTTP headers can often also be set, depending on the Swift cluster.

Note that you can set the ETag header to the MD5 sum of the contents for extra verification the object was stored correctly.

Parameters:

• container – The name of the container.
• obj – The name of the object.
• contents – The contents of the object to store. This can be a simple str, or a file-like-object with at least a read function. If the file-like-object also has tell and seek functions, the PUT can be reattempted on any server error.
• headers – Additional headers to send with the request.
• query – Set to a dict of query values to send on the query string of the request.
• cdn – If set True, the CDN management interface will be used.

Returns:

A tuple of (status, reason, headers, contents).

status:is an int for the HTTP status code. reason:is the str for the HTTP status (ex: “Ok”). headers:is a dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:is the str for the HTTP body.

request(method, path, contents, headers, decode_json=False, stream=False, query=None, cdn=False)

Performs a direct HTTP request to the Swift service.

Parameters:

• method – The request method (‘GET’, ‘HEAD’, etc.)
• path – The request path.
• contents – The body of the request. May be a string or a file-like object.
• headers – A dict of request headers and values.
• decode_json – If set True, the response body will be treated as JSON and decoded result returned instead of the raw contents.
• stream – If set True, the response body will return as a file-like object; otherwise, the response body will be read in its entirety and returned as a string. Overrides decode_json.
• query – A dict of query parameters and values to append to the path.
• cdn – If set True, the request will be sent to the CDN management endpoint instead of the default storage endpoint.

Returns:

A tuple of (status, reason, headers, contents).

status:An int for the HTTP status code. reason:The str for the HTTP status (ex: “Ok”). headers:A dict with all lowercase keys of the HTTP headers; if a header has multiple values, it will be a list. contents:Depending on the decode_json and stream settings, this will either be the raw response string, the JSON decoded object, or a file-like object.

reset()

Resets the client, closing any connections and discarding any state. This can be useful if some exceptional condition occurred and the request/response state can no longer be certain.

user_agent = None

The string to use for the User-Agent request header.

manager

Contains the ClientManager class that can be used to manage a set of clients.

class swiftly.client.manager.ClientManager(client_class, *args, **kwargs)

Bases: object

Can be used to manage a set of clients.

Parameters:

• client_class – The class to create when a new client is needed.
• args – The args for the client constructor.
• kwargs – The keyword args for the client constructor.

get_client()

Obtains a client for use, whether an existing unused client or a brand new one if none are available.

put_client(client)

Returns a client back into the pool for availability to future calls to get_client. This should only be called if get_client was used to obtain the client; with_client is a context manager that does this for you.

with_client(*args, **kwds)

A context manager that obtains a client for use, whether an existing unused client or a brand new one if none are available.

utils

Contains general tools useful when accessing Swift services.

swiftly.client.utils.generate_temp_url(method, url, seconds, key)

Returns a TempURL good for the given request method, url, and number of seconds from now, signed by the given key.

swiftly.client.utils.get_trans_id_time(trans_id)

Returns the time.time() embedded in the trans_id or None if no time information is embedded.

Copied from the Swift codebase. Copyright (c) 2010-2012 OpenStack Foundation

swiftly.client.utils.headers_to_dict(headers)

Converts a sequence of (name, value) tuples into a dict where if a given name occurs more than once its value in the dict will be a list of values.

swiftly.client.utils.quote(value, safe='/:')

Much like urllib.quote in that it returns a URL encoded string for the given value, protecting the safe characters; but this version also ensures the value is UTF-8 encoded.

standardclient

Provides the standard client for accessing Swift services.

class swiftly.client.standardclient.StandardClient(auth_methods=None, auth_url=None, auth_tenant=None, auth_user=None, auth_key=None, auth_cache_path=None, region=None, snet=False, attempts=5, eventlet=None, chunk_size=65536, http_proxy=None, verbose=None, verbose_id='')

The standard client for accessing Swift services.

Parameters:

• auth_methods – Auth methods to use with the auth system, example: auth2key,auth2password,auth2password_force_tenant,auth1 If not specified, the best order will try to be determined; but if you notice it keeps making useless auth attempts and that drives you crazy, you can override that here. All the available auth methods are listed in the example.
• auth_url – The URL to the auth system.
• auth_tenant – The tenant to authenticate as, if needed. Default (if needed): same as auth_user.
• auth_user – The user to authenticate as.
• auth_key – The key to use when authenticating.
• auth_cache_path – Default: None. If set to a path, the storage URL and auth token are cached in the file for reuse. If there are already cached values in the file, they are used without authenticating first.
• region – The region to access, if supported by auth (Example: DFW).
• snet – Uses the internalURL if Auth v2 is used or prepends “snet-” to the host name of the storage URL if Auth v1 is used. This is usually only useful when working with Rackspace Cloud Files and wanting to use Rackspace ServiceNet. Default: False.
• attempts – The number of times to try requests if a server error occurs (5xx response). Default: 5
• eventlet – Default: None. If True, Eventlet will be used if installed. If False, Eventlet will not be used even if installed. If None, the default, Eventlet will be used if installed and its version is at least 0.11.0 when a CPU usage bug was fixed.
• chunk_size – Maximum size to read or write at one time.
• http_proxy – The URL to the tunnelling HTTP proxy to use. Default: None.
• verbose – Set to a func(msg, *args) that will be called with debug messages. Constructing a string for output can be done with msg % args.
• verbose_id – Set to a string you wish verbose messages to be prepended with; can help in identifying output when multiple Clients are in use.

auth()

See swiftly.client.client.Client.auth()

get_account_hash()

See swiftly.client.client.Client.get_account_hash()

request(method, path, contents, headers, decode_json=False, stream=False, query=None, cdn=False)

See swiftly.client.client.Client.request()

reset()

See swiftly.client.client.Client.reset()

directclient

Provides a direct client by using loaded Swift Proxy Server code to work with Swift.

class swiftly.client.directclient.DirectClient(swift_proxy=None, swift_proxy_storage_path=None, swift_proxy_cdn_path=None, attempts=5, eventlet=None, chunk_size=65536, verbose=None, verbose_id='')

Direct client by using loaded Swift Proxy Server code to work with Swift.

Parameters:

• swift_proxy – Default: None. If set, the swift.proxy.server.Application given will be used instead of creating a default Swift proxy application.
• swift_proxy_storage_path – The path to the Swift account to use (example: /v1/AUTH_test).
• swift_proxy_cdn_path – The path to the Swift account to use for CDN management (example: /v1/AUTH_test).
• attempts – The number of times to try requests if a server error occurs (5xx response). Default: 5
• eventlet – Default: None. If True, Eventlet will be used if installed. If False, Eventlet will not be used even if installed. If None, the default, Eventlet will be used if installed and its version is at least 0.11.0 when a CPU usage bug was fixed.
• chunk_size – Maximum size to read or write at one time.
• verbose – Set to a func(msg, *args) that will be called with debug messages. Constructing a string for output can be done with msg % args.
• verbose_id – Set to a string you wish verbose messages to be prepended with; can help in identifying output when multiple Clients are in use.

get_account_hash()

See swiftly.client.client.Client.get_account_hash()

request(method, path, contents, headers, decode_json=False, stream=False, query=None, cdn=False)

See swiftly.client.client.Client.request()

localmemcache

Provides a local in-memory memcache client lookalike for use with Swift Proxy Server code.

See swift.common.memcached.MemcacheRing for what this is acting as.

class swiftly.client.localmemcache.LocalMemcache

decr(key, delta=1, timeout=0, time=0)

delete(key)

get(key)

get_multi(keys, server_key)

incr(key, delta=1, timeout=0, time=0)

set(key, value, serialize=True, timeout=0, time=0)

set_multi(mapping, server_key, serialize=True, timeout=0, time=0)

nulllogger

Provides a logger lookalike that throws away everything sent to it for use with Swift Proxy Server code.

See swift.common.utils.LogAdapter for what this is acting as.

class swiftly.client.nulllogger.NullLogger

client_ip = 'client_ip'

debug(*args, **kwargs)

error(*args, **kwargs)

exception(*args, **kwargs)

increment(*args, **kwargs)

set_statsd_prefix(*args, **kwargs)

thread_locals = 'thread_locals'

txn_id = 'txn_id'

warn(*args, **kwargs)

warning(*args, **kwargs)