awswrangler.s3.describe_objects

awswrangler.s3.describe_objects(path: str | list[str], version_id: str | dict[str, str] | None = None, use_threads: bool | int = True, last_modified_begin: datetime | None = None, last_modified_end: datetime | None = None, s3_additional_kwargs: dict[str, Any] | None = None, boto3_session: Session | None = None) → dict[str, dict[str, Any]]

Describe Amazon S3 objects from a received S3 prefix or list of S3 objects paths.

Fetch attributes like ContentLength, DeleteMarker, last_modified, ContentType, etc The full list of attributes can be explored under the boto3 head_object documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object

This function accepts Unix shell-style wildcards in the path argument. * (matches everything), ? (matches any single character), [seq] (matches any character in seq), [!seq] (matches any character not in seq). If you want to use a path which includes Unix shell-style wildcard characters (*, ?, []), you can use glob.escape(path) before passing the path to this function.

Note

In case of use_threads=True the number of threads that will be spawned will be gotten from os.cpu_count().

Note

The filter by last_modified begin last_modified end is applied after list all S3 files

Note

Following arguments are not supported in distributed mode with engine EngineEnum.RAY:

  • boto3_session

  • s3_additional_kwargs

Parameters:

  • path (str | list[str]) – S3 prefix (accepts Unix shell-style wildcards) (e.g. s3://bucket/prefix) or list of S3 objects paths (e.g. [s3://bucket/key0, s3://bucket/key1]).

  • version_id (str | dict[str, str] | None) – Version id of the object or mapping of object path to version id. (e.g. {‘s3://bucket/key0’: ‘121212’, ‘s3://bucket/key1’: ‘343434’})

  • use_threads (bool | int) – True to enable concurrent requests, False to disable multiple threads. If enabled os.cpu_count() will be used as the max number of threads. If integer is provided, specified number is used.

  • last_modified_begin (datetime | None) – Filter the s3 files by the Last modified date of the object. The filter is applied only after list all s3 files.

  • last_modified_end (datetime | None) – Filter the s3 files by the Last modified date of the object. The filter is applied only after list all s3 files.

  • s3_additional_kwargs (dict[str, Any] | None) – Forwarded to botocore requests. e.g. s3_additional_kwargs={‘RequestPayer’: ‘requester’}

  • boto3_session (Session | None) – Boto3 Session. The default boto3 session will be used if boto3_session receive None.

Return type:

dict[str, dict[str, Any]]

Returns:

Return a dictionary of objects returned from head_objects where the key is the object path. The response object can be explored here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object

Examples

>>> import awswrangler as wr
>>> descs0 = wr.s3.describe_objects(['s3://bucket/key0', 's3://bucket/key1'])  # Describe both objects
>>> descs1 = wr.s3.describe_objects('s3://bucket/prefix')  # Describe all objects under the prefix