awswrangler.s3.merge_datasets

awswrangler.s3.merge_datasets(source_path: str, target_path: str, mode: Literal['append', 'overwrite', 'overwrite_partitions'] = 'append', ignore_empty: bool = False, use_threads: bool | int = True, boto3_session: Session | None = None, s3_additional_kwargs: dict[str, Any] | None = None) → list[str]

Merge a source dataset into a target dataset.

This function accepts Unix shell-style wildcards in the source_path argument. * (matches everything), ? (matches any single character), [seq] (matches any character in seq), [!seq] (matches any character not in seq). If you want to use a path which includes Unix shell-style wildcard characters (*, ?, []), you can use glob.escape(source_path) before passing the path to this function.

Note

If you are merging tables (S3 datasets + Glue Catalog metadata), remember that you will also need to update your partitions metadata in some cases. (e.g. wr.athena.repair_table(table=’…’, database=’…’))

Note

In case of use_threads=True the number of threads that will be spawned will be gotten from os.cpu_count().

Note

Following arguments are not supported in distributed mode with engine EngineEnum.RAY:

  • boto3_session

Parameters:

  • source_path (str) – S3 Path for the source directory.

  • target_path (str) – S3 Path for the target directory.

  • mode (Literal['append', 'overwrite', 'overwrite_partitions']) – append (Default), overwrite, overwrite_partitions.

  • ignore_empty (bool) – Ignore files with 0 bytes.

  • use_threads (bool | int) – True to enable concurrent requests, False to disable multiple threads. If enabled os.cpu_count() will be used as the max number of threads. If integer is provided, specified number is used.

  • boto3_session (Session | None) – Boto3 Session. The default boto3 session will be used if boto3_session receive None.

  • s3_additional_kwargs (dict[str, Any] | None) – Forwarded to botocore requests. e.g. s3_additional_kwargs={‘ServerSideEncryption’: ‘aws:kms’, ‘SSEKMSKeyId’: ‘YOUR_KMS_KEY_ARN’}

Return type:

list[str]

Returns:

List of new objects paths.

Examples

Merging

>>> import awswrangler as wr
>>> wr.s3.merge_datasets(
...     source_path="s3://bucket0/dir0/",
...     target_path="s3://bucket1/dir1/",
...     mode="append"
... )
["s3://bucket1/dir1/key0", "s3://bucket1/dir1/key1"]

Merging with a KMS key

>>> import awswrangler as wr
>>> wr.s3.merge_datasets(
...     source_path="s3://bucket0/dir0/",
...     target_path="s3://bucket1/dir1/",
...     mode="append",
...     s3_additional_kwargs={
...         'ServerSideEncryption': 'aws:kms',
...         'SSEKMSKeyId': 'YOUR_KMS_KEY_ARN'
...     }
... )
["s3://bucket1/dir1/key0", "s3://bucket1/dir1/key1"]