23 - Flexible Partitions Filter (PUSH-DOWN)

• partition_filter argument:

  - Callback Function filters to apply on PARTITION columns (PUSH-DOWN filter).
  - This function MUST receive a single argument (Dict[str, str]) where keys are partitions names and values are partitions values.
  - This function MUST return a bool, True to read the partition or False to ignore it.
  - Ignored if `dataset=False`.
  

P.S. Check the function API doc to see it has some argument that can be configured through Global configurations.

import pandas as pd

import awswrangler as wr

Enter your bucket name:

import getpass

bucket = getpass.getpass()
path = f"s3://{bucket}/dataset/"

 ············

Creating the Dataset (Parquet)

df = pd.DataFrame(
    {
        "id": [1, 2, 3],
        "value": ["foo", "boo", "bar"],
    }
)

wr.s3.to_parquet(df=df, path=path, dataset=True, mode="overwrite", partition_cols=["value"])

wr.s3.read_parquet(path, dataset=True)

	id	value
0	3	bar
1	2	boo
2	1	foo

Parquet Example 1

def my_filter(x):
    return x["value"].endswith("oo")


wr.s3.read_parquet(path, dataset=True, partition_filter=my_filter)

	id	value
0	2	boo
1	1	foo

Parquet Example 2

from Levenshtein import distance


def my_filter(partitions):
    return distance("boo", partitions["value"]) <= 1


wr.s3.read_parquet(path, dataset=True, partition_filter=my_filter)

	id	value
0	2	boo
1	1	foo

Creating the Dataset (CSV)

df = pd.DataFrame(
    {
        "id": [1, 2, 3],
        "value": ["foo", "boo", "bar"],
    }
)

wr.s3.to_csv(
    df=df, path=path, dataset=True, mode="overwrite", partition_cols=["value"], compression="gzip", index=False
)

wr.s3.read_csv(path, dataset=True)

	id	value
0	3	bar
1	2	boo
2	1	foo

CSV Example 1

def my_filter(x):
    return x["value"].endswith("oo")


wr.s3.read_csv(path, dataset=True, partition_filter=my_filter)

	id	value
0	2	boo
1	1	foo

CSV Example 2

from Levenshtein import distance


def my_filter(partitions):
    return distance("boo", partitions["value"]) <= 1


wr.s3.read_csv(path, dataset=True, partition_filter=my_filter)

	id	value
0	2	boo
1	1	foo