3. Use TypedDict to group similar parameters¶
AWS SDK for pandas API methods contain many parameters which are related to a specific behaviour or setting. For example, methods which have an option to update the Glue AWScatalog, such as
to_parquet, contain a list of parameters that define the settings for the table in AWS Glue. These settings include the table description, column comments, the table type, etc.
As a consequence, some of our functions have grown to include dozens of parameters. When reading the function signatures, it can be unclear which parameters are related to which functionality. For example, it’s not immediately obvious that the parameter
s3.to_parquet only writes the column comments into the AWS Glue catalog, and not to S3.
Parameters that are related to similar functionality will be replaced by a single parameter of type TypedDict. This will allow us to reduce the amount of parameters for our API functions, and also make it clearer that certain parameters are only related to specific functionalities.
For example, parameters related to Athena cache settings will be extracted into a parameter of type
AthenaCacheSettings, parameters related to Ray settings will be extracted into
The usage of
TypedDict allows the user to define the parameters as regular dictionaries with string keys, while empowering type checkers such as
mypy. Alternately, implementations such as
AthenaCacheSettings can be instantiated as classes.
The main alternative that was considered was the idea of using
dataclass instead of
TypedDict. The advantage of this alternative would be that default values for parameters could be defined directly in the class signature, rather than needing to be defined in the function which uses the parameter.
On the other hand, the main issue with using
dataclass is that it would require the customer figure out which class needs to be imported. With
TypedDict, this is just one of the options; the parameters can simply be passed as a typical Python dictionary.
This alternative was discussed in more detail as part of PR#1855.
TypedDict such as
RaySettings have been created. They are defined in the
These parameters grouping can used in either of the following two ways:
"SELECT * FROM ...",,
"SELECT * FROM ...",,
Many of our functions signatures have been changes to take advantage of this refactor. Many of these are breaking changes which will be released as part of the next major version: