awswrangler.emr_serverless.run_job

awswrangler.emr_serverless.run_job(application_id: str, execution_role_arn: str, job_driver_args: dict[str, Any] | SparkSubmitJobArgs | HiveRunJobArgs, job_type: Literal['Spark', 'Hive'] = 'Spark', wait: bool = True, configuration_overrides: dict[str, Any] | None = None, tags: dict[str, str] | None = None, execution_timeout: int | None = None, name: str | None = None, emr_serverless_job_wait_polling_delay: float = 5, boto3_session: Session | None = None) → str | dict[str, Any]

Run an EMR serverless job.

https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html

Note

This function has arguments which can be configured globally through wr.config or environment variables:

  • emr_serverless_job_wait_polling_delay

Check out the Global Configurations Tutorial for details.

Warning

This API is experimental and may change in future AWS SDK for Pandas releases.

Parameters:

  • application_id (str) – The id of the application on which to run the job.

  • execution_role_arn (str) – The execution role ARN for the job run.

  • job_driver_args (Union[Dict[str, str], SparkSubmitJobArgs, HiveRunJobArgs]) – The job driver arguments for the job run.

  • job_type (str, optional) – Type of the job: “Spark” or “Hive”. Defaults to “Spark”.

  • wait (bool, optional) – Whether to wait for the job completion or not. Defaults to true.

  • configuration_overrides (Dict[str, str], optional) – The configuration overrides for the job run.

  • tags (Dict[str, str], optional) – Key/Value collection to put tags on the application. e.g. {“foo”: “boo”, “bar”: “xoo”})

  • execution_timeout (int, optional) – The maximum duration for the job run to run. If the job run runs beyond this duration, it will be automatically cancelled.

  • name (str, optional) – Name of the job.

  • emr_serverless_job_wait_polling_delay (int, optional) – Time to wait between polling attempts.

  • boto3_session (boto3.Session(), optional) – Boto3 Session. The default boto3 session will be used if boto3_session receive None.

Returns:

Job Id if wait=False, or job run details.

Return type:

Union[str, Dict[str, Any]]