20 - Spark Table Interoperability

awswrangler has no difficulty to insert, overwrite or do any other kind of interaction with a Table created by Apache Spark.

But if you want to do the opposite (Spark interacting with a table created by awswrangler) you should be aware that awswrangler follows the Hive’s format and you must be explicit when using the Spark’s saveAsTable method:

spark_df.write.format("hive").saveAsTable("database.table")

Or just move forward using the insertInto alternative:

spark_df.write.insertInto("database.table")