Interface to Amazon S3

Airflow provides an interface to Amazon S3 by using the Providers.

See Provider examples of Amazon S3 at <airflow_home>/build/env/lib/python3.9/site-packages/airflow/providers/ezmeral/s3/example_dags/.

Hooks

EzS3Hook

Python path: airflow.providers.ezmeral.s3.hooks.s3

Description: Interacts with Amazon S3 using Boto3 library.

Operators

EzS3CreateBucketOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_bucket

Description: Creates an S3 bucket.

EzS3DeleteBucketOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_bucket

Description: Deletes an S3 bucket.

EzS3GetBucketTaggingOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging

Description: Gets tagging from an S3 bucket.

EzS3PutBucketTaggingOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging

Description: Puts tagging for an S3 bucket.

EzS3DeleteBucketTaggingOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging

Description: Deletes tagging from an S3 bucket.

EzS3CopyObjectOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_copy_object

Description: Creates a copy of the stored S3 object.

EzS3DeleteObjectsOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_delete_objects

Description: Deletes single or multiple objects from a bucket using a single HTTP request.

EzS3ListOperator

Python path: airflow.providers.ezmeral.s3.operators.s3_list

Description: List all objects from the bucket with the given string prefix in name.

Sensors

EzS3KeySensor

Python path: airflow.providers.ezmeral.s3.sensors.s3_key

Description: Waits for a key to show up in a S3 bucket. S3 is a key-value store and key uniquely identifies the object in the bucket. S3 does not support folders and the path is just a key of resource.

EzS3KeyUnchangedSensor

Python path: airflow.providers.ezmeral.s3.sensors.s3_keys_unchanged

Description: Checks for changes in the number of objects at prefix in S3 bucket. This sensor returns True if the inactivity period has passed with no increase in the number of objects. This sensor might give an error when you use it in reschedule mode. Between rescheduled invocations, S3 bucket loses the state of the listed objects.

Transfers

EzGCSToS3Operator

Python path: airflow.providers.ezmeral.s3.transfers.gcs_to_s3

Description: Synchronizes a Google Cloud Storage bucket with an Amazon S3 bucket.

EzLocalFilesystemToS3Operator

Python path: airflow.providers.ezmeral.s3.transfers.local_to_s3

Description: Uploads a file from a local filesystem to S3.

EzMongoToS3Operator

Python path: airflow.providers.ezmeral.s3.transfers.mongo_to_s3

Description: Moves data from MongoDB using PyMongo to Amazon S3 using Boto.

EzSQLToS3Operator

Python path: airflow.providers.ezmeral.s3.transfers.sql_to_s3.

Description: Saves data from a specific SQL query into a file in S3.

EzS3ToFTPOperator

Python path: airflow.providers.ezmeral.s3.transfers.s3_to_ftp

Description: Enables the transfer of files from S3 to a FTP server.

EzS3ToSFTPOperator

Python path: airflow.providers.ezmeral.s3.transfers.s3_to_sftp

Description: Enables the transfer of files from S3 to a SFTP server.

EzSFTPToS3Operator

Python path: airflow.providers.ezmeral.s3.transfers.sftp_to_s3

Description: Enables the transfer of files from a SFTP server to S3.