Manoj Kumar
Building a GitHub Action Workflow to Transfer Data to S3 with Cron job Pipeline

Building a GitHub Action Workflow to Transfer Data to S3 with Cron job Pipeline

Manoj Kumar's photo
Manoj Kumar
·

2 min read

Welcome to our step-by-step guide on creating a GitHub Action workflow to seamlessly transfer data from your GitHub repository to an AWS S3 bucket. With this automated pipeline, you can keep your data up-to-date in the cloud, making collaboration and sharing easier than ever.

Prerequisites:

Before we begin, ensure you have the following:

  1. An AWS account with the necessary permissions to create an S3 bucket and access AWS services.

  2. A GitHub repository containing the data you want to transfer.

  3. Basic familiarity with GitHub Actions, YAML, and the AWS CLI.

Step 1: Create an S3 Bucket

  1. Sign in to your AWS Management Console.

  2. Navigate to the S3 service and click "Create Bucket."

  3. Follow the on-screen instructions to set up your S3 bucket, specifying a unique name, region, and any other desired configurations.

Step 2: Set up AWS Credentials in GitHub Secrets

  1. Generate an AWS Access Key ID and Secret Access Key in the AWS IAM console with sufficient permissions for S3 operations.

  2. In your GitHub repository, go to Settings > Secrets > New repository secret.

  3. Add your AWS credentials as secrets, for example, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Step 3: Create the GitHub Action Workflow

  1. In your GitHub repository, navigate to the .github/workflows/ directory.

  2. Create a new YAML file, e.g., data_transfer.yml, to define your workflow.

Step 4: Define the Workflow

name: Data Transfer to S3

on:

schedule:

- cron: '0 */5 * * *' # Every 5 hours

jobs:

transfer_to_s3:

runs-on: ubuntu-latest

steps:

- name: Check out code

uses: actions/checkout@v2

- name: Transfer data to S3

run: |

aws s3 sync /path/to/your/data s3://your-s3-bucket/

env:

AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}

AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_S

ECRET_ACCESS_KEY }}

Replace /path/to/your/data with the path to your data within the repository and your-s3-bucket with the name of your S3 bucket

Save the changes to your data_transfer.yml file and commit it to your GitHub repository.

Mubarak ho aapki GitHub Action Pipeline ban chuki h

TrainWithShubhamDevopsGitHubGitAWS