Infrastructure Creation

This guide provides the steps and necessary details for creating infrastructure using Terraform scripts for different cloud environments. Customize the components according to your specific requirements.

General Instructions

  1. Identify Required Components: Determine the components needed for your environment.
  2. Customize env.tfvars: Prepare and modify the environment-specific env.tfvars file.
  3. Plan and Apply: Use Terraform commands to plan and apply the infrastructure changes.

Infrastructure Components on GCP

The following components will be created in Google Cloud Platform (GCP):

  • VPC Network
  • Subnets:
    • VM Subnet
    • API Subnet
    • GKE Subnet
  • GKE Cluster
  • Three VMs with Data Disk Attached
  • Bastion Host (for private access) with Public IP
  • Storage Bucket for Airflow logs

Steps to Create Infrastructure on GCP

1. Prepare env.tfvars File

Create and modify the env.tfvars file for your environment. Below is a sample file for reference:

# Your GCP project ID
project_id         = "XXXX"

# Environment Name
environment        = "dev"

# Region and Location to install infrastructure
region             = "asia-south2"
location           = "asia-south2-a"

# VPC Network Address space
vnet_address_space = "10.0.0.0/16"

# GKE Cluster subnet CIDR
gke_subnet_space   = "10.0.0.0/20"

# GKE API Cluster subnet CIDR
api_subnet_space   = "10.0.64.128/25"

# VM subnet CIDR
vm_subnet_space    = "10.0.16.0/20"

# Appgw Subnet CIDR
appgw_subnet_space = "10.0.64.0/25"

# Node type for GKE cluster System nodes
gke_node_type        = "c2-standard-4"

# Node type for GKE cluster Worker nodes
gke_worker_node_type = "c2-standard-4"

# Node type for Bastion Host
bastion_node_type    = "n2-standard-2"

# Node type for Kafka Nodes
kafka_node_type      = "n2-standard-4"

# Secondary/Data disk size for Kafka Nodes
kafka_disk_size_gb   = 100

2. Plan the Infrastructure

Run the Terraform plan command to verify the infrastructure that will be created:

terraform plan -var-file=env.tfvars

Review the output to ensure all components and configurations are correct.

3. Modify Parameters as Needed

Adjust the parameters in the env.tfvars file according to your requirements (e.g., node types, disk size).

4. Apply the Terraform Scripts

Execute the Terraform apply command to create the infrastructure:

terraform apply -var-file=env.tfvars

Confirm the action when prompted.