GCP Infrastructure Setup

This guide outlines the setup and configuration of various Google Cloud Platform (GCP) services used to support our application infrastructure. We have leveraged a combination of managed services and Compute Engine instances to create a robust, scalable, and secure environment.

Services Overview

VPC Networks and Subnets

The Virtual Private Cloud (VPC) is used to create a secure network environment, isolating different components of the infrastructure.

VPC firewall rules are configured to control inbound and outbound traffic, ensuring that only necessary communication is allowed.

Google Kubernetes Engine (GKE)

  • Purpose: GKE is used to deploy and manage application components that are mostly stateless. The managed Kubernetes service provides scalability, automatic updates, and seamless integration with other GCP services.

  • Components: Includes multiple stateless microservices, deployed across different namespaces within the cluster.

Google Compute Engine (GCE)

  • Bastion Host: A secure entry point for administrative access to internal GCE instances. SSH access is restricted and audited.

  • Data Layer GCEs: We will have three compute engine machines for stateful data including PostgreSQL, Kafka, MongoDB. These servers will also host High availability components required for these applications such as pgbouncer, HAProxy, Patroni.

  • Security: Each GCE instance is secured using firewall rules, IAM roles, and encryption of data at rest and in transit.

Load Balancers

External Load Balancers are used to provide public access to the application services hosted on GKE. These load balancers distribute incoming traffic across the available nodes, ensuring high availability, configured with SSL/TLS certificates to encrypt data in transit, ensuring secure communication between clients and the application.

An Internal Load Balancer is used to manage and route internal traffic, particularly for logging. Logs from GCE instances are pushed to the ELK stack hosted on GKE through this load balancer.

Google Cloud Storage

Cloud Storage is used to store logs generated by Airflow. These logs are securely stored and can be easily accessed for auditing and troubleshooting.

The bucket is configured with lifecycle rules to manage storage costs by automatically deleting old logs after a specified period.