Components Installations

We will be installing required components on Kubernetes using helm charts and helmfile for easier deployments.

• Create Namespaces
• Create ImagePullSecrets
• Create Values files for all components
• Install components using helmfile sync

Create Namespaces

Create spark-jobs, airflow and xchange namespace:

kubectl create ns spark-jobs
kubectl create ns airflow
kubectl create ns xchange

Create acr-secret

For pulling images from helix repo you need to create acr-secret for pulling images, you will get this secret in a differnt file.

kubectl apply -f acr-secret.yaml

Create Custom Values Files

Nginx Ingress Controller

We are using nginx ingress controller for cloud agnostic deployments. Helm charts are being used for installation of nginx ingress controller.

Prepare custom values for Ingress controller installation:

controller:
  replicaCount: 2
  networkPolicy:
    enabled: false
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
  allowSnippetAnnotations: true
  config:
    enable-modsecurity: true
    enable-owasp-modsecurity-crs: true
    modsecurity-snippet: |
      #SecRuleEngine DetectionOnly
      SecRuleEngine On
      SecRequestBodyAccess On
      SecStatusEngine Off
      SecAuditEngine RelevantOnly
      SecAuditLogRelevantStatus "^(?:5|4(?!04))"

      SecAuditLogParts ABIJDEFHZ
      SecAuditLogType Serial
      SecAuditLog /var/log/modsec_audit.log
      SecAuditLogFormat JSON
      SecRuleRemoveById 200004

      # Enable XML and JSON parsing
      SecRule REQUEST_HEADERS:Content-Type "(?:application(?:/soap\+|/)|text/)xml" "id:200000,phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
      SecRule REQUEST_HEADERS:Content-Type "application/json" "id:200001,phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
      # Max request sizes in bytes (with/without files)
      # Note NGINX Ingress has its own annotations, keep in sync! # 200Mb (default is 12.5Mb)
      SecRequestBodyLimit 209715200
      SecRequestBodyNoFilesLimit 262144 # 250Kb (default is 128Kb)
      # SecRequestBodyLimitAction Reject # Reject if larger (we could also let it pass with ProcessPartial)
      # Update config to include PUT/PATCH/DELETE in the allowed HTTP methods (instead of fully disabling 911100)
      SecAction "id:900200,phase:1,nolog,pass,t:none,setvar:tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE"

      ##### CUSTOM Setting
      SecRuleRemoveById 920350
      SecRuleRemoveById 920300
      SecRuleRemoveById 920320
      SecRuleRemoveById 920470
      SecRuleRemoveById 942440
      SecRuleRemoveById 920420
      SecRuleRemoveById 949110

cert-manager

Cert manager is required for having certificates

crds:
  enabled: true

Spark Operator

Custom values for spark operator: spark-op.yaml

sparkJobNamespaces:
- spark-jobs
- airflow
webhook:
  enable: true
  namespaceSelector: kubernetes.io/metadata.name=spark-jobs,kubernetes.io/metadata.name=spark,kubernetes.io/metadata.name=airflow

Redis

We are installing redis on Kubernetes using helm charts:

Prepare redis_server.yaml for custom values:

architecture: standalone
auth:
  enabled: false
  sentinel: false

commonConfiguration: |-
  appendonly no

master:
  persistence:
    size: 2Gi

metrics:
  enabled: true

Install xchange Prerequisite (Permissions)

We have xchange prerequisite helm chart prepared for installing pre-requisites on Kubernetes Cluster, prepare values for it xchange-perm.yaml.

# Supported azure/gcp
cloud: "gcp"
clusterIssuer:
  email: devops@helixbeat.com

Minio

We are using minio as intermediate storage for our jobs:

Prepare values for minio tenant minio.yaml

replicas: 2
persistence:
  size: 50Gi

nodeSelector:
  nodepool-type: worker

tolerations:
- key: nodepool-type
  operator: Equal
  value: worker
  effect: NoSchedule

resources:
  limits:
    memory: 2Gi
    cpu: 700m
  requests:
    memory: 1Gi
    cpu: 50m

buckets:
  - name: dev
    policy: none

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: cert-manager
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - minio.demo.helixbeat.com
  tls:
   - secretName: minio-api-tls
     hosts:
       - minio.demo.helixbeat.com


consoleIngress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: cert-manager
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - console.demo.helixbeat.com
  tls:
   - secretName: minio-console-tls
     hosts:
       - console.demo.helixbeat.com

Airflow

Prepare airflow.yaml custom values file for installing airflow on the cluster:

You need to place below variables as provided or calculated on own:

• IMAGE_TAG: This is image release tag provided by Helix
• WEBSERVER_SECRET_KEY: Secret required for Airflow (python3 -c 'import secrets; print(secrets.token_hex(16))')
• FERNET_KEY: Secret key required for airflow (Online Generator)

executor: KubernetesExecutor
images:
  airflow:
    repository: helix.azurecr.io/airflow_helix
    tag: <IMAGE_TAG>
    pullPolicy: Always

registry:
  secretName: acr-secret

nodeSelector:
  nodepool-type: worker

tolerations:
- key: nodepool-type
  operator: Equal
  value: worker
  effect: NoSchedule

fernetKey: <FERNET_KEY>
webserverSecretKey: <WEBSERVER_SECRET_KEY>

extraConfigMaps:
  'xchange-airflow-variables':
    data: |
      KAFKA_BROKER_URL: 10.2.16.5:9092,10.2.16.4:9092,10.2.16.6:9092
      PIPELINE_OUTPUT_KAFKA_TOPIC: target-topic-handoff
      PIPELINE_MONITOR_KAFKA_TOPIC: pipeline_monitor
      CLOUD_STORAGE_BASE_URL: minio.demo.helixbeat.com
      CLOUD_STORAGE_ACCESS_KEY: xxx
      CLOUD_STORAGE_SECRET_KEY: xxx
      BUCKET_NAME_FOR_STORING_DAG_CONF: dev
      SPARK_K8S_SENSOR_TIMEOUT_IN_SECONDS: "600"
      SPARK_K8S_SENSOR_POKE_INTERVAL: "1"
      DAGS_OWNER: helix

scheduler:
  replicas: 2
  resources:
   limits:
    cpu: 700m
    memory: 1024Mi
   requests:
    cpu: 100m
    memory: 256Mi

  nodeSelector: {}
    # nodepool-type: worker

webserver:
  defaultUser:
    password: xxx

  resources:
   limits:
    cpu: 1000m
    memory: 3072Mi
   requests:
    cpu: 50m
    memory: 1024Mi

dags:
  persistence:
    enabled: false

flower:
  enabled: false

ingress:
  enabled: true
  web:
    annotations:
      cert-manager.io/cluster-issuer: cert-manager
      # k8s.apisix.apache.org/http-to-https: "true"
      kubernetes.io/ingress.class: nginx
    host: airflow.demo.helixbeat.com
    ingressClassName: nginx
    path: /
    tls:
      enabled: true
      secretName: airflow-tls-secret

workers:
  persistence:
    size: 10Gi

triggerer:
  resources:
   limits:
    cpu: 500m
    memory: 1024Mi
   requests:
    cpu: 50m
    memory: 256Mi
  persistence:
    size: 10Gi

statsd:
  resources:
   limits:
    cpu: 200m
    memory: 256Mi
   requests:
    cpu: 20m
    memory: 100Mi
logs:
  persistence:
    enabled: true
    size: 10Gi
    storageClassName: airflow-logs

extraEnv: |
  - name: AIRFLOW__API__AUTH_BACKENDS
    value: 'airflow.api.auth.backend.basic_auth'

extraEnvFrom: |
  - configMapRef:
      name: 'xchange-airflow-variables'

Create HelmFile

We will be using helmfile utility to install all the required components single command. Create helmfile.yaml as below:

Note: Make sure to update the values files path.

repositories:
- name: jetstack
  url: https://charts.jetstack.io
- name: ingress-nginx
  url: https://kubernetes.github.io/ingress-nginx
- name: spark-operator
  url: https://kubeflow.github.io/spark-operator
- name: apache-airflow
  url: https://airflow.apache.org
- name: minio-operator
  url: https://operator.min.io
- name: minio
  url: https://charts.min.io

releases:
- name: ingress-nginx
  namespace: nginx
  chart: ingress-nginx/ingress-nginx
  values:
    - ./values_files/demo/ingress-nginx.yaml

- name: cert-manager
  namespace: cert-manager
  chart: jetstack/cert-manager
  version: v1.15.0
  values:
    - ./values_files/demo/cert-manager.yaml

- name: minio
  namespace: minio-tenant
  chart: minio/minio
  version: 5.2.0
  values:
  - ./values_files/demo/minio.yaml

- name: spark-operator
  namespace: spark-operator 
  chart: spark-operator/spark-operator
  version: 1.2.15
  values:
    - ./values_files/demo/spark-op.yaml

- name: xchange-redis
  namespace: xchange
  chart: oci://registry-1.docker.io/bitnamicharts/redis
  version: 19.5.3
  values:
    - ./values_files/demo/redis_server.yaml

- name: xchange-perm
  namespace: xchange
  chart: oci://helix.azurecr.io/helm/xchange
  version: 0.1.0
  values:
    - ./values_files/demo/xchange-perm.yaml

- name: airflow
  namespace: airflow
  chart: apache-airflow/airflow
  version: 1.13.1
  values:
  - ./values_files/demo/airflow.yaml

- name: xchange-api
  namespace: xchange
  chart: oci://helix.azurecr.io/helm/xchange-api
  version: 0.1.0
  values:
  - ./values_files/demo/api.yaml

- name: xchange-camel
  namespace: xchange
  chart: oci://helix.azurecr.io/helm/xchange-camel
  version: 0.1.0
  values:
  - ./values_files/demo/camel.yaml

- name: fhir-server
  namespace: xchange
  chart: oci://helix.azurecr.io/helm/fhir-server
  version: 0.1.0
  values:
  - ./values_files/demo/fhir.yaml

- name: xchange-docs
  namespace: xchange
  chart: oci://helix.azurecr.io/helm/xchange-docs
  version: 0.1.0
  values:
  - ./values_files/demo/xchange-docs.yaml

Install Application with Helmfile

Install the application using helmfile: helmfile sync -f helmfile.yaml

After installation please have a check over ingress objects there should be Load Balancer attached and IP allocated, after which we need to create DNS entry for the host and IP mentioned.

helm install keycloak oci://helix.azurecr.io/helm/keycloak -n xchange-cert --create-namespace -f keycloak.yaml