Highly Available Cluster Setup

We’re overhauling Dgraph’s docs to make them clearer and more approachable. If you notice any issues during this transition or have suggestions, please let us know.

You can run three Dgraph Alpha servers and three Dgraph Zero servers in a highly available cluster setup. For a highly available setup, start the Dgraph Zero server with --replicas 3 flag, so that all data is replicated on three Alpha servers and forms one Alpha group. You can install a highly available cluster using:

• dgraph-ha.yaml file
• Helm charts.

Install a highly available Dgraph cluster using YAML or Helm

• YAML
• Helm

Before you begin

• Install the Kubernetes command line tool.
• Ensure that you have a production-ready Kubernetes cluster with at least three worker nodes running in a cloud provider of your choice.
• (Optional) To run Dgraph Alpha with TLS, see TLS Configuration.

Installing a highly available Dgraph cluster

1. Verify that you are able to access the nodes in the Kubernetes cluster:

   kubectl get nodes
   

   An output similar to this appears:

     NAME                                          STATUS   ROLES    AGE   VERSION
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
   

   After your Kubernetes cluster is up, you can use dgraph-ha.yaml to start the cluster.

2. Start a StatefulSet that creates Pods with Zero, Alpha, and Ratel UI:

   kubectl create --filename https://raw.githubusercontent.com/hypermodeinc/dgraph/main/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml
   

   An output similar to this appears:

   service/dgraph-zero-public created
   service/dgraph-alpha-public created
   service/dgraph-ratel-public created
   service/dgraph-zero created
   service/dgraph-alpha created
   statefulset.apps/dgraph-zero created
   statefulset.apps/dgraph-alpha created
   deployment.apps/dgraph-ratel created
   

3. Confirm that the Pods were created successfully.

   kubectl get pods
   

   An output similar to this appears:

    NAME                  READY   STATUS    RESTARTS   AGE
   dgraph-alpha-0        1/1     Running   0          6m24s
   dgraph-alpha-1        1/1     Running   0          5m42s
   dgraph-alpha-2        1/1     Running   0          5m2s
   dgraph-ratel-<pod-id> 1/1     Running   0          6m23s
   dgraph-zero-0         1/1     Running   0          6m24s
   dgraph-zero-1         1/1     Running   0          5m41s
   dgraph-zero-2         1/1     Running   0          5m6s
   

   You can check the logs for the Pod using kubectl logs --follow <POD_NAME>..

4. Port forward from your local machine to the Pod:

      kubectl port-forward service/dgraph-alpha-public 8080:8080
      kubectl port-forward service/dgraph-ratel-public 8000:8000
   

5. Go to http://localhost:8000 to access Dgraph using the Ratel UI.

You can also access the service on its External IP address.

Deleting highly available Dgraph resources

Delete all the resources using:

kubectl delete --filename https://raw.githubusercontent.com/hypermodeinc/dgraph/main/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml
kubectl delete persistentvolumeclaims --selector app=dgraph-zero
kubectl delete persistentvolumeclaims --selector app=dgraph-alpha

Dgraph configuration files

You can create a Dgraph configuration files for Alpha server and Zero server with Helm chart configuration values, <MY-CONFIG-VALUES>. For more information about the values, see the latest configuration settings.

1. Open an editor of your choice and create a configuration file named <MY-CONFIG-VALUES>.yaml:

   # <MY-CONFIG-VALUES>.yaml
   alpha:
     configFile:
       config.yaml: |
         alsologtostderr: true
         badger:
           compression_level: 3
           tables: mmap
           vlog: mmap
         postings: /dgraph/data/p
         wal: /dgraph/data/w
   zero:
     configFile:
       config.yaml: |
         alsologtostderr: true
         wal: /dgraph/data/zw
   

2. Change to the director in which you created <MY-CONFIG-VALUES>.yaml and then install with Alpha and Zero configuration using:

   helm install <RELEASE-NAME> dgraph/dgraph --values <MY-CONFIG-VALUES>.yaml
   

Exposing Alpha and Ratel Services

By default Zero and Alpha services are exposed only within the Kubernetes cluster as Kubernetes service type ClusterIP.

In order to expose the Alpha service and Ratel service publicly you can use Kubernetes service type LoadBalancer or an Ingress resource.

• LoadBalancer
• Ingress Resource

Public Internet

To use an external load balancer, set the service type to LoadBalancer.

For security purposes we recommend limiting access to any public endpoints, such as using a white list.

1. To expose Alpha service to the Internet use:

   helm install <RELEASE-NAME> dgraph/dgraph --set alpha.service.type="LoadBalancer"
   

2. To expose Alpha and Ratel services to the Internet use:

   helm install <RELEASE-NAME> dgraph/dgraph --set alpha.service.type="LoadBalancer" --set ratel.service.type="LoadBalancer"
   

Private network

An external load balancer can be configured to face internally to a private subnet rather the public Internet. This way it can be accessed securely by clients on the same network, through a VPN, or from a jump server. In Kubernetes, this is often configured through service annotations by the provider. Here’s a small list of annotations from cloud providers:

Provider	Documentation Reference	Annotation
AWS	Amazon EKS: Load Balancing	service.beta.kubernetes.io/aws-load-balancer-internal: "true"
Azure	AKS: Internal Load Balancer	service.beta.kubernetes.io/azure-load-balancer-internal: "true"
Google Cloud	GKE: Internal Load Balancing	cloud.google.com/load-balancer-type: "Internal"

As an example, using Amazon EKS as the provider.

1. Create a Helm chart configuration values file <MY-CONFIG-VALUES>.yaml file:

   # <MY-CONFIG-VALUES>.yaml
   alpha:
     service:
       type: LoadBalancer
       annotations:
         service.beta.kubernetes.io/aws-load-balancer-internal: "true"
   ratel:
     service:
       type: LoadBalancer
       annotations:
         service.beta.kubernetes.io/aws-load-balancer-internal: "true"
   

2. To expose Alpha and Ratel services privately, use:

   helm install <RELEASE-NAME> dgraph/dgraph --values <MY-CONFIG-VALUES>.yaml
   

Upgrading the Helm chart

You can update your cluster configuration by updating the configuration of the Helm chart. Dgraph is a stateful database that requires some attention on upgrading the configuration carefully to update your cluster to your desired configuration.

In general, you can use helm upgrade to update the configuration values of the cluster. Depending on your change, you may need to upgrade the configuration in multiple steps.

To upgrade to an HA cluster setup:

1. Ensure that the shard replication setting is more than one and zero.shardReplicaCount. For example, set the shard replica flag on the Zero node group to 3,zero.shardReplicaCount=3.

2. Run the Helm upgrade command to restart the Zero node group:

   helm upgrade <RELEASE-NAME> dgraph/dgraph [options]
   

3. Set the Alpha replica count flag. For example: alpha.replicaCount=3.

4. Run the Helm upgrade command again:

   helm upgrade <RELEASE-NAME> dgraph/dgraph [options]