# Home Cloud Lab My own private cloud..... The creation of my home lab signifies a focused pursuit of expertise across diverse IT domains and emerging technologies. This dedicated space serves several key purposes: 1. Exploring and mastering mini to medium-sized data center setups, emphasizing practical experience in managing complex infrastructures. 2. Constructing a flexible, portable data center using Ansible and Terraform's Infrastructure as Code principles for automated and scalable infrastructure deployment. 3. Crafting agile Kubernetes clusters adaptable to various environments, enhancing skills in deployment and management within dynamic architectures. 4. Upskilling in critical disciplines such as DevOps, DevSecOps, and System Design within a practical environment. 5. Providing a pre-Cloud development environment for testing workloads and optimizing configurations before cloud migration. 6. Operating as a secure Private Cloud, enabling controlled experimentation and innovation without compromising data privacy. In essence, my home lab serves as an agile learning ground, facilitating hands-on exploration and preparation for real-world IT challenges. Main Host - home-cloud01.lab ## Server Specifications

The Specs

Dell Precision T3600

# Architecture This section describes the infrastructure architecture of the lab

All the physical servers located in my small lab room...

# DNS Server I'm using [DNSMasq](https://thekelleys.org.uk/dnsmasq/doc.html) to manage my home DNS. # Kubernetes # Fundamental
In this lab, you will have a mix mode of learning contents. The lab contains theoritical and practical hand-on kind of contents. # Taints in Kubernetes In Kubernetes, a "taint" is a property applied to a node that marks the node as having certain constraints or requirements. Taints are used to repel or restrict the scheduling of pods onto specific nodes, making it possible to influence how pods are placed within a cluster. Taints are typically applied to nodes to indicate that they have special characteristics or should be used for specific workloads. Built-in Taints: * `node.kubernetes.io/not-ready`: Node is not ready. This corresponds to the NodeCondition `Ready` being "`False`". * `node.kubernetes.io/unreachable`: Node is unreachable from the node controller. This corresponds to the NodeCondition `Ready` being "`Unknown`". * `node.kubernetes.io/memory-pressure`: Node has memory pressure. * `node.kubernetes.io/disk-pressure`: Node has disk pressure. * `node.kubernetes.io/pid-pressure`: Node has PID pressure. * `node.kubernetes.io/network-unavailable`: Node's network is unavailable. * `node.kubernetes.io/unschedulable`: Node is unschedulable. * `node.cloudprovider.kubernetes.io/uninitialized`: When the kubelet is started with "external" cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint. # Kubernetes Observability (Monitoring) Guide

Kubernetes Cluster Monitoring

In Kubernetes operations, there are several key aspects that you should monitor to ensure the health, performance, and availability of your cluster and applications. Here are some important areas to consider: 1. Cluster health: Monitor the overall health of your Kubernetes cluster, including the status and availability of master and worker nodes. Keep an eye on metrics such as CPU and memory utilization, disk space, and network connectivity. 2. Node performance: Monitor individual worker nodes for resource utilization, including CPU, memory, and disk usage. This helps you identify potential bottlenecks or capacity issues on specific nodes. 3. Pod status: Check the status of your application pods to ensure they are running as expected. Monitor for pod failures, restarts, and termination events. Also, keep an eye on pod conditions such as readiness and liveness. 4. Container logs: Monitor the logs generated by your containers running within the pods. Logs provide valuable insights into the behavior of your applications and can help troubleshoot issues. 5. Resource utilization: Track resource usage at the pod and container level. Monitor CPU and memory utilization to ensure that your applications have enough resources to operate effectively. 6. Networking: Monitor network traffic and connectivity within the cluster. Keep an eye on network latency, packet loss, and throughput. Additionally, monitor load balancers, ingress controllers, and service endpoints to ensure proper routing and connectivity. 7. Application metrics: Monitor application-specific metrics such as response times, request rates, error rates, and throughput. This helps you understand the performance and behavior of your applications running on Kubernetes. 8. Scalability and autoscaling: Monitor the performance and effectiveness of your autoscaling configurations. Keep track of metrics that trigger scaling events, such as CPU or memory utilization thresholds. 9. Persistent volumes: Monitor the health and capacity of your persistent volumes (PVs) and persistent volume claims (PVCs). Ensure that storage resources are available and functioning correctly. 10. Security and compliance: Implement monitoring for security-related events, such as unauthorized access attempts, security policy violations, or abnormal behavior that may indicate a security breach. There are several monitoring tools available for Kubernetes, such as Prometheus, Grafana, Elasticsearch, and Datadog. These tools can help you collect, visualize, and alert on the relevant metrics and events in your Kubernetes environment. # DNS in Kubernetes (CoreDNS) This article will explain how the DNS works in K8s cluster

How CoreDNS query works

Kubernetes DNS (Domain Name System) provides service discovery and name resolution capabilities within a Kubernetes cluster. It allows applications and services running on the cluster to communicate with each other using their respective names instead of hardcoding IP addresses, which makes the cluster more scalable, maintainable, and dynamic. Here's a step-by-step explanation of how Kubernetes DNS works: 1. Pod and Service Creation: When you create a Pod or a Service in Kubernetes, each of them gets assigned a unique DNS name. 2. DNS Naming Format: * Pods: The DNS name of a Pod takes the form: `..pod.cluster.local` * Services: The DNS name of a Service takes the form: `..svc.cluster.local` Here, `` is the name of the Pod, `` is the namespace where the Pod or Service is located, and `cluster.local` is the default DNS suffix for the cluster. 3. DNS Resolution for Pods: When one Pod needs to communicate with another Pod within the same cluster, it can do so using the Pod's DNS name. The DNS resolution process starts with the Pod querying the Kubernetes DNS service (typically running on `kube-dns` or `coredns` pods). The DNS service has access to the cluster's Service and Pod IP addresses. 4. DNS Resolution for Services: When a Pod needs to communicate with a Service, it can use the Service's DNS name. Kubernetes DNS resolves the Service DNS name to the corresponding set of IP addresses of the Service endpoints. These endpoints are typically the IP addresses of the Pods backing the Service. 5. Load Balancing: Kubernetes DNS provides a level of load balancing for Services. When multiple Pods back a Service, the DNS resolution returns multiple IP addresses. The client (calling Pod) can use one of these IP addresses to communicate with the Service. Kubernetes handles load balancing across the selected Pods. 6. Automatic Updates: Kubernetes DNS automatically updates the DNS records whenever Pods or Services are created, deleted, or scaled. This ensures that the DNS system stays in sync with the current state of the cluster. 7. Custom DNS Configurations: Kubernetes allows you to configure custom DNS options for your cluster. For example, you can set up additional DNS nameservers or search domains to resolve external DNS queries or utilize other DNS services. In summary, Kubernetes DNS simplifies the process of service discovery and communication between Pods and Services within a Kubernetes cluster by providing a dynamic and scalable naming system. This abstraction helps maintain the decoupling between services and enables smooth scaling and management of applications within the cluster. # Storage in Kubernetes
In Kubernetes, storage is a crucial aspect of managing and persisting data for containerized applications. Kubernetes provides a framework for abstracting and managing storage resources, allowing containers to interact with storage volumes in a consistent and scalable manner. Here's an overview of storage in Kubernetes: 1. **Storage Resources**: * **Volume**: A Kubernetes Volume is an abstraction of a storage backend, such as a physical disk, network-attached storage (NAS), or cloud-based storage. Volumes are used to persist data beyond the lifecycle of a container. * **Persistent Volume (PV)**: A Persistent Volume is a cluster-level resource that represents a physical storage volume in the cluster. It abstracts the underlying storage details and allows administrators to manage and allocate storage resources. * **Persistent Volume Claim (PVC)**: A Persistent Volume Claim is a request made by a user or application for a specific amount and access mode of storage. PVCs are bound to PVs, allowing users to consume storage resources without needing to know the underlying details. 2. **Storage Classes**: * A Storage Class is a Kubernetes resource that defines the properties and provisioning mechanisms of PVs. It abstracts storage backend-specific details, making it easier to manage storage in a cluster. * Storage Classes allow administrators to set storage policies, such as replication, performance characteristics, and access modes, to match the requirements of different applications. 3. **Dynamic Provisioning**: * Kubernetes supports dynamic provisioning, which means that when a PVC is created, it can automatically trigger the creation of a PV with the requested storage class. * Dynamic provisioning enables on-demand allocation of storage resources, making it more efficient and scalable. 4. **Access Modes**: * PVs and PVCs can have different access modes: * **ReadWriteOnce (RWO)**: Allows read-write access to a single node (e.g., for a single-node database). * **ReadOnlyMany (ROX)**: Allows read-only access from multiple nodes (e.g., for shared configuration data). * **ReadWriteMany (RWX)**: Allows read-write access from multiple nodes (e.g., for shared file storage). 5. **Volume Types**: * Kubernetes supports various types of volumes, including: * **EmptyDir**: A volume with a lifetime tied to a pod, suitable for ephemeral data. * **HostPath**: A volume that mounts a file or directory from the host node into the pod. * **NFS, iSCSI, AWS EBS, Azure Disk**: Support for cloud-specific or network-attached storage backends. 6. **Stateful Applications**: * Kubernetes allows you to run stateful applications, such as databases, by providing mechanisms like StatefulSets and persistent storage. StatefulSets ensure that each pod has a unique network identity and stable storage. 7. **Snapshot and Clone**: * Kubernetes introduced features like Volume Snapshots and Volume Cloning to enable data management tasks, such as creating backups and cloning volumes. 8. **CSI (Container Storage Interface)**: * Kubernetes uses the CSI standard to enable third-party storage providers to integrate with the Kubernetes ecosystem. CSI allows for the development of custom storage plugins. # Container Storage Interface - CSI The Container Storage Interface (CSI) is a standard that defines a set of specifications for container orchestrators like Kubernetes to interact with various storage systems in a pluggable and extensible manner. CSI enables Kubernetes to work with a wide range of storage solutions, including cloud-based storage, network-attached storage (NAS), and storage area networks (SANs), without needing to modify the core Kubernetes codebase for each storage system. Here's an explanation of CSI and its usage in Kubernetes: **1. CSI Components:** * **CSI Driver**: A CSI driver is a plugin that implements the CSI specification for a specific storage system. Each storage provider (e.g., AWS EBS, NFS, Ceph) can develop its own CSI driver. * **CSI Controller**: The CSI controller is responsible for creating, deleting, and managing storage volumes based on the driver's specifications. * **CSI Node Plugin**: The CSI node plugin runs on each Kubernetes node and is responsible for attaching, detaching, and mounting volumes to pods. **2. CSI and Kubernetes Integration:** * Kubernetes uses CSI drivers to extend its storage capabilities without modifying the Kubernetes core code. This separation allows for easier maintenance and the ability to add new storage providers without changing Kubernetes itself. * CSI drivers are installed and managed separately from the Kubernetes cluster. When a cluster administrator deploys a CSI driver, it becomes available for use by the cluster's users and applications. **3. Usage of CSI in Kubernetes:** * **1. Deploying CSI Drivers**: Cluster administrators typically deploy CSI drivers in the cluster as DaemonSets or StatefulSets, ensuring that the CSI node plugin is available on every node where pods may run. * **2. Creating a Persistent Volume (PV)**: The first step in using CSI is to create a Persistent Volume (PV) that represents the storage resource provided by the CSI driver. This is done by creating a YAML definition for the PV with the appropriate CSI driver and parameters. * **3. Creating a Persistent Volume Claim (PVC)**: Users or applications create a Persistent Volume Claim (PVC) to request storage resources. The PVC specifies the access mode and size requirements. Kubernetes then matches the PVC to an available PV, based on the storage class and other criteria. * **4. Using the PVC in Pods**: In a Pod's YAML definition, you can specify the PVC as a volume source. When the Pod is scheduled and started, the CSI node plugin on the node attaches the appropriate PV to the pod as a volume, making the storage available to the container. * **5. Data Management**: Kubernetes users can manage data by creating snapshots of PVCs and cloning volumes. This is particularly useful for creating backups and managing application state. * **6. Scaling and Upgrading**: As your application scales and evolves, you can continue to use the same CSI driver while making necessary updates to the driver or CSI version. This decoupling of storage and Kubernetes allows for flexibility and seamless upgrades. **4. Benefits of CSI:** * **Pluggability**: CSI allows for easy integration of various storage solutions, making it possible to choose the best storage system for your specific use case. * **Flexibility**: Users can change storage providers or drivers without modifying application code or Kubernetes configurations. * **Community Support**: The CSI specification is well-supported and has a growing ecosystem of drivers maintained by both storage vendors and the Kubernetes community. # Ephemeral Storage Explain Ephemeral storage refers to temporary, non-persistent storage that is typically associated with the runtime or operation of a process or application but is not intended to persist data beyond the immediate task or session. Ephemeral storage is often used for short-term storage needs and is not meant for long-term data retention. Here are some key characteristics and use cases for ephemeral storage: 1. **Temporary Nature**: Ephemeral storage is transient by design. It is not intended to retain data once the process or application that uses it has completed or terminated. This makes it suitable for storing data that is only needed temporarily. 2. **Fast Access**: Ephemeral storage is often implemented using fast storage media, such as RAM or local disk storage on a computing device. This ensures quick access to data for the running process. 3. **Application Runtime Data**: Ephemeral storage is commonly used for storing runtime data needed by an application while it is running. For example, a web server might use ephemeral storage to store session data or cache data for improved performance. 4. **Containers and Pods**: In the context of containerized applications and Kubernetes, ephemeral storage can refer to temporary storage space allocated to a container or pod for storing runtime data. When the container or pod terminates, this storage is typically deleted. 5. **Caching**: Ephemeral storage is often used for caching frequently accessed data to reduce the need to retrieve data from slower, more persistent storage systems, such as databases or network storage. 6. **Logs and Temporary Files**: Applications may use ephemeral storage to write log files or store temporary files generated during their execution. These files are often cleaned up or deleted after their usefulness has expired. 7. **Stateless Operations**: Ephemeral storage is commonly used in stateless computing models, where applications do not rely on maintaining state between sessions or requests. Statelessness simplifies scaling and maintenance. 8. **No Data Durability Guarantees**: Ephemeral storage typically does not come with data durability guarantees. Data stored in ephemeral storage may be lost in the event of a system crash, power failure, or container termination. 9. **Resource Cleanup**: It is important to manage and clean up ephemeral storage resources properly to prevent resource leaks and ensure efficient use of available storage capacity. # Cluster CPU & Memory Inspections ## Quickly get the summary of total CPU ```bash $ kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.cpu}{"\n"}{end}' | awk '{sum += $2} END {print "Total CPUs across all nodes: " sum}' ``` ## Quickly get the summary of total Memory ```bash $ kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.memory}{"\n"}{end}' | awk '{print $1 "\t" $2 / (1024 * 1024) " MB"}' ``` ## Inspect Current Utilization Ensure that Metrics Server is installed in your Kubernetes cluster. If it's installed, you can query it to get CPU and memory usage information. 
$ kubectl top nodes
Run below command to convert from Mebibytes (Mi) to Megabytes (MB) ```bash $ kubectl top nodes | awk '{print $1, $2, $3, $4, ($5/1024/1024)"MB", $6}' ``` ## Using Kubernetes Dashboard If you have Kubernetes Dashboard installed, you can also use it to view resource usage across your cluster.

Example Nodes metrics from Home Lab K8s Cluster

# Open Source and Free Storage Solution Kubernetes ## Rook Rook turns distributed storage systems into self-managing, self-scaling, and self-healing storage services for Kubernetes. It supports various storage backends like Ceph, EdgeFS, and NFS. ## Longhorn Longhorn is an open-source distributed block storage system for Kubernetes. It is designed to be easy to deploy and manage, providing features like snapshots, backups, volume replication, and dynamic volume provisioning. ## OpenEBS OpenEBS is an open-source storage platform for Kubernetes that provides container-attached storage using local disks on Kubernetes nodes. It offers features like data replication, snapshots, backup and recovery, and multi-cloud support. ## MinIO MinIO is a high-performance, distributed object storage server that is compatible with the Amazon S3 API. It can be deployed on Kubernetes to provide scalable and resilient object storage. ## GlusterFS GlusterFS is a distributed file system that can be used as persistent storage for Kubernetes. It provides features like replication, distributed volumes, and automatic failover. ## Ceph Ceph is a distributed storage system that provides object, block, and file storage. It can be integrated with Kubernetes using Rook to provide scalable and reliable storage for containerized applications. ## NFS Network File System (NFS) is a distributed file system protocol that allows you to mount remote directories on your Kubernetes pods. While not as feature-rich as some other solutions, NFS can be a simple and effective way to provide shared storage for Kubernetes applications. # PoC Lab Po # Configure Access to Multiple Cluster using KUBE\_CONFIG Simple way on making access to multiple k8s cluster ## **Scenario** We have multiple K8s cluster: 1. A cluster in an on-premise environment (Microk8S) 2. A cluster on the Cloud environment (AWS) 3. A cluster on the Cloud environment (Digital Ocean) You may create multiple config files under the `~/.kube/` directory. Example: ```bash ┌─[admin@my-dev-svr] - [~/.kube] - [Mon May 22, 21:50] └─[$]> tree . ├── aws-cluster-01 ├── do-k8s-01-kubeconfig └── microk8s@home-cloud01.lab 0 directories, 3 files ┌─[admin@my-dev-svr] - [~/.kube] - [Mon May 22, 21:50] └─[$]> ``` As you can see on the example above, `aws-cluster-01` is the config files for cluster that hosted on AWS, `do-k8s-01-kubeconfig` is for the cluster access on the Digital Ocean, while the last one `microk8s@home-cloud01.lab` is MicroK8s cluster that reside on the on-premise environment. Then, you have to modify `.bash_profile` or `.zshrc` to combine all the access config into one liner KUBE\_CONFIG environment variable like the following: ```bash #On Prem K8s export CONFIG_MICROK8S=$HOME/.kube/microk8s@home-cloud01.lab #Cloud K8s export CONFIG_DO=$HOME/.kube/do-k8s-01-kubeconfig export CONFIG_AWS=$HOME/.kube/aws-cluster-01 #Merge all the configs export KUBECONFIG=$CONFIG_MICROK8S:$CONFIG_DO:$CONFIG_AWS ``` Finally, test the connectivity.
Gist Source: # High Availability Portable Mini Kubernetes Data Center with K3s # Lab 1: Build the K3s Infrastructure ## Nodes Inventory The following are the details of my server nodes The following are the details of my server nodes
No.TypeName ConventionsOperating SystemPurpose
1Pyshical (Raspberry Pi Model B)k3s-master-node01.home-cloud01.labUbuntu Server 23.04Kubernetes Master Node
2Virtual Machinek3s-master-node02.home-cloud01.labUbuntu Server 22.04 LTSKubernetes Master Node
3Virtual Machinek3s-master-node03.home-cloud01.labUbuntu Server 22.04 LTSKubernetes Master Node
4Virtual Machinek3s-worker-node01.home-cloud01.labUbuntu Server 18.04.6 LTSKubernetes Worker Node
5Virtual Machinek3s-worker-node02.home-cloud01.labUbuntu Server 22.04 LTSKubernetes Worker Node
6Virtual Machinek3s-worker-node03.home-cloud01.labUbuntu Server 22.04 LTSKubernetes Worker Node
### # Lab 2: Automate the K3s clusters deployment with Ansible Starting a local Kubernetes cluster in our home lab environment may become complicated if we use kubeadm or any other distributions. I seek to establish a Kubernetes cluster that operates seamlessly within an Ansible-driven ecosystem, allowing for scripted control over the cluster's nodes. This setup enables the automation of both the construction and decommissioning of nodes within the cluster, providing the flexibility to manage its lifecycle as required. Therefore, I chose `k3s-ansible` to build my on-premise Kubernetes cluster. Continue later... ## Post-Deployment After the deployment, you may need to get a copy of kube config file from each of the master nodes. Run the following commands ``` $ scp k3s-master-node01:~/.kube/config ~/.kube/k3s-master-node01 ``` Please replace k3s-master-node01 with the hostname or IP addresses of your master node. You may need to repeat the command if you have more than one master node. Please also replace the following parameters in each of the kube config file, so that when you run `$ kubectl config get-context` you will see the name of the nodes correctly. Example config file: ``` apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pP server: https://192.168.2.1:6443 name: k3s-master-node01 //change here contexts: - context: cluster: k3s-master-node01 //change here user: default //remain here. we are gonna use default user name: k3s-master-node01 //change here current-context: k3s-master-node01 //change here kind: Config preferences: {} users: - name: default //remain here user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRlZ0F3SUJBZ0lJTFBqR0ZBcndTUlF3 ``` Then, test the connectivity with the following command ``` $ kubectl get nodes ``` My results:
# Lab 3: Multi Master K3s kubeconfig files for smooth accessibility Scenario: We have multiple master nodes and we want to ensure that our operation using `kubectl` in our local machine is smooth and uninterrupted. First, create the necessary kubeconfig files and add the configuration data into it like the following:

k3s-master-node0* is the file that contains the configuration data required to access the Kubernetes clusters

Then we modify our shell profile. Depending on your setup, as for me, I'm using [zsh](https://ohmyz.sh/) as my shell. Therefore, I will modify the `.zshrc` file as below: 
#My Onprem K8s master nodes
export CONFIG_K3S_MASTER_NODE01=$HOME/.kube/k3s-master-node01
export CONFIG_K3S_MASTER_NODE02=$HOME/.kube/k3s-master-node02
export CONFIG_K3S_MASTER_NODE03=$HOME/.kube/k3s-master-node03

#Merge all the configs
export KUBECONFIG=$CONFIG_K3S_MASTER_NODE01:$CONFIG_K3S_MASTER_NODE02:$CONFIG_K3S_MASTER_NODE03:$CONFIG_MINIKUBE
Finally, you may get the results just like below:

Notice there were multiple context displayed for my master nodes

# Lab 4: Network Configurations & Tools #### Network Router I'm using Mercusys AX1800 Dual-Band Wi-Fi 6 Router #### DNS Server I have a DNS server that running on a VM's with DNSMasq installed. It serves as a DNS forwarder,caching and as a DHCP server. Following is the configs. #### DNSMasq Configs ``` domain-needed bogus-priv no-resolv server=8.8.8.8 server=8.8.4.4 local=/home-cloud01.lab/ listen-address=::1,127.0.0.1,192.168.1.3 expand-hosts domain=home-cloud01.lab dhcp-range=192.168.1.4,192.168.1.255,24h dhcp-range=192.168.2.1,192.168.2.255,24h dhcp-option=option:router,192.168.1.1 dhcp-authoritative dhcp-leasefile=/var/lib/dnsmasq/dnsmasq.leases ``` As we can see, I set the listener address to `192.168.1.3` which means this is the endpoint I can use as my secondary DNS setting on any of my devices - PC's, laptops,mobile phone, smart home appliances etc. The root domain is `home-cloud01.lab`. At the time of writing this guide, I have multiple IP segmentations: * `192.168.1.4,192.168.1.255` within this range, I use it for Pyshical and VM's based appliances. * `192.168.2.1,192.168.2.255` within this range, I use it for the Kubernetes cluster #### Deploy and Configure Nginx Ingress Controller # AWX/Ansible Tower Deployment on Kubernetes This lab tutorial shall be use in development/testing/lab/home lab environment. # Lab 1 - Understanding Requirements ### **Lab Objective** In this lab we will get the basic understanding of Ansible & Ansible Tower on how it will benefits your organization for infrastructure automations and completing pre-requisites steps for the AWX deployment. ### **What is Ansible?** Ansible is an open source community project sponsored by Red Hat, it's the simplest way to automate IT. Ansible is the only automation language that can be used across entire IT teams from systems and network administrators to developers and managers. ### **What is Ansible Tower?** Ansible Tower is the enterprise version of Ansible. It allows sysadmins to deploy all of the benefits of Ansible at scale. In a nutshell, Ansible Tower will provides user a Web GUI console that enable users to access the system from any web browser as well as RESTful API's to be used by developers. ### **What is AWX?** The AWX project is an open source community project, sponsored by Red Hat, that enables users to better control their community Ansible project use in IT environments. AWX is the upstream project from which the automation controller component is ultimately derived. ### **What is AWX Operator?** ### **Pre-requisite Steps** * [Install ](broken://pages/lD0ryqQzIumZmSQKW9gk)Kubernetes * [Install Kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize/) * Login to quay.io via CLI for local Docker image * K8s Dashboard ### **Hardware Requirements** By default, AWX/Ansible Tower required below minimum specs if you wanted to host it either on physical or virtual server. * `At least 4GB of memory` * `At least 2 CPU cores` * `At least 20GB of space` For the entire lab series, we will use the latest version of AWX. However, you may change the version to whatever version that you may required later. We are also going to use AWX Operator to manage our deployment. Usage of AWX operator has been recommended by the community as it is the quick and efficient way to get up and running of the AWX. At the time of writing this lab,version 1.1.1 is the latest stable version of AWX operator. **Kubernetes Namespace** We are going to use below namespace in this entire lab series. * Namespace name: awx *\~End of Lab 1 - Understanding Requirements\~* [*Continue to the next Lab 2 - AWX,AWX Operator Configurations & Deployments*](/devops-and-devsecops/kubernetes/poc-lab/awx-ansible-tower-deployment-on-kubernetes/lab-2-awx-awx-operator-configurations-and-deployments) # Lab 2 - AWX,AWX Operator Configurations & Deployments ### **Lab Objective** In this lab we will learns how to deploy & configure AWX as a containers to K8s cluster. ### Deployment Steps #### **Step 1: Prepare YML files for configurations.** Basically, only three YML's file required to deploy the AWX onto our K8s cluster.

Note: The YML files

create `kustomization.yml` file with the following content: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: # Find the latest tag here: https://github.com/ansible/awx-operator/releases - github.com/ansible/awx-operator/config/default?ref=1.1.1 #change to any preference version # Set the image tags to match the git version from above images: - name: quay.io/ansible/awx-operator newTag: 1.1.1 #change to any preference version # Specify a custom namespace in which to install AWX namespace: awx ``` as you can see the version of AWX Operator is defined on `newTag` key as well as on the resources key. create `awx-demo.yml` file with the following content: ```yaml apiVersion: awx.ansible.com/v1beta1 kind: AWX namespace: awx metadata: name: awx-demo spec: service_type: nodeport # default nodeport_port is 30080 nodeport_port: 30000 #you may change this node port between 30000-32768 ``` Here you will see minimal content definitions specified in the YML because AWX Operator has taken care the actual AWX deployment files. #### **Step 2: Deploy the AWX Operator controller manager.** Now, with all the k8s object config files in hand, we are going to execute it one by one, Execute the build in kustomization.ym**l** ```bash $ kubectl apply -k . ```

Objects that has been created

Once executed, AWX operator will deploy the following objects on k8s.

Deployment object

Pod object

Replica Sets object

Services object

ConfigMap object

Ensure that all AWX controllers manager objects are up and running,
#### Step 3: Deploy the AWX. Now, this is where we want to execute our AWX deployment. Run`$ kubectl apply -f awx-demo.yml` After the execution, we can see below additional objects has been created

awx-demo and awx-demo-postgres pods created

Notes, for the pods to get ready, it might take sometimes so we will just have to wait for a while.

awx-demo-postgres-13 stateful sets created

awx-demo-service & awx-demo-postgres-13 services has been added

awx-demo-awx-configmap created

the PVC for AWX

Finally, ensure that all the required secrets has also been created during the deployment.

All the secrets

**Step 4: Verify all the k8s object resources & connectivity to the service** At this stage, we will need to verify below resources has been created inside the `awx` namespace. Below screenshots is from my end for your references.
**Step 5: Verify connectivity from the Service** Now we have everythings up and running, we need to do final check and it can be done by the AWX API check. **Run** `$ minikube service list`

Exposed service in Minikube

Referring to above image, this is the URL that we will need to use to query the AWX Ping API's `AWX Service URL: http://192.168.39.154:30000` To query the API, we can run `$ curl -L -q http://192.168.39.154:30000/api/v2/ping 2>/dev/null` or `$ curl -L -q $(minikube service awx-demo-service --url -n awx)/api/v2/ping 2>/dev/null` We shall receives below response if everythings is good.

Response from the Ping API's

*\~End of Lab 2 - AWX,AWX Operator Configurations & Deployments\~* [*Continue to Lab 3 - Ingress Networking & Connectivity via Nginx*](/devops-and-devsecops/kubernetes/poc-lab/awx-ansible-tower-deployment-on-kubernetes/lab-3-ingress-networking-and-connectivity-via-nginx) # Lab 3 - Ingress Networking & Connectivity via Nginx ### **Lab Objective** In this lab we will establish Ingress networking via Nginx Ingress Controller so we are able to access the AWX from the FQDN address. ### Deployment Steps **Step 1: Create Nginx Ingress** Create `awx-nginx-ingress.yml`file with the following content ````yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: awx-ingress namespace: awx spec: ingressClassName: nginx rules: - host: tower.home-cloud01.lab http: paths: - path: / pathType: Prefix backend: service: name: awx-instance-serivce port: number: 30000 ``` ```` **Step 2: Ensure Minikube has Nginx Ingress Classes** `$ minikube addons enable ingress`

Once enable, we shall see nginx available in Ingress Classes

**Step 3: Deploy Ingress Nginx.** **Run below command,** $ kubectl apply -f awx-nginx-ingress.yml Above command will create below Ingress object

awx-nginx-ingress object created

To see more details about our HTTP/HTTPS network routing, we shall be able to describe the object by running the following command. `$ kubectl describe ingress-nginx`

More details can be seen when we describe the object

As you can see on the above image, my specified subdomain tower.home-cloud01.lab was routed to the backend service - awx-nginx-service (the AWX service) on port 80. #### Step 4: Modify hosts file We will need to modify hosts file in our host server. `$ kubectl get ingress awx-nginx-ingress -n awx | grep -v NAME | awk -F' ' '{print $4 " " $3 }' | sudo tee -a /etc/hosts` Verify the host has been added into the `/etc/hosts` file

my /etc/hosts file

**Step 5: Verify the connectivity.** So now we have the Ingress setup for our network routes. Let's test the connection. `$ curl -L -q http://tower.home-cloud01.lab/api/v2/ping 2>/dev/null` If you get the same response as per the following image, then you are in a good conditions.

Ping response via the Ping API's

**Congratulations!** Now you have the AWX up and runninng in your Kubernetes cluster. Next we will explore on the Day 2 operations where we will access the AWX from the Web console. *\~End of Lab 3 -* Ingress Networking & Connectivity via Nginx*\~* [*Continue to Lab 4: AWX Web Console & accessibility on LAN*](/devops-and-devsecops/kubernetes/poc-lab/awx-ansible-tower-deployment-on-kubernetes/lab-4-awx-web-console-and-accessibility-on-lan) # Lab 4: AWX Web Console & accessibility on LAN

The Diagram

# Troubleshooting: Ansible Tower - Reset admin password ## Option 1 - Reset Password 1. Access the awx-web Pod 2. Use the awx-manage cli to change the password for the specific user ``` $ awx-manage changepassword admin #change password for user admin ``` ## Option 2 - Add Password Get the awx-web Pod's name 1. Access the awx-web Pod 2. Use the awx-manage cli to add new superuser ``` $awx-manage createsuperuser ``` 3. Login with the new superuser and change the previous user password # Monitoring & Observability # Splunk # Splunk Deployment for SME (Malay Version) #### 1. Keterangan Solusi Untuk *POC* ini, kita akan menggunakan Vagrant & Virtualbox untuk VM server.
#### 2. Architecture Diagram ![](https://docs.splunk.com/images/b/b9/Small_enterprise_deployment.png) **2.1 Butiran Server** | Peranan | Jumlah Server | | ----------- | :-----------: | | Search Head | 1 | | Indexer | 2 | | Forwarder | 1 | **2.2 Spesifikasi Server** ``` searchead1 Operating System: Centos 7 CPU Core: 2 Memory: 2GB Private IP Address: 192.168.20.21 indexer01 Operating System: Centos 7 CPU Core: 2 Memory: 2GB Private IP Address: 192.168.20.22 indexer02 Operating System: Centos 7 CPU Core: 2 Memory: 2GB Private IP Address: 192.168.20.23 forwarder01 Operating System: Centos 7 CPU Core: 2 Memory: 2GB Private IP Address: 192.168.20.24 ``` #### 3. Langkah-langkah Penyediaan Server Sila clone repo dibawah ini untuk mendapatkan fail skrip Vagrant. Skrip ini akan membantu untuk menjalankan pemasangan Splunk Enterprise secara auto ke empat-empat server. **Nota!** **Sila ikuti langkah-langkah seperti didalam fail *****README.md***** untuk pemasangan & konfigurasi server.** Repository URL : #### 4. Langkah-langkah Konfigurasi Splunk **4.1 Pengaktifan SSL**
Log masuk ke dalam Splunk Web di . Pergi ke **Settings > Server Settings > General Settings**. Pada bahagian **Splunk Web**, pergi ke pilihan **Enable SSL (HTTPS) in Splunk Web?** dan pilih **Yes**.\ \ Ulang semula proses yang sama untuk kesemua Splunk server dan restart Splunk setelah selesai. **4.2 Menambah *****Search Peer***** pada *****Search Head***
Log masuk ke dalam Splunk Web di . Pergi ke **Settings > Distributed Search** dan klik pada butang **+ Add new**. Sila masukkan butiran ini pada paparan **Add search peers** \ \`\`\` Peer URI = // alamat IP indexer01 Remote username = admin // nama administrator yang anda cipta sebentar tadi Remote password = Confirm password = \`\`\` ![Gambar 1](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Screenshot2020-11-10at2.32.48AM.png) **4.3 Menambah *****Receiver Port***** pada Indexer01 & Indexer02 Splunk server.** \ Log masuk ke dalam Splunk Web pada \[[https://192.168.20.22:8000\]\\(https://192.168.20.22:8000)](https://learn.farizizwan.com/devops-and-devsecops/monitoring-and-observability/splunk/https:/192.168.20.22:8000]\\\(https:/192.168.20.22:8000\)). Pergi ke \*\*Settings > Forwarding and receiving\*\* ,pada seksyen \`Configure Receiving\` klik \`+ Add new\` link.\ \ Sila isi port \`9997\` pada kotak \*\*Listen on this port\*\*. ![Gambar 2](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Screenshot2020-11-10at2.58.27AM.png) **4.4 Pengkaktifan Aplikasi Splunk Forwarder** Log masuk ke dalam Splunk Web di . Pergi ke **Apps > Manage Apps** . Cari **SplunkForwarder** pada ruangan **filter**. Klik **Enable**. Setelah selesai, sila *restart* Splunk *forwarder01*. **4.5 Menambah *****Receiver Host***** pada Forwarder server.** Log masuk ke dalam Splunk Web di . Pergi ke **Settings > Forwarding and receiving** ,pada seksyen **Configure Forwarding** klik butang **+ Add new**.\ \ Sila isi alamat IP `indexer01` (192.168.20.24:9997) pada kotak **Host**.\ \ Ulang semula proses yang sama untuk menambah `indexer02`. ![Gambar 3](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Screenshot2020-11-10at2.51.59AM.png) #### 5. Ujian Pengesahan Untuk ujian pengesahan bagi mengesahkan pelaksanaan ini adalah berjaya, kita akan membuat simulasi penghantaran data seperti berikut: ![Gambar 4](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/SplunkSmallEnterpriseDD\(1\).png) Gambar 4. Aliran data Splunk.\ \###### 5.1 Butiran Gambar 4. Aliran data Splunk : 1. Splunk Monitor akan mengambil data syslog dari local Linux system.\ 2\. Splunk Heavy Forwarder menghantar data kepada Splunk Indexer server.\ 3\. Apabila pencarian data berlaku pada Search Head ,ia secara automatik akan menjalankan \*query\* melalui TCP keatas Indexer. Dengan erti kata lain Search Head akan meminta data dari Indexer melalui sambungan TCP.
**5.2 Langkah-langkah Ujian** 1. Menambah data *Linux Syslog* menggunakan pilihan **File & Directories Monitoring**. Sila ikut gambar-gambar dibawah untuk rujukan. ![Gambar 5](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Collect1.png) ![Gambar 6](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Collect2.png) ![Gambar 7](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Collect3.png) ![Gambar 8](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Collect4.png) ![Gambar 9](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/Collectdone.png) 1. Membuat ujian penambahan data pada Linux Syslog. Jalankan *command* ini didalam `forwarder01` server. ``` $ logger Testing Message Logs ``` 1. Mengesahkan data telah berjaya ditambah secara *Realtime*. Log masuk ke dalam Splunk Web di **Search Head** . Pergi ke aplikasi *Search & Reporting* . Pada ruangan *Search* sila masukkan *query* berikut ``` index="forwardersyslog" ``` Jika anda mengikuti tutorial ini dengan tepat, anda akan dapat keputusan seperti gambar dibawah. ![Gambar 10](https://blog.farizizwan.com/bl-content/uploads/pages/9d2084e03af847ec13420e123591c3b2/lasttestdone.png) Merujuk gambar diatas, anda dapat lihat, *message logs* terakhir yang diterima adalah **"Testing Message Logs"** yang telah dimasukkan pada langkah 2.\ \ Sekian,\ Terima Kasih\ Fariz Izwan Kamaruzzaman\ \ Hubungi saya untuk sebarang masalah berkaitan artikel ini.\ Email:
Rujukan:\ # System Administration - Linux # RedHat Derivatives # Add User to Sudoers `Add user to sudo group via usermod` `$ su root` `$ usermod -aG wheel username` Add user to sudo group via visudo `$ su root` `$ visudo` and then insert following after the `%wheel line.` `username ALL=(ALL) NOPASSWD:ALL` do not forget to change the `username` to the user of yours.
# Yum Update vs Yum Upgrade "`yum upgrade"` and "`yum update"` will perform the same function that updates the packages to the latest version.. 1. "yum upgrade" updates the packages and forces the removal of obselete packages. 2. "yum update" update the packages without removing obselete packages. # DNF update vs upgrade "dnf update" vs "dnf upgrade" are no difference. "dnf update" is an alias of "dnf upgrade" # SMP PREEMPT\_DYNAMIC Definitions ## Summary “SMP” means that the kernel was built with support for SMP (symmetric multi-processing). “PREEMPT” means that the kernel was built as a [preemptible kernel](https://elixir.bootlin.com/linux/v5.7.1/source/kernel/Kconfig.preempt#L38). “RT”, on version 4.19, means that the kernel was built as a fully preemptible (real-time) kernel, using [the Preempt RT](https://wiki.linuxfoundation.org/realtime/start) patches. ## Long Definitions "SMP PREEMPT\_DYNAMIC" refers to a Linux kernel configuration option that enables the Symmetric Multiprocessing (SMP) support with dynamic preemption. Let's break down the components: 1. SMP (Symmetric Multiprocessing): SMP refers to a computer architecture that allows multiple processors or cores to work together on a single system. With SMP support enabled, the Linux kernel can effectively utilize multiple processors, distributing tasks among them for improved performance and concurrency. 2. PREEMPT: Preemption in the context of the Linux kernel refers to the ability to interrupt a running task before its completion and switch to another task. Preemption allows for better responsiveness, especially in scenarios where multiple tasks are competing for system resources. 3. DYNAMIC: The "PREEMPT\_DYNAMIC" option indicates that the Linux kernel can dynamically determine when to preempt a running task based on predefined criteria. This approach allows for more flexibility and fine-grained control over task preemption. Combining these elements, "SMP PREEMPT\_DYNAMIC" signifies a kernel configuration that enables support for symmetric multiprocessing, allowing multiple processors to work together, while also incorporating dynamic preemption for better responsiveness and resource utilization. # Virtualization # KVM **Verify KVM extension** ``` grep –E ‘svm|vmx’ /proc/cpuinfo ``` If you receive no output then it’s likely that virtualization extensions are not enabled in your BIOS and you will need to take steps to rectify this before moving forward with the KVM installation.
**How to check if KVM is enabled & active** Check if KVM module has been loaded `$ lsmod | grep kvm` Using [virt-host-validate](https://libvirt.org/manpages/virt-host-validate.html) `$ sudo virt-host-validate`\
# Storage # Solution & System Design # Public Cloud # Fundamental # AWS In this lab journey, I will share my AWS cloud infrastructure knowledge base on my experiences working with several organizations. How we host an Online Travel Agency portal on AWS How we host a Personal Financing comparison portal on AWS # Azure In this lab journey, I will share you I do setup my cloud infrastructure base on my experiences working with several organizations. How we host a mission critical Plant Operation system on Azure. # GCP # Lab # Simple CD menggunakan Github Action ## Belajar DevOps: Simple CD menggunakan Github Action Tutorial dibawah ini adalah cara saya mengemaskini halaman landing page farizizwan.com dengan mengautomasikan deployment source code terbaru menggunakan Github Action. Ia adalah simple CD pipeline dimana ia cuma akan melakukan action berikut: * Akses ke server farizizwan.com yang di host di AWS Lightsail menggunakan SSH protocol. * Melakukan `git pull` untuk mengemaskini source code. Script ditulis didalam YML format seperti dibawah ``` name: CICD on: [push,pull_request] jobs: push: runs-on: ubuntu-latest steps: ########################################## # # Update source code on farizizwan.com server # ########################################## - name: Git pull uses: fifsky/ssh-action@master with: command: | cd /home/public_html/farizizwan.com/ && git pull origin master host: ${{ secrets.HOST }} user: ${{ secrets.USER }} key: ${{ secrets.PRIVATE_KEY }} ``` Gist: **Contoh Deployment Script 1**
Untuk menetapkan / menambah **Action Secret** pergi ke `https://github.com//settings/secrets/actions` ![secret](https://blog.farizizwan.com/bl-content/uploads/pages/b4838d184a3097d8bdf39f5ddeef183c/Screenshot2021-03-03at1.38.35PM.png) #### Memahami Github Action Syntax pada YML script `on` = script secara automatik akan running apabila ia mengesan action `push` atau `pull request` seperti yang dinyatakan.\ `jobs` = ini adalah pipeline yang akan menjalankan kerja-kerja yang kita nyatakan.Ia akan running secara parallel atau sequential. contoh anda akan mempunyai kerja-kerja Unit Test didalam CI pipeline, anda boleh buat begini ``` Sequential jobs: unit_test: - name: Checkout Source Code - name: Running Unit Test Integration deploy_to_staging: - name: Checkout Source Code - name: Deploy to Staging Server deploy_to_production: - name: Checkout Source Code - name: Deploy to Production Server ``` ``` Parallel jobs: main_step: - name: Checkout Source Code - name: Running Unit Test Integration - name: Deploy to Staging - name: Deploy to Production ```
`jobs.runs-on`: Untuk menyatakan OS pilihan apa yang perlu menjalankan Github Action ini. Ia adalah seperti sebuah VM sementara atau runtime sementara yang akan menjalankan kesemua Jobs yang anda nyatakan dalam Github Action script. Jadual dibawah menunjukkan list OS dan mesin yang ditawarkan oleh Github. Anda juga boleh menggunakan server anda sendiri sebagai build runtime. Ia adalah seperti \[ DevOps Pipeline Agents).\ ![list\_agent](https://blog.farizizwan.com/bl-content/uploads/pages/b4838d184a3097d8bdf39f5ddeef183c/Screenshot2021-03-03at2.02.14PM.png)\ `steps`: Untuk menjalankan tugas-tugas dibawah setiap Jobs yang dicipta. Ia akan berjalan secara sequential.\ `steps.name` : Nama Step.\ `steps.uses`: Anda juga boleh menggunakan semula script-script yang ada didalam [Github Marketplace](https://github.com/marketplace?type=actions). Seperti contoh diatas saya menggunakan semula script daripada [`fifsky/ssh-action`](https://github.com/fifsky/ssh-action).\ `steps.with`: Nyatakan parameter yang anda perlu pass pada Step yang dijalankan. Seperti Contoh Deployment Script 1 diatas, saya perlu pass tiga parameter untuk step Git pull yang menggunakan [`fifsky/ssh-action`](https://github.com/fifsky/ssh-action) script iaitu `host`,`user`,`key`. `Key` adalah Private Key untuk kegunaan akses melalui SSH. Rujuk nota diatas cara untuk menambah secret ini. Manakala `command` pula adalah command yang anda mahu jalankan setelah berjaya akses ke deployment server. #### Ringkasan kelebihan menggunakan Github Action Saya menggunakan Github Action ini kerana repositori farizizwan.com berada di Github. Memandangkan spesifikasi server adalah minimum jadi saya berkeputusan untuk tidak membuang masa menggunakan Jenkins kerana untuk farizizwan.com hanyalah static HTML landing page dan memerlukan kerja-kerja integrasi yang minimum,justeru saya dapat jimat lebih banyak masa disini. Ini sahaja perkongsian kali ini, saya akan kongsikan bagaimana hendak menggunakan Azure DevOps Pipeline pula dimasa akan datang. Sekian,\ Terima Kasih\ Fariz Izwan Kamaruzzaman\ 3 March 2021 Rujukan:\ # Private Cloud & On-Premise # On-Premise to Cloud Design # AWS Direct Connect What do you need to know when you want to bridge your on-premise data center to AWS Cloud data center? For AWS they offered a Direct Connection service. This is the service that has two type of implementations. 1. Dedicated Direct Connect 2. Hosted Direct Connect **Dedicated Direct Connect** This is the type of Ethernet connection where physical Ethernet card are attached to your account and it will provide a dedicated network circuits within an AWS Direct Connect on-premise location. This network circuits will not be shared with any others AWS Partner or customer. Connectivity of port speed up to 100Gbps. **Hosted Direct Connect** This is the type of Ethernet connection where physical Ethernet card are attached to your account but the setup will be done by AWS Direct Connect Partner such as Equinix,Megaport and more. Connectivity of port speed up to 10Gbps only. # High Availability & Fault Tolerance Design Understanding HA & Fault Tolerance High availability and fault tolerance are two different approaches to ensure the continuous operation and reliability of a system, application, or service. While they both aim to minimize downtime and maintain seamless functionality, they achieve this in distinct ways. 1. **High Availability (HA)**: High availability focuses on keeping a system or service running without any interruptions or disruptions. The primary goal is to maximize uptime and provide uninterrupted access to users. HA achieves this by setting up redundant components, such as servers, networks, or databases, and distributing the workload across these redundant elements. If one component fails, the load is automatically transferred to the backup, so there is no noticeable impact on the end-users. Key points of High Availability: * Quick detection of failures and automatic failover to the redundant components. * Minimal or no downtime during failover processes. * Typically suitable for critical systems and applications that require continuous operation. 2. **Fault Tolerance**: Fault tolerance, on the other hand, focuses on the ability of a system to continue functioning correctly even in the presence of faults or failures. The emphasis here is on maintaining data integrity and service availability despite hardware or software failures. Fault tolerance involves redundant components and mechanisms that can detect and correct errors or failures to ensure smooth operation. Key points of Fault Tolerance: * Immediate detection of faults and immediate correction or recovery. * Continues operating even when a fault occurs, without interrupting service. * It's generally associated with systems that cannot afford even the briefest downtime, such as real-time applications or critical infrastructure. In summary, high availability aims to prevent downtime by quickly switching to redundant components, while fault tolerance emphasizes the ability to recover from faults without any interruption in service. Both approaches are crucial for ensuring reliable and continuous operations in different scenarios, and they are often used together to provide a comprehensive and robust system that can handle unexpected failures and maintain the desired service levels. # Disaster Recovery Understanding DR,RPO,RTO,RLO

Example AWS Disaster Recover Stretegy. Source: https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html

### Understanding Disaster Recovery Disaster recovery of a data center is a comprehensive and strategic approach to ensure the continuity and availability of critical digital information and services hosted within the data center, even in the face of unexpected disasters or disruptive events. It involves planning, implementing, and testing measures to recover data, applications, and systems swiftly and effectively. Here's a breakdown of the key components of disaster recovery for a data center: 1. **Risk Assessment and Planning**: The process begins with identifying potential risks and vulnerabilities that could affect the data center, such as natural disasters, hardware failures, cyber-attacks, or human errors. Based on these assessments, a disaster recovery plan is developed, outlining the actions to be taken in different disaster scenarios. 2. **Redundancy and Backups**: Critical data, applications, and configurations are duplicated and regularly backed up to secure storage systems, both within and outside the primary data center. Redundancy helps ensure that if one part of the data center fails, the backup systems can take over seamlessly. 3. **Disaster Recovery Sites**: Data centers often have secondary or remote locations, known as disaster recovery sites, where copies of the data and systems are maintained. These sites act as backup centers and can take over the operations if the primary data center becomes unavailable. 4. **Failover and Failback Procedures**: Failover is the process of switching from the primary data center to the disaster recovery site when a disaster occurs. Failback is the reverse process of returning to the primary data center once the issues are resolved. These procedures are designed to minimize downtime and data loss. 5. **High Availability and Load Balancing**: Technologies like load balancing and high availability architecture ensure that workloads are distributed across multiple servers and data centers. This way, if one server or data center becomes overloaded or fails, others can pick up the load. 6. **Data Replication and Synchronization**: Data replication tools keep copies of data synchronized between multiple data centers in real-time or with minimal delay. This ensures that the data at the disaster recovery site is up to date. 7. **Testing and Training**: Regular testing of the disaster recovery plan is crucial to identify and fix potential issues. Data center staff must be trained on the procedures and protocols for executing the recovery plan effectively. 8. **Continuous Monitoring**: The data center is continuously monitored to detect any signs of potential problems, allowing for quick action before a disaster occurs or worsens. Overall, disaster recovery of a data center is a critical aspect of business continuity planning, as it helps organizations maintain their operations, protect sensitive data, and minimize the impact of unexpected disruptions on their customers and stakeholders. ## Understanding the RTO,RPO,RLO

Source: https://www.buffalotech.com/blog-helpful-tips/reliable-business-continuity-at-an-entry-level-price

## RPO **RPO** stands for Recovery Point Objective, and it is a crucial metric used in disaster recovery and data protection planning. In simple terms, RPO represents the maximum acceptable amount of data loss that an organization is willing to tolerate in the event of a disaster or system failure. It answers the question: "How much data are you willing to lose?" ## **RTO** RTO stands for Recovery Time Objective, and it is a critical metric used in disaster recovery and business continuity planning. In simple terms, RTO represents the maximum allowable downtime for a system, application, or service following a disaster or disruption. It answers the question: "How quickly do you need to recover and restore normal operations?" ## RLO RLO stands for Recovery Level Objective. Recovery level objective is a metric that defines the minimum resources an organization needs in the aftermath of a disruptive event. Resources required to recover and resume business operations to an acceptable level typically include people, processes, technology and facilities

Example AWS Cross Region backup