So, configmap is solving the problem of storing the information that can be used by the application.

Secrets in Kubernetes also deals the same as the configmaps but it deals with sensitive information. Non-sensitive information is stored in configmaps and sensitive information is stored in secrets. The sensitive data of secret before stored in etcd, gets encrypted in Rest. Kubernetes does the default encryption but it also allows us to do the custom encryption.

Strong RBAC is enforced for secrets and the least privileged access is implemented for secrets.

Let's get directly into the hands-on experience.

Let's create the configmap file as cm.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-cm
data:
  db-port: "3306"

Apply

kubectl apply -f cm.yml

Our configmap is created.

We will take these data as environment variable in the kubernetes cluster.

We have the deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-python-app
  labels:
    app: sample-python-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-python-app
  template:
    metadata:
      labels:
        app: sample-python-app
    spec:
      containers:
      - name: python-app
        image: subash07/python-app:latest
        ports:
        - containerPort: 8000

Create the deployment

kubectl apply -f deployment.yml

We can see that there is no environment variable concerning db.

Let's modify the deployment.yml .

env:
    - name: DB-PORT
      valueFrom:
        configMapKeyRef:
          name: test-cm
          key: db-port

We will add this in the deployment.

Apply it

kubectl apply -f deployment.yml

So our pods are getting restarted.

Lets exec into one of the pods

Perfect, it is working fine.

Instead of directly using the environment variable, we will mount it as a file.

Apply

kubectl apply -f deployment.yml

It's working perfectly fine. Now let's change the port from 3306 to 3307 in configmap and verify whether it gets updated or not.

configmap is updated.

So the changes in the configmap are also updated in the app.

Let's create a simple secret to store the username password

kubectl create secret generic test-secret --from-literal=db-port="3307"

Our secret is created.

Here the db-port is encrypted.

I hope you liked this blog. If any mistake, do comment down below 🚀.

Do like and share this blog♥️

RBAC is abbreviated as "Role Based Access Control". It can be broadly devided into two groups:

• Users Management
  While acting as a root user in local development, such as when using Minikube, security considerations may not be as crucial. However, in organizational settings where multiple users have access to the cluster, there is a potential risk of users inadvertently or maliciously modifying sensitive resources. To avoid this risk, the cluster administrator or DevOps engineer must define user access, specifying who can access the cluster and edit resources. Role-Based Access Control (RBAC) is employed for this purpose, enabling DevOps engineers to define access based on roles..

• Service Accounts

  Similar to User management, we can manage access to the services or the applications that are running on the Kubernetes cluster.

Here are some key concepts in RBAC:

1. Service Accounts/Roles

2. Role/ cluster role Bindings

   1. Role Binding/ClusterRoleBindings
      Kubernetes does not manage the users. Kubernetes offloads the user management to the Identity providers. For example, if we try to log into any application, we don't need to create a user account, we can simply use multiple options such as signing in with Google, Facebook, Twitter, GitHub, Instagram, and so on. This is exactly what Kubernetes does. It does not create the user but will pass it to the Identity providers. So the Kubernetes API server acts as an OAuth server which passes it to the identity providers.

If our cluster is in Amazon EKS, then we can use the AWS IAM users to log into the Kubernetes Cluster for which we need to create the IAM OAuth provider.

To grant access to users, we will first create the roles and assign these roles to the developers. The role is a YAML file where we mention the grants to be provided to the users. If the role is created within a specific namespace,then it is a role, but if the role is created across the cluster then it is called a cluster role.

To attach these roles to the users/developers, we will use the role-binding.

To create the service account in the required namespace.

kubectl -n kube-system create serviceaccount subash

From Kubernetes Version 1.24, the secret for the service account has to be created separately with annotation

subash-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: subash
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: subash
type: kubernetes.io/service-account-token

Create the cluster Role

subash-cluster-role.yaml

This YAML file creates only ready access to all namespaces

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: subash
rules:
  #Allow read access to Pods and related resources in all namespaces.
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/log
      - pods/status
      - pods/portforward
      - namespaces
      - services
      - cronjobs
    verbs:
      - get
      - list
      - watch
      - create
      - update

Cluster Role Binding

subash-cluster-role-binding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: subash-binding
subjects:
  - kind: ServiceAccount
    name: subash
    namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: subash

We've successfully created a Service Account named "subash" in the "kube-system" namespace, along with a corresponding secret. Additionally, we've created a ClusterRole named "subash" that provides read access to specific resources across all namespaces. We've also created a ClusterRoleBinding that associates the "subash" ServiceAccount with the "subash" ClusterRole in the "kube-system" namespace.

Now, if we want to use this Service Account in a Pod within the "kube-system" namespace or another namespace, we need to reference the Service Account in our Pod specification.

spec.yml

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  namespace: kube-system
spec:
  serviceAccountName: subash  # Reference the Service Account here
  containers:
  - name: my-container
    image: nginx:latest

Apply the pod

kubectl apply -f pod-spec.yaml

After applying the Pod specification, the Pod will run with the specified Service Account, and it will have the permissions defined in the "subash" ClusterRole. Adjust the Pod specification based on your application's requirements and the level of access needed for the Service Account.

I hope you liked this blog. If any mistake, do comment down below 🚀.

Do like and share this blog♥️

Firstly, we will initiate the creation of an EC2 instance either manually from AWS ui or using Terraform. Subsequently, within the instance, our focus will shift to the installation of Hashicorp Vault. This installation lays the groundwork for subsequent integration with Terraform, establishing a smooth connection between the two tools. Through this orchestrated process, we aim to set up an environment where HashiCorp Vault becomes an integral part of our Terraform workflows, enhancing security and management of sensitive information.

Our instance is created and ready.

In the ec2 instance, we will install the Vault.

Install gpg

sudo apt update && sudo apt install gpg

Download the signing key to a new keyring

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

Verify the key's fingerprint

gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

Add the HashiCorp repo

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update

Finally, Install Vault

sudo apt install vault

To start the Vault

vault server -dev -dev-listen-address="0.0.0.0:8200"

After executing all the commands above, our vault is running successfully.

Open another terminal and log into the instance and run the following command that is displayed.

To access the vault we need to give the access to instance to access port 8200.

Lets access the Vault in http://3.88.160.206:8200/

Now to log into the Vault, the token is available in the terminal. Copy the code and paste it.

We have successfully logged into the Vault.

At the sidebar, we can see the Secret ENgines, Access, Policies. Let's know about these.

Secret Engine

The Secret Engine in HashiCorp Vault is responsible for managing and dispensing secrets, which are sensitive pieces of information such as API keys, passwords, or certificates. They are nothing but the different types of secrets that we can create in Hashicorp Vault.

We create a simple engine and the engine is enabled. IT will be used to create the key-value pairs.

A secret is created inside the Hashicorp Vault. only the root user has access to it. If anyone wants to access the secret through Terraform or Ansible, they need the grant. The grant is only possible from the Access feature of Vault. Here in the Hashicorp vault, Access can be considered as an IAM role and Policy can be considered as IAM policy.

In the Access section, there are different authentication methods, we chose the AppRole. But we cannot create the role using UI, we need do it from the terminal.

Enable the authentication method

vault auth enable approle

Let's create the policy

vault policy write terraform - <<EOF
path "*" {
  capabilities = ["list", "read"]
}

path "secrets/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "kv/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}


path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "auth/token/create" {
capabilities = ["create", "read", "update", "list"]
}
EOF

Create the Role

vault write auth/approle/role/terraform \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=terraform

After creating the AppRole, you need to generate a Role ID and Secret ID pair. The Role ID is a static identifier, while the Secret ID is a dynamic credential.

Generate Role ID:

vault read auth/approle/role/terraform/role-id

Generate Secret ID:

vault write -f auth/approle/role/terraform/secret-id

We will create the instance and will use the secret created in the vault.

main.tf

provider "aws" {
  region = "us-east-1"
}

provider "vault" {
  address = "3.88.160.206:8200"
  skip_child_token = true

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id = "ed169c1c-c96b-613a-479a-c1b82de9a720"
      secret_id = "5a65c56e-3132-2410-b83a-bd52658dfcf2"
    }
  }
}

data "vault_kv_secret_v2" "example" {
  mount = "kv" 
  name  = "test-secret" 
}

resource "aws_instance" "my_instance" {
  ami           = "ami-053b0d53c279acc90"
  instance_type = "t2.micro"

  tags = {
    Name = "test"
    Secret = data.vault_kv_secret_v2.example.data["subash"]
  }
}

terraform plan

It shows the 1 to add and in the tag section name=" test" and secret="subash" will be imported from the hashicorp vault.

terraform apply

The Terraform apply is successful and the resource is added.

Our resource is created successfully.

We can see in the tag section the secret named "subash" is also imported from the Hashicorp vault that we had created earlier.

Destroy the resources

terraform destroy

This integration enhances the security of your infrastructure by centralizing and managing secrets in the HashiCorp Vault, allowing Terraform to securely access and use these secrets during infrastructure provisioning. It's a great practice for ensuring the integrity of sensitive data and minimizing the risk of unauthorized access or data leaks.

I hope you liked this blog. If any mistake, do comment down below 🚀.

Do like and share this blog♥️

We will copy the application code using file provision. The file provisioner is used to copy files or directories from the local machine to a remote machine. This is useful for deploying configuration files, scripts, or other assets to a provisioned instance.

The remote-exec provisioner is used to run scripts or commands on a remote machine over SSH connections. It's often used to configure or install software on provisioned instances.

Let's do it hands-on

We will create a simple Python application named "app.py"

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello, Terraform from DevOps Learner!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=80)

We created a simple Python application using the Flask framework. When it is executed, "Hello, Terraform from DevOps Learner!" will be printed in the browser.

Our application code is ready. Now let's provision the resources in the AWS.

Let's create the Terraform code "main.tf"

Define the AWS provider configuration and all the resources that need to be created are explained in each block of code. We will merge these code at the end in the main.tf file.

provider "aws" {
  region = "us-east-1"  # Replace with your desired AWS region.
}

variable "cidr" {
  default = "10.0.0.0/16"
}

We provided the provider where we needed to create the resources. In this case, we have chosen the "aws" and the variable has a default value of "10.0.0.0/16," which represents the CIDR block for the VPC (Virtual Private Cloud).

Create key-pair

resource "aws_key_pair" "example" {
  key_name   = "terraform-demo-subash"  # Replace with your desired key name
  public_key = file("~/.ssh/id_rsa.pub")  # Replace with the path to your public key file
}

This block of code will create the key-pair file for the instance that is stored on our local machine.

• create VPC

resource "aws_vpc" "myvpc" {
  cidr_block = var.cidr
}

In this block of code, the vpc will be created with the cidr that is declared in the variable block i.e 10.0.0.0/16.

• Create Subnet

resource "aws_subnet" "sub1" {
  vpc_id                  = aws_vpc.myvpc.id
  cidr_block              = "10.0.0.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = true
}

In this block of code, a subnet will be create with the single availabiltity zone.

• Create InternetGateway

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.myvpc.id
}

This block creates an internet gateway and associates it with the VPC.

• Create Route Table

resource "aws_route_table" "RT" {
  vpc_id = aws_vpc.myvpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
}

This block creates a route table and adds a default route directing traffic to the internet gateway.

• Route Table Association

resource "aws_route_table_association" "rta1" {
  subnet_id      = aws_subnet.sub1.id
  route_table_id = aws_route_table.RT.id
}

This block associates the subnet with the above route table.

• Creating the security configuration

resource "aws_security_group" "webSg" {
  name   = "web"
  vpc_id = aws_vpc.myvpc.id

  ingress {
    description = "HTTP from VPC"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "Web-sg"
  }
}

The ingress is used to configure the inbound configuration for the instance, we have added two security rules one for SSH and the other for TCP to serve the application. Egress is used to configure the outbound configuration for the instance

• Creating the ec2 instance

resource "aws_instance" "server" {
  ami                    = "ami-0261755bbcb8c4a84"
  instance_type          = "t2.micro"
  key_name      = aws_key_pair.example.key_name
  vpc_security_group_ids = [aws_security_group.webSg.id]
  subnet_id              = aws_subnet.sub1.id

• Connecting to the ec2-instance

 connection {
    type        = "ssh"
    user        = "ubuntu"  # Replace with the appropriate username for your EC2 instance
    private_key = file("~/.ssh/id_rsa")  # Replace with the path to your private key
    host        = self.public_ip
  }

• File provision code

File provisioner to copy a file from local to the remote EC2 instance

 provisioner "file" {
    source      = "app.py"  # Replace with the path to your local file
    destination = "/home/ubuntu/app.py"  # Replace with the path on the remote instance
  }

• Provision code

provisioner "remote-exec" {
    inline = [
      "echo 'Hello from the remote instance'",
      "sudo apt update -y",  # Update package lists (for ubuntu)
      "sudo apt-get install -y python3-pip",  # Example package installation
      "cd /home/ubuntu",
      "sudo pip3 install flask",
      "sudo python3 app.py &",
    ]
  }
}

Now we will merge all code and apply the terraform commands.

Configure aws credentials

aws configure

terraform init
terraform plan

Here we will be creating the 8 resources.

terraform apply

g

All the resources are created successfully in our AWS account.

To list the resources created

terraform show

The instance is created and lets verify whether the app is running not.Copy the ip address of the instance and SSH into the instance.

ssh -i ~/.ssh/id_rsa ubuntu@public_ip

Our app is inside the ec2 instance and the python3 is also installed but the application is not running. Since we had added "&" at the end of the code and this code takes more time than the normal one so the terraform exits the process and moves to the other processes.

sudo python3 app.py

Great!! Our application is successfully deployed using the Terraform.

Delete the resources

terraform destroy

Throughout the entire blog, we covered the essential steps of creating AWS resources, including a VPC, subnet, security group, and EC2 instance, all orchestrated through Terraform's Infrastructure-as-Code (IaC) capabilities. The use of variables enhances flexibility, and the file and remote-exec provisioners enabled the smooth deployment of both infrastructure and application code.
Happy Learning!!

Terraform State files

Terraform state files commonly named "terraform.tfstate" maintain the current state of deployed infrastructures. These files store configurations, dependencies, and other essential information for Terraform to make informed decisions during subsequent runs. These files are updated upon each resource created.

Advantages of Terraform State File:

• Resource Tracking

• Concurrency Control

• Stores metadata of each resource

Disadvantage:
Security Risk - If the state files are exposed then much sensitive information will be leaked.

To overcome the disadvantages, a remote backend will be used to store the Terraform state file outside of our local file system and version control. Using S3 as a remote backend is a popular choice due to its reliability and scalability.

Terraform State Locking

The lock file, typically ".terraform.lock.hcl", prevents concurrent modifications to the Terraform state. This helps avoid conflicts when multiple users or processes attempt to apply changes simultaneously. Suppose if two users are concurrently trying to create the resources in the cloud provider then the terraform will be confused whom to allow and whom not to allow. It cannot allow multiple users to create resources simultaneously. So, state locking is introduced which will allow only a user at a time to deploy the resourcesYour explanation is generally correct, but there seems to be a small confusion in the file name and some clarifications could be made. Here's a refined version:

Terraform State Locking

The lock file in Terraform, typically named ".terraform.lock.hcl", serves an important role in preventing concurrent modifications to the Terraform state. When multiple users or processes attempt to apply changes simultaneously, conflicts can arise. For instance, consider a scenario where two users are concurrently trying to create resources in a cloud provider(like AWS) using Terraform. Without state locking, Terraform might become confused about which changes to accept and which to reject.

The purpose of state locking is to ensure that only one user at a time can deploy changes to the infrastructure. This process prevents conflicts and maintains the integrity of the Terraform state. By using the lock file, Terraform serializes the execution of operations, allowing for a controlled and orderly deployment process. This is especially important in collaborative environments(like in organizations, or companies) where multiple users may be managing the same infrastructure concurrently.

Therefore, state locking is a protective measure that helps manage the coordination of Terraform operations, avoiding the potential pitfalls of conflicting changes when working on shared infrastructure.

DynamoDB is used for state locking when a remote backend is configured. It ensures that only one user or process can modify the Terraform state at a time.

Let's do it practically.

Create main.tf, variables.tf and terraform.tfvars

provider "aws" {
    region = "us-east-1"

}
resource "aws_instance" "subash_instance" {
  ami = var.ami_value
  instance_type = var.instance_type_value
}

variable "ami_value" {
    description = "ami value of the instance"

}
variable "instance_type_value" {
    description = "value for the instance type"

}

ami_value = "ami-0c7217cdde317cfec"
instance_type_value = "t2.micro"

Let's deploy the resource

terraform init
terraform plan
terraform apply

Before this we need to configure the aws credentials

aws configure

We have configured the AWS credentials and now we are ready to deploy the resources using Terraform

Resource is created and the state file is present.

To keep the statefile secure we will implement the remote backend. We will store the state file in the S3 bucket in AWS. For that, we will create the S3 bucket using Terraform.

In the main.tf file add the following resource

resource "aws_s3_bucket" "s3_bucket" {
    bucket = "subash1816-demo-bucket"  
}

Let's verify if the bucket is created or not.

Our bucket is created and now in this bucket, we will store the state file. For that we we create the backend.tf file.

terraform {
  backend "s3" {
    bucket = "subash1816-demo-bucket"
    key    = "subash/terraform.state"
    region = "us-east-1"
  }
}

In key attribute, we set the path for the state file to be stored. A Subash directory will be created and inside it we will have the state file.

terraform init
terraform apply

Once we hit the terraform apply, a folder named "subash" will be created and inside it the state file will be attached.

We can see inside the bucket, subash is the folder and terraform.state file is attached.

Now we will delete the terraform.state locally and we will hit the terraform apply command and check whether the new resources will be created or not.

As per the Terraform, when the state file is deleted, terraform will have no any data on whether the resource is created or not and it cannot compare the resources to be created with the provisioned resources since there is no state file. Without the state file, Terraform won't be able to track the current state of the infrastructure it manages.

In our case, the local state file will be deleted but the remote state file is present. So let's check it out.

Apply:

terraform init
terraform apply

To Implement the state-locking mechanism

To create the dynamodb, add the dynamodb resource in the main.tf file.

resource "aws_dynamodb_table" "terraform_lock" {
  name           = "terraform-lock"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

In the backend, add the following attribute.

encrypt        = true
dynamodb_table = "terraform-lock"

Here once init the terraform, we will face the problem.

So we need to create the s3, dynamodb before creating the remote backend.

So we deleted the backend.tf and applied the following command.

terraform init
terraform apply

Our resources are created again. Now we will create the backend.tf

Previously we were not able to create the table since there was error. But now we are able to get the dynamodb table.

terraform {
  backend "s3" {
    bucket = "subash1816-demo-bucket"
    key    = "subash/terraform.state"
    region = "us-east-1"
    encrypt = true
    dynamodb_table = "terraform-lock"
  }
}

Once the backend.tf is ready. Apply the command.

terraform init
terraform apply

Therefore, we have enabled the state locking by using the Dynamodb. DynamoDB is used by Terraform as a backend to store the state file and manage the state lock.

Terraform Destroy

Once the provision is completed, delete the resources.

terraform destroy

Great!! Using the blog, we explored the critical aspects of Terraform state files and state locking for secure infrastructure deployment in the cloud. This approach ensures that sensitive information is stored securely, conflicts are avoided, and infrastructure changes are managed in a controlled manner.

Happy Learning!!

Terraform

Terraform is an open-source infrastructure as code (IaC) tool that allows us to define and provision infrastructure in a declarative configuration language. Terraform is a simple and superfast infrastructure deploying the tool. It supports different cloud providers like AWS, GCP, Azure, and so on.

Advantages of Terraform

• Infrastructure as Code

• Multi-cloud support

• Scalability

• Community driven

Terraform lifecycle

Terraform has a lifecycle that defines the stages involved in managing infrastructure using Terraform configurations. Here are the key stages in the Terraform lifecycle:

• Terraform init - Terraform downloads the required provider plugins and sets up the working directory in this stage.

• Terraform Plan - In this stage, Terraform analyzes the configuration files and the current state of the infrastructure to determine what changes need to be made

• Terraform Apply - In this stage Terraform executes the planned changes to the infrastructure and creates the infrastructure.

• Terraform Destroy - In this stage the Terraform deletes all the infrastructures that were created/deployed.

In Terraform HCL language is used. We can not remember the codes to create any resource so we will take the references from the official documentation.

Create an ec2 instance in the AWS using Terraform as IaC.

main.tf

provider "aws" {
    region = "us-east-1"

}
resource "aws_instance" "ubuntu" {
    ami = "ami-0c7217cdde317cfec"
    instance_type = "t2.micro"
    key_name = "nod.pem"

}

Provider block specifies the provider for which we are configuring resources, in this case, it's the AWS provider and the region specifies at which location we are creating the resources.

In the resource block, aws_instance is the type of resource and the instance will be referred as "ubuntu". The remaining attributes like ami, instance_type are the mandatory attributes so we had mentioned in the code.

Since we are creating the infrastructure in AWS, we will configure the AWS credentials in our terminal first.

 aws configure

We have our main.tf file and AWS is configured. Let's use the Terraform command.

Terraform init

Terraform is initialized successfully.

terraform plan

terraform apply

Once we hit the command, we need to enter a value "yes" to start the creation of resources. If not entered the resources will not be created.
Our resource is created and we will verify in AWS.

The ec2 instance is created successfully.

Destroy the resource

terraform destroy

The resource is destroyed successfully.

Case2: Creating resources using variables and output
But if we look at the code sensitive information like ami id, and instance type are exposed which should be kept secured. For this case, we will use the variables and if we want to get some information after the resources are created we will get that via the output.tf.

Let's create the resource using the variables and output.

main.tf

provider "aws" {
    region = var.aws_region

}
resource "aws_instance" "ubuntu" {
    ami = var.ami_value
    instance_type = var.instance_type_value

}

variables.tf

variable "aws_region" {
    description = "AWS region where the resources will be created"

}
variable "ami_value" {
    description = "value for the ami"


}

variable "instance_type_value" {
    description = "value for the instance type"

}

terraform.tfvars

aws_region = "us-east-1"
ami_value = "ami-0c7217cdde317cfec"
instance_type_value = "t2.micro"

terraform.tfvars is a file that allows us to store and organize variable values separately from our main Terraform configuration files. This file typically contains values for variables used in our Terraform configurations, and it provides a way to keep sensitive or environment-specific information separate from the main code.

output.tf

output "public-ip-address" {
 value = aws_instance.ubuntu.public_ip 
}

This output definition allows us to easily access the public IP address once the Terraform configuration is applied. Output.tf files are useful for retrieving information about the provisioned infrastructure.

Our configuration is completed and let's deploy it.

Terraform commands

terraform init
terraform plan
terraform apply

Once these commands are run one by one we will get the following output at the end.

As we mentioned the public IP address to be exposed in the output file, the public ip address is displayed. Let's verify it in the AWS.

We can verify the public IP is the same i.e. 3.88.201.167. We have successfully created the resource and exposed the public IP.

Destroy the resources

terraform destroy

We had created the two ec2 instances using different approaches are now terminated using the terraform command.

Through this hands-on experience, we demonstrated the creation of an EC2 instance in the AWS cloud, incorporating variables and outputs for enhanced security and information retrieval.

Happy Learning!!

What is EKS?

EKS, abbreviated as "Elastic Kubernetes Service", is an AWS service managed by the Amazon Web Service. Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications.

With Amazon EKS, we can easily run K8s on AWS without having to manage any underlying infrastructure. It simplifies the process of setting up, operating, and scaling a K8s cluster. We can easily deploy, manage, and scale the containerized applications using Kubernetes while taking advantage of AWS services.

The objective of the project: To install the 2048 game on the EKS cluster in AWS. The game will be deployed in the private subnet and will be accessed from the external world in the public subnet via the Application Load Balancer (ALB)

Prerequisites:

• kubectl

  use the link: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

• eksctl

    sudo wget -O /usr/local/bin/eksctl https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz
    sudo chmod +x /usr/local/bin/eksctl
    file /usr/local/bin/eksctl
    eksctl version
  

• AWS CLI

    sudo apt update
    sudo apt install -y awscli
  

Download these tools on your machine if they are not installed. We can look for the official documentation for the installation of these tools.

Configure the AWS credentials

Configure your account details then only we can access the AWS services and able to create the EKS cluster in AWS.

Once the credentials are successfully done then create the EKS cluster.

Create a cluster using Fargate

eksctl create cluster --name game-cluster --region us-east-1 --fargate

Be patient and wait for the cluster to be created. It takes around 8-10 minutes to create the cluster.

Our EKS cluster is ready now.

Update the kube-config file

We need to update the kubectl to work with our EKS cluster.

aws eks update-kubeconfig --name game-cluster --region us-east-1

The file is updated.

Let's create a deployment file for the 2048 game

First of all, let's create a fargate profile.

eksctl create fargateprofile \
    --cluster demo-cluster \
    --region us-east-1 \
    --name alb-sample-app \
    --namespace game-2048

We can verify in the compute section of the cluster whether the profile is created or not.

The Fargate profile is created successfully and the namespace is game-2048. We can create the instances on both the namespaces.

Deploy the Deployment, Service and Ingress

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/examples/2048/2048_full.yaml

This is taken from the official documentation of AWS. If you want to learn from official click here.

We can see the pods are running and also the service is in a running state, we can see it is running on the node port but the external IP is not allocated. It means anybody within AWS VPC or having access to the VPC can communicate with the pod using the Node IP along with the port. But to access this game from the external world or by the customers we have deployed the ingress.

Ingress is created with the class alb and port 80 but not the address. Only with the address, the customers can access the game. The address is not allocated because the Ingress controller is not created.

So we will create an Ingress controller that will look after the Ingress resources and create and configure the entire load balancer.

Before creating the Ingress controller, configure the IAM OIDC provider.

eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve
#replace the cluster name

IAM OIDC is integrated successfully.

Every controller in Kubernetes is a pod. So we will install an ALB controller and grant this to access the AWS services such as ALB.

Download IAM policy

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json

Create an IAM policy

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

Create IAM Role

eksctl create iamserviceaccount \
  --cluster=<your-cluster-name> \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::<your-aws-account-id>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve
#replace the cluster name and the AWS account ID in above code

The role is created successfully.

Deploy ALB controller

Add the Helm repo

helm repo add eks https://aws.github.io/eks-charts

Update the repo

helm repo update eks

Install the controller

helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \
  --set clusterName=<your-cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=<region> \
  --set vpcId=<your-vpc-id>
#replace cluster name, region, vpc-id

Load Balancer is installed perfectly without any errors.

Verify the Deployments

kubectl get deployment -n kube-system aws-load-balancer-controller

The load balancer controller is running with 2 replicas.

Let us see whether this load balancer controller has created an application load balancer or not.

We can see the load balancer was created by the load balancer controller just a few minutes ago.

Watching this ingress resource, the load balancer controller created the load balancer. Copy the address: k8s-game2048-ingress2-6d3ad9a3d9-1213836616.us-east-1.elb.amazonaws.com and access it on the web browser.

So our game is live and we can enjoy the game. Congratulations on completing the project smoothly.

Note: Delete all the resouces once the project is done.

We celebrate the successful deployment of the classic 2048 game on an Amazon EKS cluster, showcasing the power and simplicity of managing containerized applications on AWS. By carefully installing tools like kubectl, aws cli, and eksctl, configuring AWS credentials, and creating an EKS cluster with Fargate profiles, we laid the foundation for a smooth deployment.

Utilizing K8s resources and the AWS Load Balancer Controller, we ensured secure external access to the game through an Application Load Balancer. The journey concluded with the satisfaction of witnessing the game go live. In the future, we will bring many more projects based on Kubernetes. Stay updated and keep learning.

Happy Learning!!

AWS CloudWatch is a centralized monitoring service that allows users to gain insights into the performance and operational health of our AWS resources. It collects and tracks metrics, monitors log files, and sets alarms to notify users of potential issues.

Essentially, CloudWatch acts as the gatekeeper for AWS which will help us in monitoring, alerting, reporting, and logging.

Why do we need Cloudwatch?

Cloudwatch solves several critical challenges:

• Proactive Monitoring

• Resource optimization

• Automated Response

• Log Management

Advantages of AWS CloudWatch:

• Comprehensive Monitoring: CloudWatch allows you to monitor various AWS resources such as EC2 instances, RDS databases, Lambda functions, and more.Real-Time Metrics: It provides real-time monitoring of metrics, allowing you to respond quickly to any issues or anomalies that might arise.

• Automated Actions: With CloudWatch Alarms, we can set up automated actions like triggering an Auto Scaling group to scale in or out based on certain conditions.

• Log Insights: CloudWatch Insights lets us to analyze and search log data from various AWS services, making it easier to troubleshoot problems and identify trends.

• Dashboards and Visualization: Create custom dashboards to visualize your application and infrastructure metrics in one place, making it easier to understand the overall health of your system.

Real Usage of AWS CloudWatch:

1. Auto Scaling: CloudWatch plays a pivotal role in Auto Scaling by monitoring metrics such as CPU utilization. When a threshold is breached, Auto Scaling can dynamically adjust the number of instances to maintain optimal performance.

2. Application Monitoring: CloudWatch is extensively used for monitoring the performance of applications deployed on AWS. Metrics such as latency, error rates, and request counts help in identifying and resolving issues promptly.

3. Cost Management: By closely monitoring resource utilization metrics, CloudWatch assists in optimizing costs by identifying underutilized or overprovisioned resources.

4. Log Analysis: CloudWatch Logs are widely employed for centralized log management. Developers and administrators can easily search, analyze, and visualize logs to troubleshoot issues efficiently.

Let's get the knowledge of Cloudwatch practically.

Log into your AWS account and search cloudwatch.

Once we click the cloudwatch we will see the following interface.

If we click on Logs and then to log group, it will display all the logs of our activities without any configuration.

We can access these logs anytime whenever we need it. It also gives log insights if the query is passed to it.

Another feature is metrics. It provides information about disk utility, CPU information,ec2 instances, and many more.

We can see that the cloudwatch is monitoring 1002 default metrics. If we click on each service, we can know at which parameter it is monitoring the services.

Create an ec2 instance.

Our instance is created and access the instance using SSH protocol.

Successfully accessed and run the command "top" to get the resource utilization.

Now move to the cloudwatch and click on metrics, we will choose the ec2 instance.

We have selected the CPU utilization metrics.

Even in the ec2 dashboard, we can see the different monitoring options.

These monitoring are all related to the cloudwatch only. If we click on Manage Detailed Monitoring and enable it, we will get the metric every 1 minute.

In the ec2 instance, we will create a Python program to simulate an increase in CPU usage for testing CloudWatch.

Now after a minute, we will check the Cloudwatch metrics.

We can see the spikes in the CPU graph.

With metrics, we knew the critical challenges via monitoring but to act upon the metrics is done by the alarm. If the metrics reach a certain point, then the alarm will alert the developers/ admins about the metrics.

Click on Alarms

Create the alarm.

Choose to Select a metric > ec2 instance > across all instances/per instance >choose CPU utilization.

We have set the parameters for the alarm. Whenever the CPU spikes 50% or more, the alarm will be sent to the admin/users.

Create the Topic. Click Next.

We have set the message information for the alarm also. Click on create alarm.

We can see the alarm is not created in the dashboard but the message is alerted. Click here and view the message.

We need to confirm the subscription via email.

Click on confirm subscription.

Confirmed.

Now it is activated. But we will not get any alarm notification because the alarm is not triggered.

We will trigger it by running the Python script again in our ec2 instance.

We can see the spike has reached above the 50%(threshold). It must send the email regarding this event. Let's check the email.

We have received the mail and it clearly says the CPU utilization reached 50%.

We have successfully configured the ec2 resources and monitored them via the Cloudwatch.

Do not forget to delete the resources

Therefore, AWS CloudWatch stands out as a vital ally in the cloud landscape, offering a centralized and smooth monitoring solution for AWS resources. The practical walkthrough showcased its seamless integration with EC2 instances and the creation of alarms for timely notifications. CloudWatch not only provides comprehensive insights but also empowers users to take meaningful actions, making it an indispensable tool for maintaining optimal performance and ensuring the reliability of AWS resources in dynamic cloud environments.

Happy Learning!!

What is a Kubernetes cluster?

A Kubernetes cluster is a system for orchestrating and managing containerized applications, providing a platform for automating the deployment, scaling, and operation of application containers.

The cluster consists of two main components: the master node and worker nodes. The master node controls and manages the overall state of the cluster, including scheduling applications, maintaining cluster configuration, and responding to events. On the other hand, worker nodes host the actual containers and execute the workloads. They communicate with the master node, receive instructions, and ensure that containers are running as intended.

Kubernetes automates tasks such as container scheduling, load balancing, and self-healing, providing a robust and scalable platform for deploying and managing applications across diverse environments.

Why monitoring is required?

Monitoring a Kubernetes cluster is important because it helps you to:

• Detect and resolve issues with the cluster and its applications before they become major problems

• Optimize the performance of the cluster and its applications by identifying bottlenecks and other inefficiencies

• Track the health and utilization of cluster resources such as nodes, pods, deployments, and persistent storage

How monitoring of the Kubernetes Cluster is done?

In this blog, monitoring of the Kubernetes cluster will be done using Prometheus and Grafana. It involves deploying Prometheus within the cluster to collect metrics from various components and store them in a time-series database.

Grafana is then installed to create customized dashboards that visualize real-time metrics fetched from Prometheus. This monitoring stack is often set up using Helm charts, simplifying deployment and configuration.

Let's delve into the monitoring of clusters practically.

Start the Minikube cluster

minikube start

Our local cluster is ready using the Minikube and it is in running state.

We will install the Prometheus and Grafana using the helm. If the helm is not installed use this link.

Add the Helm chart for Prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Update the Helm repo for the latest updates

helm repo update

Install the Prometheus

helm install prometheus prometheus-community/prometheus

Our Prometheus is installed and can be accessed via port 9091

Let's verify the Prometheus installation

The Prometheus pods are running along with the Prometheus server. Kube-state-metrics as seen in the image is used to expose some Kubernetes metrices like API servers, deployments, pods, etc.

The Prometheus server is created using the ClusterIP mode. Let's convert this service into a Nodeport service.

Expose Prometheus Service

kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext

We exposed the Prometheus server using the node port and got the Kubernetes cluster ip using Minikube IP. Now we will access the Prometheus using http://192.168.58.2:31958

Our Prometheus server is ready to serve. The first step for monitoring the cluster is done.

Grafana

Add the helm repo

helm repo add grafana https://grafana.github.io/helm-charts

Update the helm repo

helm repo update

Install the Grafana

helm install grafana grafana/grafana

So our Grafana is installed and running.

To get the password for the Grafana run the command as shown in the reference image.

We can see that the grafana is running in the ClusterIP.

Expose the Grafana

kubectl expose service grafana — type=NodePort — target-port=3000 — name=grafana-ext

Grafana is exposed.

We can easily access the Grafana login page.

Successfully logged into the Grafana dashboard.

Now we will add Prometheus as the data source.

Click on Data Source > choose Prometheus.

In the connection: insert the Prometheus URL.

Click on Save and test.

Now click on Building a dashboard or from the homepage we can create the dashboard.

Instead of creating the dashboard from the beginning, we can simply import the dashboard that is already pre-built.

From here we can copy the ID of the dashboard i.e. 13332

Click on load to load the dashboard.

Click on Import.

You can import the multiple dashboards.

Using the id:15282
And the Dashboard can be visualized as below.

Using the dashboard:3662

We can also expose the kube-state metrices.

To expose Kube-state metrics

kubectl expose service prometheus-kube-state-metrics  --type=NodePort --target-port=8080 --name=prometheus-kube-state-metrics-ext

We can also set this kube-state-metrics endpoint as a job in Kubernetes to know about the kube-state details.

Hence, the detailed steps covered the installation of both monitoring tools (Prometheus and Grafana) using Helm charts. The exposure of services via NodePort for simplicity, and the integration of pre-built dashboards in Grafana to visualize Kubernetes metrics.

It is important to note, however, that while using NodePort for external access is suitable for local or testing environments, it may not be the best practice for production setups. In production, a more secure and scalable approach involves utilizing an Ingress controller or LoadBalancer service type for external access to Prometheus and Grafana. These options provide better control over routing and security, ensuring a more robust and production-ready monitoring solution for Kubernetes clusters.

Happy Learning!!

CloudFront

AWS CloudFront is like a magic carpet for our website. It's a content delivery service that helps our web pages, images, and videos around the world at lightning speed. How? By storing copies of your website's stuff in special places called edge locations.

CloudFront acts as a Content Delivery Network (CDN) service, strategically caching website content at its edge locations. These locations are strategically placed worldwide, aligning with where users access the data.

Consider a scenario where a website is initially uploaded from Australia, and users, like User1 in India and User2 in the US, access the same site. Without CloudFront, the website would take time to load for both users due to the geographical distance. However, with CloudFront in action, copies of the website are distributed to the nearest edge locations, such as India for User1 and the US for User2. This way, when users access the site, it's like grabbing it from a nearby storage rather than halfway across the globe. CloudFront significantly reduces loading times, ensuring users around the world can effortlessly and swiftly access the website.

CDN provides the lowest latency as the content is cached to the nearest edge locations.

Let's deep dive into the practical so that we can get hands-on experience with CloudFront service.

Objective: Host the static website in the S3 bucket and access the website using CloudFront. Use Amazon S3 to store your static website content and use Amazon CloudFront to distribute and serve that content globally, providing a faster and more reliable experience for users

Log into the AWS account.

First of all, we will upload our static website in the S3 bucket.

Create a S3 bucket.

AWS Region: default

Bucket name: demodevopsprofile.com

Object Ownership: ACLs Disabled

Block Public Access settings for this bucket: Block all public access

Bucket Versioning: Enabled

Click on Create bucket.

To host a static website, click on the bucket name and choose the properties tab. At the bottom choose the "edit" option on static website hosting.

Now choose enabled and name the root file of the static website.

Click on Save Changes.

Now go back to the bucket and upload the files of the website.

You can easily access the code using the GitHub link.

We will try to access the website that we had uploaded.

Copy the URL from here and try to access it in the browser.

We observe that direct access to the site is forbidden due to blocking all public access at the S3 bucket level. Accessing the site directly from the S3 bucket is not preferred, as it poses security concerns and tends to have longer loading times. To address this, we will implement a CloudFront distribution on top of the S3 bucket to enhance the website's accessibility.

Now we will move to CloudFront. In the search bar, search CloudFront.

Click on Create Distribution

Origin domain: website link

Origin access: Legacy access identities (create OAI)
\>Bucket policy : Yes, update the bucket policy

Web Application Firewall (WAF): enabled (for enhanced security), for now we keept it default(disabled)

Click on Create Distribution.

Our CloudFront distribution is created. Once the deployment is completed we will try to access the website using the link: https://d24tgcl2ko5qby.cloudfront.net

Congratulations to us, we have successfully accessed our website using the CloudFront distribution instead of the S3 bucket.

In the end, AWS CloudFront is like having a super-fast, globally connected buddy for your website. It makes sure your fans get your awesome content without a hitch. So, if you want your website to be a rockstar, give CloudFront a try!

Happy Learning!!

Virtual Private Clouds (VPCs) have become a major factor in building scalable and secured infrastructures. It provides a way to create isolated and customizable networks within the cloud environment, offering businesses the flexibility and control they need. In this blog post, we will delve into the components that make up a Virtual Private Cloud and explore how they work together to create a robust and secure networking foundation.

Why do we need VPC?

• Isolation and Security

• Customization of Network Architecture

• Scalability

• Hybrid Cloud Connectivity

• High Availability and Fault Tolerance

Components of VPC

1. VPC: The VPC itself is the top-level container that holds all other components. It is a logically isolated section within the cloud where users can deploy their resources and make it more secure. VPCs allow any organization or users to define their IP address range, route tables, and subnets.

2. Subnets: Subnets are subdivisions of a VPC and are associated with a specific availability zone. Subnets allow us to isolate and group resources based on the user's needs.

3. Route Tables: Route tables define the rules for routing traffic within the VPC. Each subnet is associated with a route table, determining how traffic is directed. Route tables control the flow of traffic between subnets and the internet, ensuring proper communication.

4. Internet Gateway (IGW): The Internet Gateway is a component that facilitates communication between instances within the VPC and the Internet. It acts as a gateway for outbound and inbound traffic, allowing instances to access the internet and be accessed from the internet.

5. Network Access Control Lists (NACLs): NACLs are stateless filters that control traffic at the subnet level. They act as a firewall, allowing or denying traffic based on rules defined by the user. NACLs are operated at the subnet level and provide an additional layer of security.

6. Security Groups: Security Groups are stateful firewalls that operate at the instance level. They control inbound and outbound traffic for instance by defining rules.

7. Elastic Load Balancer (ELB): ELB is a service that automatically distributes incoming application traffic across multiple targets, such as EC2 instances, in multiple availability zones.

8. Virtual Private Network (VPN) and Direct Connect: VPN establishes a secure tunnel over the public internet, while Direct Connect provides a dedicated network connection.

9. Elastic IP Addresses: Elastic IP addresses provide a persistent IP address that remains associated with an instance, even if it is stopped and restarted.

10. Peering Connections: VPC peering allows the connection of two VPCs, enabling them to communicate with each other as if they are within the same network. It is useful when resources from different VPCs need to interact.

Now we know the basic terminologies that are used at the VPC level. Its time for us to practically implement and know the usage of AWS VPC services.

Search VPC > click on VPC > create VPC

In clicking Create VPC, you will be prompted to fill in the details below:

Resources to creat : VPC and more

Name tag auto-generation: aws-prod-project

IPv4 CIDR block**:** default

IPv6 CIDR block: No ipv6

Tenancy: default

Number of Availability Zones (AZs): 2

Number of public subnets:2

Number of private subnets:2

NAT gateways : 1 per AZ

VPC endpoints: None

Then click on Create VPC

This is the configuration that I have done.

Our VPC is successfully created

Create ec2 with the autoscaling group

In AWS autoscaling group cannot be created directly, we need to choose Create Launch Template.

In the launch template page,enter the required details

• Launch template name - required: aws-demo-proj

• Template version description: describe the template

• Application and OS Images (Amazon Machine Image) - required: choose any supposed Ubuntu

• AMI: Free tier

• instance type: t2.micro (Free Tier)

• key pair: use existing or create a new key pair

• In Network settings, subnet: default (do not change)

• Firewall (security groups): create a security group

  -Add a security group name, description

• VPC: Choose a newly created VPC

• Inbound Security Group Rules: add 2 rules ssh(to access ec2 instance) and custom TCP (port 8000 to access the application)

• Remaining all as it is (Default)

Click Create Launch Template

This is a sample of how I configured the template.

Our template is successfully created and now create the autoscaling group.

Our template is now available, if you cannot see the template just refresh the page. After adding the details click next.

Here we have chosen our VPC and in the Availability Zones and subnets, we choose the private subnets since we want our instances in the private subnet as in the architecture of the project.

Click on Next.

Configure advanced options: keep everything default

Click on Next.

In the Configure group size, keep desired capacity=2, min.=1, and max=4

Automatic scaling= No scaling policies

Click on Next and after this all are optional to add notifications and tags. I have skipped these and created the autoscaling group.

We can see our 2 instances are launched as we kept desired state as two.

Now let's verify whether the autoscaling group has created instances in us-east-1a and us-east-1b.

Perfect the autoscaling group has created the two instances in two zones in the private subnet.

Before creating an application load balancer we need to install the application in the server(instances).

Since we need the public IPV4 address to ssh into any instance. We can see clearly in the image we have no IPv4 address as it is in a private subnet and we don't want our server to be access public. To solve this issue, a bastion host comes into the picture. Bastion host acts as a mediator between private subnet and public subnet.

So we will create a bastion host in the public subnet and try to access the server.

The process is the same to create a bastion host as creating the instance but in a bastion host, three important points are: VPC - choose the same VPC in which the above-created servers are running, choose auto-assign public IP enable and inbound rules allow ssh.

Click on the launch instance.

Our instance is created and we will SSH into the bastion host and from this we will try to access the other two instances and install the application.

To get access to two of these private subnets, we also need a key-pairs file in Bastion. So we will copy our pem file in Bastion host also.

scp -i ~/Downloads/nod.pem ~/Downloads/nod.pem ubuntu@3.238.77.37:/home/ubuntu

Now let's access the bastion-host instance using SSH protocol

ssh -i .pem_fiel ubuntu@3.238.77.37

We have done this step successfully.

We can see our pem file is available in the instance. Now we will SSH into the other instance using the same pem file.

As of now we are in the bastion-host instance and from here we will ssh into instance 1 using its private IP address

ssh -i pemfile ubuntu@private_ip

We can see the private IP of instance 1 so we have access to the instance. In this instance, we will create a very simple application and run it on port 8000.

In the same way, we will create our second application on instance 2.

ssh -i nod.pem ubuntu@10.0.138.152

Now, Create a Load balancer and attach these instances as target groups

In the EC2 Dashboard select load balancers. Then click on Create the load balancer.

In the next step select Application load balancer.

Click create.

Give the name to the load balancer and it must be internet-facing and has an IPV4 address type.

Choose the VPC that we created earlier as our servers are running on that VPC. In Loadbalancer we need to select the public subnets for mapping.

Select the Create security groups and port 80. For default action, if there is no any security group, click on Create Security group.

Choose instance as our target, give the target group name, keep other default, and choose next.

Select the target instances in which the application will be running or application is running and also click include as pending below. Then click Create Target group.

In the load balancer configuration select the recently created target group.

Finally, our load balancer is created and also we can see the HTTP protocol at port 80 is not reachable. So click on the Security tab, choose security group, and in the inbound rule add HTTP at port 80.

Now let's check the load balancers.

Now it is reachable.

To check the health status of the Targets click on the project name.

We can see our servers are both healthy. We will try to access our applications using this link aws-prod-project-1090487507.us-east-1.elb.amazonaws.com, it will be available under the Details section (DNS name) in the load balancer page.

Our first application is perfectly running.

Again let's try to access the same URL in another tab.

We can see our second app running on the second server is loaded successfully. Hence the load balancer has successfully distributed the server requests.

We covered the basic building blocks like subnets and route tables, and walked through practical steps for setting up a secure and scalable cloud environment. Think of VPCs as the foundation for creating a safe and flexible space in the cloud, and learning about them is a valuable skill for anyone working with online services.

Happy Learning!!

In our previous blog, we learned about the Service and its offerings like service discovery, load balancing, and exposure to the external world. Now why do we need the Ingress controller? Before Kubernetes released v1.1, there was no Ingress so people used to use Kubernetes with service only. Even before people migrated to Kubernetes, they used the legacy system like they use virtual machines or physical servers and on top of it they installed their applications and to access these applications they used enterprise-level load balancers like nginx.

The enterprise-level load balancing offered different capabilities like ratio-based load balancing, sticky session(if a request is going to a particular pod, then all the requests should go to the same pod), path-based load balancing, domain-based load balancing, whitelisting (allow specific IP range), black listing(block specific IP range) and so on.

How does Ingress solve the problem?

The load balancing offered by the Kubernetes service was a simple and round-robin mechanism i.e. simply distributing the traffic request. Kubernetes service was missing following services:

1. Enterprise and TLS load balancing

2. For every load balancer type service, the cloud provider would charge for each static public IP address

Ingress is solving the problem that Kubernetes service did not have enterprise-level load balancing capabilities. Kubernetes users will create a resource called an Ingress controller and the load balancing services like Nginx, will implement the Ingress controller and as a Kubernetes user, will deploy the ingress controller in the Kubernetes cluster using Helm chart or YAML manifest. Once it is deployed then the developer will create an Ingress yaml resource for their Kubernetes service. So the Ingress controller watches on Ingress resources.

Let's dive practically and understand how to deploy Ingress controller that watches on our Ingress resources.

Start the Minikube

minikube start

Previously in Kubernetes service, we had created the pods we will use the same pods. If you don't know how to do it please click here to create the pods.

Now let's create an Ingress resource and setup a host based load balancing

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-example
spec:
  rules:
  - host: "foo.bar.com"
    http:
      paths:
      - pathType: Prefix
        path: "/bar"
        backend:
          service:
            name: python-django-service
            port:
              number: 80

To apply

kubectl apply -f ingress.yml

Our ingress resource is created.

We can the address field is empty.

We encountered the error as we could not reach our application because for this ingress resource we have not created the Ingress controller.

So let's install the INgress controller. IN the market there are lots of ingress controller, I am going to install the nginx ingress controller since it is very light-weight.

To enable the NGINX Ingress controller, run the following command:

minikube addons enable ingress

Verify that the NGINX Ingress controller is running

kubectl get pods -n ingress-nginx

Let's check whether the INgress controller identifies the ingress resources that we had created.

kubectl logs ingress-nginx-controller-6cc5ccb977-tq2ll -n ingress-nginx

So the ingress controller has identified our ingress resource and has successfully synced also.

If you noticed previously there was no IP address in the Address field but now there is IP allocated and updated. Now, we can access the ingress resources in foo.bar.com/bar since we have used the ingress controller and it has updated the configuration .

Now update the /etc/hosts configuration file so that whenever we hit the foo.bar.com the 192.168.58.2 will be accessed.

We will try to access the foo.bar.com

We are successfully able to send the request to the foo.bar.com. We have done it successfully.

By using Ingress controllers, Kubernetes users can efficiently manage external access to their applications, benefiting from features like host-based load balancing and improved traffic routing. As Kubernetes continues to evolve, understanding and using Ingress controllers will be important for optimizing application deployment and accessibility within a Kubernetes cluster.

Happy Learning!!

AWS CFT implements the principle of IAC which is not implemented by the AWS CLI. There are different IAC tools like Terraform, Crossplane but AWS CFT is majorly for the AWS cloud only. IAC is a concept where we write codes to create resources in the cloud. IAC tool acts as a middleman between the user and the cloud provider.

For quick actions, we can simply use the AWS CLI, but to create multi-structured resources, we can use AWS CloudFormation (AWS CFT). AWS CFT supports JSON and YAML templates. YAML is the preferred template as it is the most widely used template. But if you are comfortable with JSON then it is also good to use, the limitation of JSON is that we cannot make comments in between the codes which makes it difficult to understand the code if it is complex.

To get deep knowledge about AWS CFT use the official documentation link:
https://docs.aws.amazon.com/cloudformation/

Let's dive into the AWS CFT practically.

Log into your AWS account. Search for CloudFormation and click on it.

In the cloud formation page, click on Create Stack, stack is the one that implements the template that we have written.

Then click on Create template in designer.

You will get into this page, you can simply drag the resources in the plane and the code will be generated in JSON or YAML based on your requirement.

We simply dragged the bucket and the code is generated in the below terminal.

Resources:
  S3BUCKET:
    Type: 'AWS::S3::Bucket'
    Properties: {}

The code snippet represents an AWS CloudFormation template to create an Amazon S3 bucket. The logical name is "S3BUCKET," and it specifies the resource type as 'AWS::S3::Bucket.' The template is currently basic, without additional properties, indicating a simple S3 bucket creation.

Since we have already created a template we will upload the yaml template.

Our template file is uploaded. The sample yam file we uploaded is:

Resources:
  S3BUCKET:
    Type: 'AWS::S3::Bucket'
    Properties:
        BucketName: "demo-aws-subash1122"
        VersioningConfiguration:
            Status: Enabled

Give a name to the stack and skip all the pages and submit it.

We can see our resource is being created. Let's verify whether the bucket is created or not.

The bucket is created successfully along with a template file in S3. AWS Cloud saves all the templates in the S3 bucket that we created.

In Cloudformation there is a feature of drift, that allows you to identify and understand any changes made to the stack resources outside of CloudFormation.

Let's delete the bucket that we had created and check the drift status in cloudformation.

The bucket is deleted.

Here we can see in the drifted results that the bucket is deleted.
In AWS CloudFormation, "drift" detects the difference between the expected stack resource configuration and the actual configuration of those resources in the deployed stack

Again let's create an Instance using the AWS CloudFormation Template.

createinstance.yaml

AWSTemplateFormatVersion: "2010-09-09"
Description: "This is our first instance using CloudFormation"
Resources:
  mydemoinstance:
    Type: 'AWS::EC2::Instance'
    Properties: 
      ImageId: ami-0c7217cdde317cfec
      KeyName: nod
      SecurityGroupIds: 
        - sg-047bf8e20308ceb4f
      InstanceType: t2.micro

Note: for key pair either use existing one or create the new key pair from the EC2dashboard

Let's upload it in the cloud formation.

Our template file is uploaded and named the stack as ec2-instance.

We can see the creation is completed and we will verify it in ec2 dashboard.

Hence the instance is also created successfully. You can also try the same template to create the instance and just replace the security groups and key pair files.

We have successfully accessed the instance that we have created using the CloudFormation.

Note: Don't forget to delete the resources that you created earlier and stay safe from the charges of AWS services

In a nutshell, AWS CloudFormation is like a wizard for setting up our cloud resources on Amazon. It uses written instructions (templates) to create and manage your cloud resources, making it easier and more consistent. We showed how it can create an S3 bucket and an EC2 instance just by following the template. This is handy for automating and organizing your cloud infrastructure without manual hassle.

Happy Learning!!

S3 Buckets

S3 buckets serve as containers for storing objects, which are essentially files, within Amazon S3. Each bucket must have a globally unique name across the entire AWS platform. Conceptually, you can liken an S3 bucket to a top-level directory that serves as a repository for organizing and storing your data.

Why S3 Buckets

S3 buckets offer a dependable and immensely scalable storage solution suitable for a multitude of use cases. They are simply utilized for different tasks such as backup and restoration, data archiving, storage of website content, and serving as a primary data source for big data analytics, S3 buckets prove versatile in meeting diverse storage needs.

Key Benefits of AWS S3 Buckets

• Durability and availability: S3 provides high durability and availability for your data. S3 ensures 99.999999999% durability for stored objects by automatically replicating data across multiple servers and data centers.

• Scalability: You can store and retrieve any amount of data without worrying about capacity constraints.

• Security: S3 offers multiple security features including access control lists (ACLs), bucket policies, and integration with AWS Identity and Access Management (IAM).

• Performance: S3 is designed to deliver high performance for data retrieval and storage operations.

• Cost-effective: S3 offers cost-effective storage options and pricing models based on our usage patterns.

Additionally, Versioning allows us to preserve, retrieve, and restore every version of every object stored in a bucket, providing protection against accidental deletion or overwrites.

Let's know more about S3 buckets and their usage practically.

Let's create a bucket, search S3> click S3 >Click Create Bucket

AWS Region: default

Bucket name: demo-s3-prod-example.com

Object Ownership: ACLs Disabled

Block Public Access settings for this bucket: Block all public access

Bucket Versioning: Disabled

Then click on Create bucket and our bucket is created successfully.

Upload a file/object in the S3 bucket

Let's add a simple file to our bucket.

The index.html file contains simple HTML code.

<!DOCTYPE html>
<html>
  <head>
    <title>My First HTML Page</title>
  </head>
  <body>
    <h1>Welcome to my page!</h1>
    <p>This is a paragraph.</p>
    <ul>
      <li>Item 1</li>
      <li>Item 2</li>
      <li>Item 3</li>
    </ul>
  </body>
</html>

Let's try to upload the same file again and check whether the copy of the file can be uploaded or not without the change in the file.

We uploaded the same index.html file.

But we cannot see the copy of the file.

S3 also offers the versioning of the file like the version control systems. We can retrieve the file with any version.

Let's customize the code of index.html and upload it in the s3 bucket. But before that enable the versioning in the s3 bucket configuration

<!DOCTYPE html>
<html>
  <head>
    <title>My First HTML Page</title>
  </head>
  <body>
    <h1>Welcome to my page!</h1>
    <p>This is a paragraph.</p>
  </body>
</html>

We can see the two versions of the index.html file. We can create multiple versions of the file and can retrieve any version of the file.

Bucket Permission

Anyone within the organization who has S3 full access permission and easily access the S3 bucket resources. Based on the requirements, we need to set the bucket permission so that the data within the buckets are safe and secure.

Let's create a use called demo-s3 with no access to the s3.

Our user demo-s3 is created and we will log in as a demo-s3 and try to access the bucket that we created.

We cannot access the bucket since we have not given access to the user to access these buckets. We need to attach the policy for the users so that the user can access the S3 resources.

Now we will grant permission to this user so that the user can access the S3 buckets.

The permission is added to the demo-s3 user.

Now the user can easily access the S3 bucket.

Though we have attached the policy so that the user can access the buckets but we don't want any users to do anything on our buckets. So lets make it more secure using the bucket policy.

{
  "Version": "2012-10-17",
  "Id": "RestrictBucketToIAMUsersOnly",
  "Statement": [
    {
      "Sid": "AllowOwnerOnlyAccess",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*",
        "arn:aws:s3:::your-bucket-name"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalArn": "arn:aws:iam::AWS_ACCOUNT_ID:root"
        }
      }
    }
  ]
}

This policy says that the bucket is only accessible to the root user and restricted to other users.

This is what I have attached to the bucket policy. Now lets verify whether the user can retrieve the s3 bucket resources as they have s3 full access.

We can see the user can not access the resources from the bucket. It is more important to identify whom to give access or not.

Host a static website in S3 bucket

Hosting a static website is very easy and simple in S3 as it is very cheap and affordable.

Let's upload an index.html file or use the previous index.html file as above. Now make the static website enabled. First, click on bucket-name > permission > static website hosting(edit)>enabled> index.html

As shown in the image you can follow the steps and save it.

After saving it, at the bottom, you will find the URL to access the website. Now simply copy the link and paste it into the browser.

You can see we are not able to access our website and a 403 forbidden error is shown. This is because if you remember while we created the bucket we choosed "Block Public Access settings for this bucket: Block all public access" due to which we are not able to access the website.

We will uncheck the block all public access so that we will be able to access the website.

Now we will try to access the website.

After doing everything right also we are not able to access the website. This is because we had set the bucket policy that only the root user will have access but not others. Now we will change the policy so that everyone can access the website having access to the internet.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::demo-s3-prod-example.com/*"
        }
    ]
}

Save the policy and refresh the browser.

We can access our sample website finally.

Through these practical demonstrations, we learned how to create an S3 bucket, upload files, and showcased the importance of versioning for better data management in AWS. The significance of permissions and security features, such as IAM user policies and bucket policies, was practically demonstrated for controlled access. Whether you're a developer, system administrator, or business owner, AWS S3 is an essential tool in the cloud computing landscape, empowering you to store, manage, and retrieve data smoothly.

Happy Learning!!

Why Do We Need Kubernetes Services?

In Kubernetes, a DevOps engineer deploying a pod as part of a deployment might face challenges when managing dynamic IP addresses, especially when scaling the application. For instance, if a pod goes down and a new one is created by the replica set, the associated IP address change can disrupt end-user access.

In the real world, this is not implemented. For example, an XYZ website does not tell its user to access the site in this and that IP as the request increases. This auto-healing behavior maintained the pods but not the IP address associated with which users use to access them.

• Load Balancing

To solve the issues of the above scenario, we can create a service on the top of deployment. Service acts as a load balancer. Service uses the component of K8s that is kube-proxy. As the service offers load balancing, the test users can access the application by the service name provided by the Kubernetes instead of the IP address as it changes frequently. So based on the request received the load balancer distributes the requests to different pods.

• Service Discovery

If the user tries to access the pod whose IP is changed then the service also resolves this problem. Service does not track the IP addresses as the pod is created frequently with a new IP every time. For large projects there may be multiple pods and IPs are changed frequently, if the Service starts to track the IPs then the service will fail. Now what the service does is the service would not bother about the IP. So service uses a new process the Labels and Selectors for service discovery. For every pod created, selectors and labels will be applied. So service will watch for the labels instead of the IP address.

• Expose to the external World

Service can allow us to access our application outside the Kubernetes cluster. We can do it in many ways. Service provides 3 types of service:

1. ClusterIP service
   If we create any service using clusterIP mode, as it is a default mode then the application can be accessed only inside the Kubernetes cluster.

2. Node Port service
   If we create any service using the Node port mode, then the application can be accessed within the organization i.e. anybody with the same network can access the application. They won't get the master node access but the work node access.

3. Loadbalancer service
   If we create a service of load balancing type then we will get the elastic load balancer IP address as it is the public IP address. So whoever wants to access our application from anywhere in the world can easily access the application using the elastic IP address. So this load balancing works on the cloud, depending upon the cloud provider as the cloud provider provides the creation of elastic IP.

Let's dive into practical

Start the Minikube cluster

Our Minikube is ready to use now.

This is our Python application directory. If you want to use this application you can clone the repo here.

Let's build the docker image and push it into a public registry like DockerHub. In case you did not push it to the public registry then you will face the problem of Imagepull Backoff error and the pods will not get ready or start.

docker -t image_name name path_of_docker_File

So our image is built.

Let's create a deployment file called "deployment.yml"

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-python-app
  labels:
    app: sample-python-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-python-app
  template:
    metadata:
      labels:
        app: sample-python-app
    spec:
      containers:
      - name: python-app
        image: subash07/python-app:latest #after I pushed the image to dockerhub
        ports:
        - containerPort: 8000

kubectl apply -f deployment.yml

Created.

To get the deployment information

kubectl get deploy

Here we can see the status as Imagepull Backoff and my pods are not ready.

This occurs because my docker image is not in the public registry, that'swhy it could not pull the image that I mentioned in the YAML file.

In case you want to use your local image run the following commands.

Execute the Minikube Docker daemon configuration:

eval $(minikube docker-env)

Build your Docker image:

docker build -t demo-pyapp .

This ensures that the Docker image is built using the Minikube Docker daemon, making it available for use within your Minikube cluster.

If it doesnot work then push the image to dockerhub or any public registry and use the image name as I have done it

We can get the IP address of the pods using kubectl get pods -o wide

Our two pods are running as we set the replicas: 2 in deployment.yml

To get detailed information about pods we can add verbosity which ranges from 0-9

• Level 0: Normal output

• Levels 1-3: Informational messages

• Levels 4-6: Debugging information

• Levels 7-9: Tracing information

kubectl get pods -v=7

This helps us to debug the pod error.

Using these given IP addresses we can access our application inside the cluster only not outside the cluster.

We can see our app is running inside the Minikube cluster. We can check in both IP addresses to see our app running inside the cluster.

But if we try to access the application from outside the cluster, we won't be able to access it. Since the IP of the external browser and clusterIP are not the same as our app is in ClusterIP(default).

If we delete the pod then the replica set will immediately create the new pod with a different IP but in the same subnet.

So our application is only accessible to those who have clusterIP. To solve the problem of accessing the app from the same network and external world network we need the service.

Type= Nodeport: to access the app within the organization(Same network)

Type= LoadBalancer: to access the application from the external world anyone who has access to the internet.

Let's create the service file = service.yml

apiVersion: v1
kind: Service
metadata:
  name: python-django-service
spec:
  type: NodePort
  selector:
    app: sample-python-app
  ports:
    - port: 80
      targetPort: 8000
      nodePort: 30007

We need to select the label from the template section of deployment.yml to insert in the selector section of service.yml. The target port is set to 8000 as our application is running in the port 8000.

Run the service .yml file

kubectl apply -f service.yml

To get services created

kubectl get svc

We can see the app running in the node port which is different from clusterIP. We can access the app either by doing Minikube SSH or using the Minikube IP address in our browser.

This is inside the cluster.

To access from the external browser within the same network.

Here we can access the application from our browser also. If other people will try to access it, they won't be able to access the application as it is not a load balancer type service.

To access the application from the external browser from another IP we need to use load balancer type.

We changed the node port type to load balancer then the external IP is <pending> as we are executing it from the local environment. The load balancer IP is only provided by the cloud providers so we need to use it in the cloud services like EKS, GKE, AKS.

We can simply edit the svc file or create a new load balancer yaml file = Loadbalancer.yml file

apiVersion: v1
kind: Service
metadata:
  name: python-django-service
spec:
  type: LoadBalancer
  selector:
    app: sample-python-app
  ports:
    - port: 80
      targetPort: 8000
      nodePort: 30007

Kubeshark

Kubeshark is a very simple application that is used to show the real-time flow of the traffic request. Kubeshark offers real-time, cluster-wide, identity-aware, protocol-level visibility into API traffic, empowering its users to see with their own eyes what’s happening in all (hidden) corners of their K8s clusters.

To install in Linux

sh <(curl -Ls https://kubeshark.co/install)

To run the CLI, use the tap command.

kubeshark tap

In one of the tabs of the terminal Kubeshark is running.

And in the next tab, we sent the 8-9 requests to our application, we will see this traffic request being distributed in the two pods with the IPs: 10.244.0.67 and 10.244.0.66. Let's check it out.

This is the sample image taken from the internet because I got the error on my minikube as kubeshark-worker-daemon-set pod did not create due to which the request sent from my terminal did not reach the KUbeshark. The error message "Error response from daemon: invalid CapAdd: capability not supported by your kernel or not available in the current environment: 'CAP_CHECKPOINT_RESTORE'" indicates that the CAP_CHECKPOINT_RESTORE capability is not supported or available in your Minikube kernel. But you can try from your side.

In summary, Kubernetes Services emerge as an important component in managing the complexities of deploying applications at scale. By adopting a label-based service discovery approach, Services overcome the limitations of tracking individual pod IPs. The practical implementation within a Minikube cluster highlighted the significance of Services in providing external access to applications.

Happy Learning!!

What is Lambda Function?

AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS). It allows you to run code without provisioning or managing servers.

Features:

Serverless Computing

EventDriven Execution

Pay-as-you-go model

Real-Time Use Case :

Scenario: Automated Snapshot Cleanup

Problem Statement:

• Suppose you want to ensure efficient resource management in AWS and avoid unnecessary costs associated with unattached EBS (Elastic Block Store) snapshots. This means you created unlimited resources that are inactive but are causing unnecessary costs. So we need to create an automated program that runs periodically and removes/deletes these inactive resources

How does AWS Lambda bring the solution?

1. Lambda Function Creation:

   • Develop a Lambda function that identifies and deletes unattached EBS snapshots. This function utilizes the AWS SDK to interact with the AWS EC2 service.

2. CloudWatch Events:

   • Schedule the Lambda function to run at specific intervals using CloudWatch Events.

   • For example, you can set it to run daily or weekly to regularly clean up unattached snapshots.

3. Automation and Cost Optimization:

   • This Lambda function, triggered by CloudWatch, automates the cleanup process, ensuring that unattached EBS snapshots are regularly removed.

   • This automation optimizes costs by preventing the accumulation of unused storage resources.

This explained scenario is our main objective in this project. Let's start the project:

Create an ec2 instance

Here for this instance, we have attached a volume with id vol-0b61a32be947716c3 with 8GB capacity.

Create the Snapshot for Volume.

Go to ec2 dashboard > click "snapshot" > click "create snapshot"

Here, I've selected a volume and chosen the specific volume ID for which I intend to create a snapshot. Feel free to pick the available volume ID that corresponds to the snapshot you wish to generate.

Create lambda function

Search for "Lamda function" in the search bar > click on "lambda" >click "create lambda" then:

Author from scratch

Function name: cost-optimize-ebs-snapshot

Runtime: python 3.12

architecture: x86_64

Permissions: default

Save configuration

Now copy the code for the Lambda from GitHub. Copy the code and paste it into the

editor.

This Python script, utilizing the Boto3 library, defines an AWS Lambda function for the automated cleanup of unattached EBS snapshots. The function fetches all owned snapshots and active EC2 instances and then iterates through the snapshots. Unattached snapshots are promptly deleted, and those attached to volumes not linked to running instances are also removed. Boto3 functions such as describe_snapshots and delete_snapshot simplify AWS interactions, showcasing efficient automation for cost optimization.

save the code and deploy. Once it is deployed, click on test, and a window will open and enter the event name.

As we can see on clicking the Test, our code is failed and shows the following error.

Since the code execution timed out in just 3 seconds, so I have increased the execution time up to 10 seconds.

Remember: Minimizing the execution time is advisable, as AWS charges are influenced by the duration of execution. AWS considers the execution time as a factor in determining costs.

Let's test our code once the execution time is increased to 10 seconds.

We can see that the error indicates that the user, with the given ARN (arn:aws:sts::128571802491:assumed-role/cost-optimize-ebs-snapshot-role-g6a8dccq/cost-optimize-ebs-snapshot), lacks the necessary permissions for the ec2:DescribeSnapshots action. So we are required to attach the policy, granting permissions for the ec2:DescribeSnapshots action.

click on the Execution role, a window will open, and click "Add permissions". Click on Create Inline Policy.

Now we have chosen the policy rules including Describe Snapshots and DeleteSnapshots.

Once completing the configuration click on Create Policy and name the Policy name as below.

Now let's test the code after attaching the policy.

The error DescribeInstance shows that permission is not allowed.

Let's create a new policy that includes: DescribeInstance & DescribeVolume in the same way in which we created the Snapshot policy. Once created save the policy as below.

We can the below policies attached.

Now let's execute and view the output.

Let's conduct a successful test run by deleting our EC2 instance along with the attached volume. Afterward, we'll execute the test to verify whether the unattached snapshot gets deleted or not.

Observing the deletion of an EBS snapshot with the snapshot ID, we have successfully initiated the Lambda function manually. Now, let's streamline this process by automating the execution through the CloudWatch service.

CloudWatch is a monitoring and management service provided by Amazon Web Services (AWS). It helps you collect and track metrics, collect and monitor log files, and set alarms. With CloudWatch, you can gain system-wide visibility into resource utilization, application performance, and operational health.

To attach to the cloudwatch, search cloudwatch in the search bar and click on it.

In the Cloudwatch screen, we can see the events in the left pan and click on rules and then > "Create Rule".

Now create the rule as follows:

Click on "Continue in EventBridge Scheduler"

In this step, choose recurring schedule so that at any time if there is any unattached snapshot, it will automatically deleted once the recurring schedule is executed and for this set the cron job at what time you want the execution to be started.

Now in this step, choose the AWS Lambda as this contains our cost optimization program.

Choose cost-optimize-ebs-snapshot as the Lambda function.

Absolutely! We've successfully set up cost optimization in AWS using a Lambda function and automated it by scheduling the execution with CloudWatch using a cron job. This ensures that unattached snapshots are automatically deleted based on the specified schedule, reducing unnecessary costs and improving resource management.

After the completion of the project do not forget to terminate the services like lambda function, snapshots, and ec2 instances as they may incur higher charges.

Happy Learning!!

What is IAM?

AWS IAM is a web service that enables Amazon Web Services (AWS) customers to manage users and user permissions in the AWS cloud. It is a centralized service that enables us to manage access to AWS services and resources securely. The root user is the one who manages all the users and the permissions allowed to the permission. By using the IAM service, we can ensure that only authorized users have the right level of access to perform specific actions. In short AWS IAM does the authentication and authorization process. It defines the specific user to access the AWS services by authentication and the specific user to use the specific AWS services by authorization process.

Key Components of IAM:

• Users: Users are the persons who interact with AWS services. They are created by the root user. They can use those resources if they have permission. Permissions are the policies attached to the user by the root user.

• Groups: Groups are the collection of IAM users. If a certain group of users have the same permissions and are performing the same actions then instead of creating a policy for each user, we can simply create a group, attach the policy to the group, and finally add those specific users to this group. Groups reduce the burden of creating each IAM user policy.

• Roles: IAM roles are similar to the users, but they are not associated with specific individuals. Instead, roles are assumed by entities such as AWS services or applications, providing temporary permissions.

IAM Policies:

Policies are the building blocks of IAM. Policies define the permissions provided to the users, groups, or roles. AWS IAM uses JSON-based policy language, allowing fine-grained control over what actions can be performed on which resources. Policies must be attached else the users or any groups won't be able to use the AWS services. By default, the IAMUserChangePassword policy is attached to the user.

Let's create the users with and without attaching policies, attach them to the groups, and explore many more practically.

Log in as the Root user and create a user without attaching any policy.

In the search bar of AWS, search IAM > Users > Create user

On the next page,

User name : demo

Provide user access to the AWS Management Console - optional: allow

choose: I want to create an IAM user

Console password: Autogenerated password

Users must create a new password at next sign-in - Recommended: allow

Click Next

Set permissions: Leave as it is. keep default

click next

From the above steps, we got the following details.

On creating the user, we will get the user credentials either by copying details or downloading the credentials in a CSV file.

Note: Before logging in as a new user, as a root user I created a s3 bucket to verify whether an IAM user would be able to view it or not.

As you can see,I have two buckets in the S3

Now let us log in as an IAM user - demo-subash

note: Account ID or account alias is taken from console sign in url https://subash07.signin.aws.amazon.com/console which can be found in csv file. subash07 in the url is th account alias

change the password once you are logged in as an IAM User

Now let's try to access the s3 bucket that we created as a root user.

Search s3 in the AWS search bar > click S3

Here you can see I am logged in as subash-demo and I don't have permission even to view the created S3 buckets. Not only this, we cannot use any resources in AWS with this account. This is why attaching policy is important in AWS IAM.

We are authenticated to use an AWS account but are not authorized to use any AWS services with this account.

Let's create a new IAM user with the attached policy.

After logging in as a root user, In the search bar of AWS, search IAM > Users > Create user

Every step is the same as the previous one, only at set permissions attach the required policy. I have attached the AmazonS3FullAccess policy with this policy a user can do anything with the bucket. This is for demo only, if you are working in a company then based on the requirements you to customize or edit the policy and then attach it to the user or groups.

Create the user and save the credentials.

Log in as a testuser1.

Access the S3 bucket.

In this case, we can access the S3 buckets. Let's create a bucket as "testuser1".

Since we have full s3 bucket access we can not only create but also delete, move, and configure the bucket permissions and securities.

From the above two users created, we have attached the different policies each time. It was just for demo but when we try to implement it at the company level then it will be the burden of work just to create the user and attach a policy to them every time. This is not an effective way to do the IAM management. So to overcome these burdens, groups are created in IAM so that similar users with the same access policy will be in the same group.

Let's create a group and explore it.

After logging in as a root user, In the search bar of AWS, search IAM > Group Users > Create Group

We can see we have created the group and added the users inside the group. Furthermore, we have attached the policy to the group also. We can add the users later once the group is added and customize the policies based on the requirements of users in the group.

AWS IAM is a powerful tool for securing our AWS resources. By following best practices, staying informed about new features, and regularly auditing policies, we can create a simplified access control system. Mastering IAM is not just a best practice but it's a key pillar in building a secure and scalable AWS environment.

Happy Learning!!

What is Pod?

A pod is similar to the container but the container is created using the Docker CLI commands while a Pod is created by writing the manifest in the YAML file. Therefore, to create a pod we should know YAML.

In a single pod, single or multiple containers can be created. By the way, in Kubernetes, everything is dealt with YAML file whether it is about creating pods, service, deployment, etc.

If we create multiple containers inside the pod, then Kubernetes provides shared networking and shared storage. In this way, the containers inside the pod can communicate with each other using the local host.

Setup

Kubectl - command line tool

Minikube - local development environment, single node architecture

• To install Kubectl

sudo apt install kubectl -y

• TO install minikube

sudo apt-get install -y minikube

• To start minikube

minikube start

Our minikube is running and lets run some commands

• To get node information

kubectl get nodes

• To get Pods information

kubectl get pods

• Let's write our first pod.yml file

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

Here, this YAML code describes a Kubernetes Pod named "nginx" that runs a single container using the NGINX image version 1.14.2. The container exposes port 80, allowing traffic to access the NGINX web server within the Pod. This Pod definition is a basic configuration used for deploying a simple NGINX web server in a Kubernetes cluster.

• To run the pod

kubectl apply -f pod.yml

Our pod is created successfully.

• To get more details about the pod

kubectl get po pod_name -o wide

• To enter into the container

minikube ssh

We do not need to enter the IP address to log into the container.

We can see our first ever Kubernetes application is created and is running perfectly.

• To inspect the pod

kubectl describe pods pod-name

• To get logs of the pod for any issue

kubectl logs pod_name

In this way, we can debug and view the logs of the Kubernetes pods. Pods are simple and easy to use. To enhance with features like autoscaling, and auto-healing, we will use the services and deployments in the upcoming blogs.

Happy Learning!!

Before we start our journey with kOps, we will get into its fundamental aspects, explore key features, draw comparisons with other popular alternatives, and go through a practical example illustrating how to use kOps on AWS which will give us hands-on experience with the necessary insights and skills using this effective tool.

Development Level Environments of Kubernetes

• Minikube

• Kind

• K3S

• K3D

• Micro k8S

Production Level Kubernetes Environments

• EKS

• AKS

• GKE

• Openshift

• Rancher and so on

What is kOps?

Kops, short for Kubernetes Operations, is an open-source tool designed to simplify the process of creating, deploying, and managing Kubernetes clusters, particularly in production environments. It is the most widely used tool.

With kOps, we can automate the management of Kubernetes clusters as kOps can easily create, apply, and update cluster configurations.

Kubernetes kOps is supported by AWS Cloud, Google Cloud Platform, Microsoft Azure, Digital Ocean, etc.

Alternatives to kOps

kOps is not the only tool available for cluster management. There are several alternatives to kOps but kOps appears ahead of all the tools available. Some tools are:

• Kubeadm - kubeadm does not support the provisioning of infrastructure.

• Eksctl - Only supports AWS

• kubespray - Kubespray does not support the provisioning of infrastructure

To set up a Kubernetes Cluster in AWS with kOps

Create an ec2 instance and install the following dependencies

• Python3

• AWS CLI

• Kubectl

• An active domain with a dedicated “kops” subdomain: For this demo, we will use .k8s.local

• IAM user with below permissions

  1. AmazonEC2FullAccess

  2. AmazonS3FullAccess

  3. IAMFullAccess

  4. AmazonVPCFullAccess

Step 1: Create an IAM user called “kops” with the required permissions.

Note: If you are using the admin user, the below permissions are available by default

Now login as an IAM user i.e kops so that we will create the instance, and k8s clusters.

Once you log into your account, go to Security Credentials > generate the Access key and Secret key to configure AWS in CLI.

Step 2 Create an ec2 instance and access it using ssh protocol

Once you are logged into the ec2 instance then RUn aws configure to verify user credentials.

Note: Update the instance and Install AWS CLI in the ec2 instance

sudo apt update
sudo apt install awscli
aws configure

Install dependencies

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y python3-pip apt-transport-https kubectl

pip3 install awscli --upgrade

export PATH="$PATH:/home/ubuntu/.local/bin/"

Install kOPs

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64

chmod +x kops-linux-amd64

sudo mv kops-linux-amd64 /usr/local/bin/kops

kOps is installed successfully.

Kubernetes Cluster Installation

Create an S3 bucket for storing the KOPS objects

kOps stores its configurations, keys, and related items, in an S3 bucket to manage Kubernetes clusters. Therefore we need to create a dedicated S3 bucket for this purpose.

aws s3api create-bucket --bucket kops-subash-storage --region us-east-1

Verify it in AWS

Create the cluster

kops create cluster --name=demok8scluster.k8s.local --state=s3://kops-subash-storage --zones=us-east-1a --node-count=1 --node-size=t2.micro --master-size=t2.micro  --master-volume-size=8 --node-volume-size=8

Here our Kubernetes cluster configuration is created but not started. So to start the Kubernetes cluster run the command

kops update cluster --name demok8scluster.k8s.local --yes --admin

Output:

Run the comand

kops validate cluster --wait 10m

Wait for about 10 minutes for the cluster to come up.

When the cluster is ready, you will see output similar to this:

Since I used t2.micrwhich has less storage and computing power we could not validate the configuration

But after running the command kops validate cluster --wait 10m clusters will be created.

To get nodes

kubectl get nodes

nodes:

• Master node

• worker node

Deploy a simple workload - nginx and expose it at port 80

kubectl create deployment my-nginx --image=nginx --replicas=1 --port=80 
kubectl expose deployment my-nginx --port=80 --type=LoadBalancer

Verify Nginx is running

kubectl get pods

To get the load balancer details

kubectl get svc my-nginx

Earlier, we saw that kOps created one master and one node by default.

kops get instance group

Here, the instance group name ‘nodes-us-east-1a’ is for the node role. We can edit it and update the ‘maxSize’ and ‘minSize’ to values 3. First, open the editor with this command:

kops edit instancegroups nodes-us-east-1a

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-1-8T05:16:35Z"
  labels:
    kops.k8s.io/cluster: demok8scluster.k8s.local
  name: nodes-us-east-1a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20211118
  instanceMetadata:
    httpPutResponseHopLimit: 1
    httpTokens: required
  machineType: t3.medium
  maxSize: 3
  minSize: 3
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-us-east-1a
  role: Node
  subnets:
 —us-east-1a

Save and quit the editor. Apply the changes by running:

kops update cluster --name demok8scluster.k8s.local --yes --admin

After a few minutes, you can verify that the node count is 3.

kubectl get nodes

Delete the demo cluster and resources

Since we're operating a Kubernetes cluster in AWS, it's crucial to be aware that the underlying infrastructures such as EC2 instances and LoadBalancers—incur costs. Therefore, it's important to remember to delete the cluster once we've completed the demo to avoid unnecessary expenses.

Execute the following commands to effectively delete both the resources and the cluster:

kubectl delete svc my-nginx
kubectl delete deploy my-nginx
kops delete cluster --name demok8scluster.k8s.local --yes

In this way, we can use kOps for the different operations in Kubernetes. Furthermore, to know about kops CLI usage use this link https://kops.sigs.k8s.io/cli/kops/

Therefore, kOps makes managing our Kubernetes cluster on AWS easy by automating the setup of necessary resources like instances and load balancers. It strikes a balance between control and simplicity, simplifying our cluster management tasks.

Thank you!!

Happy Learning!!

GitHub:

GitHub is a web-based platform for version control using Git. It provides different features such as code hosting, version tracking, and collaboration tools like pull requests and issues.

Jira

Jira is a versatile project management and issue-tracking software developed by Atlassian. It is widely adopted in software development as Jira allows teams to plan, track, and manage their work efficiently.

How we will integrate Github with Jira?

So, in this project, we will use Python API, and webhooks to communicate between GitHub and Jira.

Whenever an issue is created in the GitHub repository, webhooks tell GitHub to monitor a particular comment for eg: "/Jira" done by the developer or owners. Then the JSON parser provides the JSON file to the Python application which we created as an API whose URL is configured inside the webhooks. Then the Python API calls the Jira API by sending the JSON which was received from GitHub. ON receiving it, Jira creates the ticket automatically.

This streamlined process ensures that when developers or owners add a specific comment in GitHub, the Flask app acts as a bridge to communicate with Jira, triggering the automatic creation of a corresponding ticket.

Setup requirements:

Jira Setup

Flask app

ec2 instance

Github webhooks

Jira API

• Setup Jira

Anyone can simply create an account at https://www.atlassian.com/software/jira. Once creating the account click on your profile and choose Jira as the software.

• Then click Get it for free

• A page will open and we do not need to do anything and simply Click on next.

On the next page enter the email and site name, the site name is automatically generated or you can customize the site name as per your desire.

• After this choose Scrum as the template

• And Now choose "Select a team-managed project"

• Add the project details

• After configuring all these details, now our project is created.

Now we have created our project and the Jira setup is done. To integrate our project with GitHub or to make communication between the two systems we need the API tokens.

Let's create the API token. In your profile Go to Security> click Create and manage API tokens >create API> copy the API token

Note: Keep safely the API token, the same token can't be generated again, else we need to generate the new API token.

We have created the API token successfully.

• From Jira API documentation, we need to get the Python code to create an issue.

import requests
from requests.auth import HTTPBasicAuth
import json

url = "https://your-domain.atlassian.net/rest/api/3/issue" #add your url

auth = HTTPBasicAuth("email@example.com", "<api_token>")

headers = {
  "Accept": "application/json",
  "Content-Type": "application/json"
}

payload = json.dumps( {
  "fields": {
    "description": {
      "content": [
        {
          "content": [
            {
              "text": "My First Jira Ticket",
              "type": "text"
            }
          ],
          "type": "paragraph"
        }
      ],
      "type": "doc",
      "version": 1
    },
    "issuetype": {
      "id": "10007"
    },
    "project": {
      "key": "SUB" #add your key
    },
    "summary": "First Jira Ticket",
  },
  "update": {}
} )
response = requests.request(
   "POST",
   url,
   data=payload,
   headers=headers,
   auth=auth
)
print(json.dumps(json.loads(response.text), sort_keys=True, indent=4, separators=(",", ": ")))

I have removed the lines from the code as we don't need it. We only needed the project type*, issue type*, and summary* as they are the compulsory fields that need to be completed.

Note: In the above code insert the url replace "https://your-domain.atlassian.net", insert your email address used to login jira and finally the api token that you had earlier created, in project key, check the project it will be at the side of you project name inside (), and for the issue type, check their ids from project>board>3 dots(...)>configure board>issue types>choose any option>check id in url

• Once this is done, we can test our code whether the ticket is created or not.

• Our code is successfully run and the ticket is created and now let's verify it in the Jira.

Here we can see the tickets being created. By the way, there are three tickets because I executed the code three times that's why three tickets are generated.

Our Jira API is finely working and created our first Jira ticket, now let's create a Python Flask app framework

note: It is a skeleton of the main code

import flask from Flask
import requests
from requests.auth import HTTPBasicAuth
import json
app = Flask(__name__)
"@app.route("/createJIRA", methods=['POST']"
def createJIRA():
  #jira_code here

app.run('0.0.0.0', port=5000)

• You can access the source code here in this GitHub link.

Now our Python application is also ready. We will create an ec2 instance in the AWS cloud.

• Access the instance using SSH protocol as our instance is ready

ssh -i ~/path-to-pemfile/pem_file user_name@ipv4address

• Create our flask application inside the instance along with .env file to store our API token details.

Since the API token is very sensitive content, we need to keep it secure so that no one can misuse it.

• Let's see whether our flask app runs or not.

Great! Our app is perfect working and now verify it using the GitHub Webhooks.

To create a webhook, first, we need to choose the repo for which we need to configure the webhook. In the same repository window, click on settings > click on webhooks > click on Add Webhooks.

• Then the following page will open.

In this page enter the URL of your Python application and in Let Me select individual events choose ISSUES. Click on Add Webhook.

• Check whether the GitHub actions passed or not.

Note: Before testing the Github actions , you need to set the inbound rules for port 5000 in your ec2 instance otherwise the test will fail as it won't be able to access our python application as it is in the ec2 instance.

Success!!

• Let's do a comment on an issue and check in Jira whether a ticket is created or not.

• Now let's verify in Jira.

As there are multiple tickets but the latest ticket is SUB 5 which is created just a minute ago. This means our implementation is perfectly done.

Webhook actions show the issues created and opened as we opened it in Jira.

What if the developer comments something else in the issues, then the Jira will create a ticket if we don't mention our logic while making the API call.

 # Check if the condition is true before making the API request
    if "/jira" in request.json["issue"]["body"]:
        # Perform the API request only if the condition is true
        response = requests.post(url, data=payload, headers=headers, auth=auth)        
    else:
        print("Comment should be /jira only")

We have added this logic so that whenever there is a comment in the issue, it verifies whether the comment is "/jira". If yes then only the API call will be made and Jira creates a ticket otherwise if there is another comment it will notify telling that the comment should be only /jira.

Congratulations, we have successfully implemented the project. Using different technologies like Python, Flask, and webhooks, we've streamlined the process, promoting efficiency. We have also performed security practices by keeping sensitive information hidden.

Happy coding! Keep learning!!