Working with Objects inside Kubernetes

Learning about Kubernetes Objects

18 min readAug 1, 2021

If you have deployed or managed any application in Kubernetes, you have definitely interacted with at least one Kubernetes object, knowingly or unknowingly. Kubernetes has something called Objects in it’s architecture. Any action that a user or Kubernetes takes, is on a single or multiple objects to achieve the end goal of the action.

Topics to be covered

What is a Kubernetes Object
Object segregation inside the Kubernetes API definition
Application Deployment
Working with Objects
What Next?

What Is a Kubernetes Object?

A Kubernetes object is a “record of intent” — once you create the object, the Kubernetes system will constantly work to ensure that object exists. By creating an object, you’re effectively telling the Kubernetes system what you want your cluster’s workload to look like; this is your cluster’s desired state.

Let’s take an example to understand this better. Say you have to deploy a simple application which prints a “Hello World!” message when the endpoint is invoked. The simplest way in the Kubernetes definition you can deploy a Pod which has the application code packaged in a Dockerfile. Ideally you will never use a Pod template directly, rather use a Deployment object for additional features that Kubernetes offers, but we’ll take a simple example now.

apiVersion: apps/v1
kind: Pod
metadata:
  name: hello-pod
spec:
  containers:
  - name: hello-world 
    image: hello-world:latest #image name in the docker registry
    ports:
    - containerPort: 80 #port to run the app into

If you take this yml definition (pod.yml) and run the below command, you will be able to deploy a pod with the required image on port 80 of the container

$ kubectl apply -f pod.yml

Congrats, you are able to deploy your simplest pod inside Kubernetes. If you run the get command, you will be able to see the deployed pods:

$ kubectl get pods
NAME               READY   STATUS       RESTARTS       AGE
pod/hello-pod       1/1    Running         0           18s

But wait, are you able to access it via the browser, or can curl to the required endpoint? No, you cannot since you do not know the address. So you need additional Kubernetes objects like Services, Ingress to expose the application to the end user. We’ll see the specifics later in the article, but hope this gives a basic idea about a Kubernetes object in general.

For any feature like networking, services, confidential data management, auto-scaling, defining networking capabilities, persistent data storage or others, Kubernetes offers an object definition to be configured and used to yield the expected output. There are some bare-minimum configurations required for each object which allows Kubernetes to take the required actions. Below is defined a basic template of an object in Kubernetes:

apiVersion: apps/v1    #kubernetes api version
kind: Deployment       #Kubernetes object name
metadata:              #metedata related to the object
  name: hello-world
spec:                  #object definition to be defined
  replicas: 2
status:                #current status of object 
  hostIP: 169.25.16.215

apiVersion: Just like any other open-source project, Kubernetes also follows a versioned manner of it’s feature release or bug-fixes. There can be alpha,beta,stable version of an object with major or minor differences in implementation. Hence one needs to specify the apiVersion to be used so that Kubernetes can use the version specific definition and implementation of that object.

kind: It is the name of the object in Kubernetes. For ex: Deployment

metadata: Metadata definition related to the object needs to be specified here. Values such as name, labels, clustername can be specified here. These metadata values can be used when we try to reference an object inside any other object. For ex: when we create a service to abstract the underlying pods, we can specify the pod metadata inside the service selector to grab only the pods, with a specific metadata value.

spec: This is where the object definition is defined by the user. Anything and everything that the user desires for the object in the end state has to be defined here. Some parameters may be mandatory for an object, and differs from object-to-object. For ex: If a user wants 2 replicas of a pod in a Deployment object, specify replicas: 2 inside the spec of Deployment object.

status: This part describes the current state of the object, which is updated by Kubernetes system and its components. The control plane constantly monitors each objects current status and tries to match it to it’s desired state.

The entire definition of all Kubernetes objects can be found here.

Object Segregation inside the Kubernetes API Definition

The Kubernetes ecosystem itself provides a whole bunch of functionalities in all aspects of an application deployment and management. On top of this, there are some other projects which offer a pluggability inside the Kubernetes ecosystem. For example: Flannel, Project-Calico, Canal and WeaveNet are networking tools that can be plugged into Kubernetes via the Container Network Interface and provide flexibility. More information around installation and configuration can be found here. So the objects inside the Kubernetes ecosystem are segregated and logically grouped. Below is a diagram which shows the logical segregation of the API catalog:

There are more such logical grouping, a complete list can be found here. When we specify the apiVersion value in an object definition, the grouping value is itself a part of the apiVersion, so that’s another way to find the group. One another reason for grouping objects is that, the objects within a group are more inter-related in terms of a common functionality. For ex: objects with the networking group are all concerned with providing a networking capability. So in cases of a new feature or a bug, the underlying objects in a group mostly the ones to be affected. So this offers a config-driven approach while coding for the group objects.

Application Deployment

The diagram below shows a deployment model of an application in Kubernetes. You may/may not use all objects defined below based on your use-case.

We’ll describe the objects in this deployment, which object is responsible for what part in the application, so later when we deep dive into individual objects, it it more relatable.

Namespace is a logical boundary which one defines rules and boundaries for applications deployed inside it. We’ll see these rules below under Network Policy and Resource Quota.
Deployment is the main object which abstracts ReplicaSet (used to specify no of replicas of application) and in turn Pod. The Pod object describes the container level details of container registry image, port to run app on, container level resource utilizations, probes, environment variables to use, volumes and more.
Service is used to abstract individual pods from interacting directly with external requests. A service object keeps check of health of underlying pods, and forwards requests from end users to the pods. A good reason to use service is not to deal with ip level details of pods. The service object deals with networking with pods, and in case pod ips change due to any reason, service object can handle that.
An Ingress object defines the domain name and port details on which end users can access the application with. A request from end user passes through the network policy filter defined for the application. If validated, only then the request is allowed to go further. Network policy filter defines the IP’s or domains which can access the application tier.
HPA or Horizontal Pod Autoscaler is the monitoring object, which monitors resource utilization for the pods, and performs scaling up/down depending upon the resource utilization. One has to specify a threshold value depending on which pods may be created or killed.
Any confidential data can be put into a Secret object. Depending on the need of that data, it can be either used as a Volume Mount or directly as an environment variable. Another common use of a secret is to pull images from a container registry. This secret will have the necessary information to get images from a registry.
ConfigMaps are where you can define your configuration values. ConfigMaps can be created using manual input of the key-value pair or can be generated from a file. File based approach is the better one, since you can have values changing based on your environment, just define your environment specific file and point it to the configmap. Similar to secrets, configmaps can also be used either as a volume mount or directly as an environment variable.
Persistent Volume Claims (PVC) is a request object for use by a user. Just like pods consume node resources, PVCs consume Persistent volume resource and access while reading/writing data in volumes.
Network Policy is a list or rules to allow specific domains/ip address to allow connectivity inside a pod. Only requests that meet the filter rules of a network policy can go further (both incoming and outgoing) else they get terminated at the namespace boundary itself.
Resource Quota defines the permissible limits that is allowed for a namespace. An application should not infinitely utilize resources. Imagine you have a Kubernetes cluster running, and have multiple application deployed on the cluster, each in a different namespace. For some reason, if one of the application misbehaves and tries to consume all resources on the cluster, other application will face resource constraints and will be affected. If there is an upper boundary defined for each application at namespace level, it will not affect any others at least.
Although one may define resource quota at a namespace level, a pod or container can still try to consume everything available in the resource quota. To control resource utilization at a pod/container level, Limit Ranges can become handy. One can also specify storage limits for a PVC in a namespace and define a ratio b/w request and limit for a resource in a namespace.
Container Registry is the place where your images are stored and can be retrieved during deployment. The most widely used is Docker registry, and is provided by Docker. Other providers do include Google, Amazon, Microsoft, Oracle, JFrog and many more.

Working with Objects

Deployment

Deployment object abstracts out ReplicaSet object and in turn Pod template. Pods are the smallest atomic level in Kubernetes where containers run. A pod allows sharing of file system, network interfaces etc among the containers using linux namespaces, cgroups and other kernel features. We know that pods are mortal, once they die, it’s their end. ReplicaSet provides a mechanism to monitor and maintain the required number of pods via the replication controller. In this way, one can ensure prevention against failure and high availability. Hence in a stable and important application, one would never use pods directly, but via a deployment object.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-deployment
  labels: 
    app: hello
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello
  template: 
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello-app
        image: hello-world:latest
        ports:
        - containerPort: 80

The above snippet shows a deployment configuration for a simple app, that would run 2 replicas of a hello-world image on the container port 80. Simply running the below command would deploy 2 pods running hello-app.

$ kubectl apply -f deployment.yml$ kubectl get all
NAME                         READY   STATUS       RESTARTS       AGE
pod/hello-pod-85851-vg12a     1/1    Running         0           18s
pod/hello-pod-85851-bh27a     1/1    Running         0           18sNAME                           READY  UP-TO-DATE   AVAILABLE    AGE
deployment/hello-deployment     1/1       1            1        19sNAME                                DESIRED   CURRENT  READY    AGE
replicaset/hello-deployment-85851      1         1       1       19s

One important parameter here, in fact in all Kubernetes objects is labels defined for an object. Labels are simple identifiers for an object but very powerful when trying to interconnect different objects. One great article explaining labels and selectors is here.

Service

Firstly you would wonder why to use a service object. Why can’t users directly talk to pods that serve the underlying application. Well here’s the reason : Pods are ephemeral in nature. A deployment can create and delete pods dynamically. Since each pod running in a deployment has it’s own IP address and the set of active pods change all the time, so do the IPs. So the problem is where would the user hit the application. A Service object abstracts the set of running pods and directs users to healthy pods thus serving as a single interface for the users. In other words, if you are familiar with Service Discovery and Load balancing concepts, the Service object implements service discovery for your application. The below snippet shows a configuration setup for a service.

apiVersion: v1
kind: Service
metadata:
  name: hello-service
  labels: 
    app: hello
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: hello

Run the below command to apply the service object, and that would apply to the pods that were running with the above deployment.

$ kubectl apply -f service.yml$ kubectl get service --show-labels
NAME                      TYPE      CLUSTER-IP    EXTERNAL-IP  PORT(S)      AGE      LABELS
service/hello-service   ClusterIP  189.12.114.124   <none>
80/TCP       8s      app=hello

The service object created now has abstracted the pods with the label app=hello. Now any external requests go via the service discovery mechanism provided by the service object.

In the previous article, we briefly mentioned a component called kube-proxy that is present on every worker node. Well, this is the backbone of how service are implemented and work inside Kubernetes. Kube-proxy runs in three different modes: userspace, iptables and IPVS, with iptables being the default.

In userspace mode, kube-proxy itself acts as a proxy server and delegate requests accepted by ip table rule to the backend pods. This adds an additional hop to the message flow. In the iptables mode, the kube-proxy creates a set of iptable collection rules to forward requests received from clients directly to the underlying pods on the network layer, thus reducing the additional hop from userspace mode. In the IPVS mode, the kube-proxy makes use of the IPVS based virtual server for request routing without using iptables. IPVS is a transport layer load balancing feature which is available in the Linux kernel based on Netfilter and provides a collection of load balancing algorithms. The main reason for using IPVS over iptables is the performance overhead of syncing proxy rules when using iptables. When thousands of services are created, updating iptable rules takes a considerable amount of time compared to few milliseconds with IPVS. More information on IPVS mode can be found here.

Ingress

There are 2 things that serve for the Ingress functionality. First, Ingress is an object that allows to Kubernetes cluster from outside the cluster. One can configure access by creating a collection of rules that can define which inbound connections reach which services. Ingress can be configured to give Services externally reachable URLs, load balance traffic, terminate SSL/TLS. One can also simply configure an HTTP load balancer for applications running on Kubernetes, represented by one or more Kubernetes internal services. Second, Ingress controller is an application that runs in a cluster and configures an HTTP load balancer according to Ingress resource. The load balancer can be a software load balancer running in the cluster or a hardware or cloud load balancer running externally. Different load balancers require different Ingress controller implementations.

Ingress offers an API object that can manage external access to the services in a cluster, usually HTTP. Sometimes is mistaken or thought of as a Service type, which it is not. It’s basically an entry point into the cluster. Ingress allows to consolidate routing rules into a single resource and expose multiple services with the same load balancer. Below diagram shows how to link multiple services under ingress.

The below snippet shows how to link services to Ingress in the above diagram via the yml definition.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ingress
spec:
  rules:
  - host: abc.com
    http:
      paths:
      - path: /path1
        pathType: Prefix
        backend:
          service:
            name: path1-service
            port:
              number: 5000
      - path: /path2
        pathType: Prefix
        backend:
          service:
            name: path2-service
            port:
              number: 5002
  - host: xyz.abc.com
    http:
      paths:
        path: /
        backend:
          service:
            name: xyz-service
            port:
              number: 9000

Horizontal Pod Autoscaler

If you have been an IT engineer in the pre-cloud era, you will know how many times we used to over-provision infrastructure resources to cater to high traffic needs that may be expected once in a blue moon. For high availability and resiliency, big honking compute resources were underutilized. Nonetheless to say, the additional dollars that were spent for these. With the cloud evolution, this bad practice is coming to an end. You use how much you need at a time and you pay only for that. Scale resources up and down on the fly. All major cloud products provide auto-scaling functionality and Kubernetes is no exception. The way Kubernetes facilitates this is via the Horizontal Pod Autoscaler object. All one needs to do is create a configuration with values like resource thresholds, minimum and maximum resources and link it with the correct scaling resource. That’s pretty much enough to get started and see it in action. Below is a simple snippet to illustrate this.

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: hello-hpa
spec:
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageValue: 0.5
        type: AverageValue
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello-deployment

This snippet defines a configuration with a minimum 2 and maximum 5 instances of the pod defined in the hello-deployment object. The scaling action should take place when the average cpu usage is 0.5 cpu cores. The way how Kubernetes calculates usage is out of scope for this article, but you can refer here for the exact math. The HPA is implemented as a control loop with a period controlled by the control manager. Each interval, the control manager fetches the usage metrics from the Metrics server and decides based on thresholds if a scaling action is needed.
To see the HPA in action run the below command, this will give a streaming output of current status of the HPA object, and burst request to your ingress endpoint via tool like JMeter.

$ kubectl get hpa -wNAME       REFERENCE                    TARGET     MINPODS  MAXPODS  REPLICAS   AGE
hello-hpa  Deployment/hello-deployment  100m/500m  2        5
2          5s
hello-hpa  Deployment/hello-deployment  168m/500m  2        5
2          12s
hello-hpa  Deployment/hello-deployment  292m/500m  2        5
2          19s
hello-hpa  Deployment/hello-deployment  438m/500m  2        5
2          28s
hello-hpa  Deployment/hello-deployment  497m/500m  2        5
2          32s
hello-hpa  Deployment/hello-deployment  575m/500m  2        5
2          39s
hello-hpa  Deployment/hello-deployment  593m/500m  2        5
3          45s
hello-hpa  Deployment/hello-deployment  525m/500m  2        5
3          55s
hello-hpa  Deployment/hello-deployment  489m/500m  2        5
3          63s
hello-hpa  Deployment/hello-deployment  452m/500m  2        5
2          75s

You’ll see the resource utilization going up and once it crosses threshold, a new pod will be spun up and scaled down upon thresholds met (indicated by the bold lines above).
Here we have used the resource metrics for scaling, but custom metrics can also be used in HPA to meet application specific scaling requirement. This repo shows a working example of how to deal with custom metrics.

Secret

In any application, be it an enterprise or a simple blog, one might have some data which is confidential and cannot be checked into source code. This can include API keys, SSL keys, any password and many more. Kubernetes provides a mechanism to deal with such data via an object called Secret. One can by self create a secret object or use the kubectl command line to create a secret object and apply to a namespace. The most widely used in the kubectl command line to generate secrets. Let’s say you have a file which has secret data and you wanna generate a secret object out of it. This can be achieved by the below command:

#This will directly create a secret object
$ kubectl create secret generic hello-secret --from-file=secret-data1.txt,secret-data2.txt
#This will create a file with the secret object which can then be applied via kubectl command
$ kubectl create secret generic hello-secret --from-file=secret-data1.txt,secret-data2.txt -o yaml | grep -v creationTimestamp secret-object.yml$ kubectl apply -f secret-object.yml

This creates a secret of type Opaque which is generally a user-defined secret value needed. There are also other standard secret types defined which are used for very specific purposes. Also the data in the secret object can be encoded or plain text as defined by the secret type definition. Now the secret is created; great, but how do we use them inside our application. If you refer the Application Deployment diagram above, you’ll see a Volume Mount resource via which Secret is connected to the Deployment and in turn Pods. One way is this. We have to create a volume to reference the secret by it’s name and then a volume mount inside the container where that file or data needs to be mounted to. Once defined, the secret data will be present in the container location specified by the volume mount. Now you can use it as it’s locally present on the container. Alternatively, one can also use the value from secret object to create an environment variable and then reference it out where needed. The below snippet shows the configuration inside a Pod template:

apiVersion: v1
kind: Pod
metadata:
  name: hello
spec:
  containers:
  - name: hello-app
    image: hello-world:latest
    volumeMounts:
    - name: secret-data
      mountPath: "/etc/secret/"
      readOnly: true
    env:
    - name: SECRET_VARIABLE
      valueFrom:
        secretKeyRef:
          name: hello-secret
          key: secret-data1
  volumes:
  - name: secret-data
    secret:
      secretName: hello-secret

This puts the required secret file under the /etc/secret/ folder inside the pod with only read permission. Now the file can be read from the application code and configured with the secret data. Alternatively an environment variable SECRET_VARIABLE can be created to reference out data from the secret object.

Config Maps

A normal and good practice while application development is to set configuration data separate from code. This is good because one can manage configuration for different environments in separate files and use them as and when desired and also avoid build cycles while changing config data for testing purposes. Now there are several ways of generating a config-map. Create a config-map object manually for the configuration data you have or use a kubectl command line or use config-map-generator. We’ll see them all

1. Create a config map yml file by referencing the API definition here. Apply that file via kubectl
$ kubectl apply -f config-map.yml2. Generate config map via kubectl from a file
$ kubectl create configmap app-config --from-file=configuration.properties3. ConfigMap Generator via kustomize.
configMapGenerator:
 - files:
   - configuration.properties
   name: app-config

The 3rd method is used when using kustomize to manage all object during a deployment object. Simply reference the file into the configMapGenerator parameter and kustomize will work to generate a config map dynamically at run time.
Accessing values from a config map is identical to accessing values from secrets. Can be done via a volume mount or a reference with an environment variable. Below snippet shows both mechanisms

apiVersion: v1
kind: Pod
metadata:
  name: hello
spec:
  containers:
  - name: hello-app
    image: hello-world:latest
    volumeMounts:
    - name: config-data
      mountPath: "/etc/conf/"
      readOnly: true
    env:
    - name: CONFIGURATION_VARIABLE
      valueFrom:
        configMapKeyRef:
          name: hello-config
          key: config-data1
  volumes:
  - name: config-data
    configMap:
      name: app-config

With volume mount, the entire configuration file gets placed into the /etc/conf/ directory and can be read by the application. With environment variable, individual values from the config-map can be used as required.

Network Policy

One would like to control who and how can access your application. If you have an internal enterprise application, you would not want any requests coming from outside your enterprise’s network to come in. Alternatively, if you have an open ended application, you might want it to be accessible on the open internet. Sometimes you will also go on a hybrid approach between these depending on use case. Network policy allows you to define a set of networking rules which say who and how can access your application. These are essentially firewall rules specific to your namespace where application is deployed. There are broadly 2 things to be defined: what requests are allowed to come inside the application (Ingress) and what are allowed to go outside the application (Egress). Under these we define the protocol, hosts, ips, ports, domains which should be allowed or denied. You may come across different default network rules, either by a cluster admin or a cloud provider where things may be deny-all or allow-all. This can be configured, but depends on your Kubernetes solution provider.
Apart from Kubernetes itself, there are various other pluggable APIs like Calico, Romana, Flannel which provide networking capability inside a Kubernetes cluster. The usage of these external solutions is dependent on integrating them via the Container Network Interface if you are doing yourself, or if you have an admin maintaining your Kubernetes cluster, then which integrations are done and supported by them. Cloud providers can have some custom implementation as well. Below we’ll see a simple snippet using the Kubernetes native networking solution

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hello-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: hello
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 196.198.12.0/8
        except:
        - 196.198.12.43/24
    - podSelector:
        matchLabels:
          app: xyz
    ports:
    - protocol: TCP
      port: 9000
  egress:
  - to:
    - ipBlock:
        cidr: 193.173.0.0/16
    ports:
    - protocol: TCP
      port: 8000

Once defined and applied this policy on a namespace, any incoming or outgoing request gets filtered through this networking rule and then is allowed or rejected from going further.

Resource Quota

Defining resource utilization limits is not mandatory but a good practice to ensure fair utilization between multiple applications that might be running on the same cluster. Just in case, a single application misbehaves and tries to consume all resources available in the cluster, other applications should not be affected by this. We try to limit resources available to a single namespace to ensure fair and optimum utilization of the cluster.

What Next ?

This is just the beginning of your Kubernetes journey. There is more to all objects that we have discussed here. The entire API reference and usage can be found here. All usage requirements and field descriptions are mentioned here with possible enum values and defaults wherever applicable.
One more thing that one needs is debugging effectively and properly while building applications in Kubernetes. Here is one reference visual that shows steps and process on how to debug Kubernetes objects.