We see on some Kubernetes cluster upgrading from 1.30 -> 1.31 following errors on cilium, coredns, kube-proxy, ... pods on Control Planes:

Warning  Failed     15s (x3 over 12s)  kubelet            Error: services have not yet been read at least once, cannot construct envvars

The pods will not start on the updated Control Plane, so we must do it with a little workaround to ensure a seamless upgrade.

Upgrade the Cluster without errors

First of all if you face that problem, no worries, you can easily rollback the kubeadm/kubelet and patch it then. You can also exchange the kubectl but its not needed.

Rollback of kubeadm (Debian based OS)​

Redownload the old kubeadm/kubelet and restart with systemctl:

# Rollback to 1.30
wget -P /usr/local/bin https://dl.k8s.io/release/v1.30.X/bin/linux/amd64/kubeadm
wget -P /usr/local/sbin https://dl.k8s.io/release/v1.30.X/bin/linux/amd64/kubelet
systemctl restart kubelet

Upgrade the Cluster before update the binaries​

Download the new kubeadm to your home directory and rename it to avoid confusion:

wget https://dl.k8s.io/release/v1.31.X/bin/linux/amd64/kubeadm
mv kubeadm kubeadm-v1.31.X
chmod +x kubeadm-v1.31.X
./kubeadm-v1.31.X   upgrade apply -y v1.31.X

If you have a cluster with more then one Control Plane update all Control Plane Nodes first before exchanging the binaries and reboot the Nodes.

And thats it !

If you still have problems with upgrading your Kubernetes clusters let us know. We can help you!

In 2023, Clyso released the Ceph Analyzer, giving your operations teams a great tool for inspecting the health of your Ceph clusters, offering in-depth reporting and recommendations to fix many non-trivial issues. Two years later, we are pleased to announce the release of Clyso Kubernetes Analyzer!

Get your 30 days of Kubernetes Analyzer now!

What does Clyso Kubernetes Analyzer do?​

Is your Cluster in a good shape, or do you think it is? We will check it for you!

Features:

• Comprehensive Cluster Inspection: Perform a full inspection of all cluster components with a single command.
• Pod Inspection: Retrieve detailed information about pods, including container statuses and logs from the last restart.
• Node Inspection: Gather system information and statuses from all nodes in the cluster.
• Component Inspection: Inspect critical Kubernetes components like CoreDNS, etcd, CNI, and CSI.
• Certificate Expiration Verification: Check the expiration dates of Kubernetes certificates to prevent unexpected outages.
• Health Checks: Perform health checks on cluster components to ensure they are functioning correctly.

The Analyzer will check your cluster and give you a report with recommendations on any problems it found.

Required Permissions:

To run the Full System Analysis Tool, the following Kubernetes permissions are required. You can create a ClusterRole with the necessary permissions using the following YAML configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: full-system-analysis-role
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["pods/exec", "pods/log"]
  verbs: ["create", "get"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["services", "endpoints"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses", "csidrivers"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets", "daemonsets"]
  verbs: ["get", "list"]

This ClusterRole grants the necessary permissions to inspect and manage various Kubernetes resources.

How to use it​

1. Grab you a free Demo from here
2. You will receive a .zip file that you have to unzip:

$ unzip fsa_tool_clyso_0.16.0_linux_amd64.zip
 LICENSE
 config
 fsa

3. Now you see a fsa and a config file. This two files you have to copy on a control-plane node where it can execute kubectl and kubeadm or if not possible (f.e. on Talos Linux) to a machine that can reach the cluster and has a configured kubectl.
4. Create a config directory on your "$HOME" or your choosen user and copy the config file into it:

## Create the directory
$ mkdir $HOME/.fsa

## Copy the config
$ cp config $HOME/.fsa/config

5. Copy also the fsa Binary to your place where you like to execute it:

## Copy the fsa
$ cp fsa $HOME/fsa

6. Execute now the Binary, this will generate a file called report.json:

$ ./fsa inspect all -o json
...
JSON report written to report.json

7. Upload the report.json to https://analyzer.clyso.com/#/analyzer/kubernetes
8. Check your report and improve your Cluster!

Let us know if you are missing something or find improvements!​

Summer, an ideal time to grow some containers! At Clyso, we are using Gardener extensively, with Clyso Linux by Garden Linux as base images for nodes and containers.

Composing an image​

Clyso Linux is based on Debian Testing. The distribution’s repositories provide a curated set of base packages and can be installed and managed with the usual apt and dpkg utilities.

$ podman run --rm -it ghcr.io/gardenlinux/gardenlinux:1604.0 sh -c 'apt update && apt list | wc -l'
...
1994

$ podman run --rm -it debian:testing-20240812-slim sh -c 'apt update && apt list | wc -l'
...
66400

By comparison, Garden's repositories are much smaller than Debian's, and it is often necessary to add extra repositories when installing packages:

echo 'deb http://deb.debian.org/debian testing main' > /etc/apt/sources.list.d/debian.list
apt update

See gardenlinux/builder#31 issue for details about repository changes.

Building an image​

The easiest and most idiomatic way to build a Clyso Linux based container image is to write a Dockerfile and use docker/podman/buildah or similar. See available gardenlinux base images.

Another way is to use gardenlinux/builder build tool. See Getting started docs on how to set up a build project and its features. Features then entirely replace the Dockerfile, as all build steps are specified there. To finally build the image, run:

feature='<Feature name>'
out_dir="<Path where to store build artifacts>/${feature}"
# Build the image. We are in the gardenlinux base directory.
TYPE=container build --target "${out_dir}" "${feature}"
# Now we just need to import the .tar OCI archive. For this example we are assuming amd64 arch.
podman import "${out_dir}/container-${feature}"-amd64-*-local.tar my-image

Note that the resulting image consists only of a single layer and is therefore not the best choice for normal use. Nevertheless, this is a handy way to test things before creating node images, which is the best use case of the tool.

Happy gardening!

We habe backuped kubernetes cluster before using Ark.

The Ark project was then acquired by VMWare circa 2019 and renamed Velero.

Currently, we are supporting another customer in backing up data stored in multiple Kubernetes environments to be ready for disaster recovery.

Today, Kubernetes is the first choice for running microservices in the public or private cloud. More and more developers and enterprises are building their applications on the modern microservice architecture.

Many of them are using Kubernetes for automated deployment of their workloads and want to benefit from the new flexibility and robustness. We are working on a solution for our customers to simplify and unify Day One and Day Two operations in their operations. With the increasing number of clusters, the management, updating and monitoring should be able to deal with it efficiently.

Since 2018, we have been accompanying Rook.io in its development and had direct exchanges with various members of the project at Cephalocon in Beijing 2018 and Barcelona 2019.

In 2019, we began serving customers in production who use Rook.io to manage Ceph.

Storage Operators for Kubernetes

Rook transforms distributed storage systems into self-managing, self-scaling and self-healing storage services. It automates the tasks of a storage administrator: provisioning, bootstrapping, configuring, deploying, scaling, upgrading, migrating, disaster recovery, monitoring, and resource management. Rook leverages the power of the Kubernetes platform to deliver its services to any storage provider through a Kubernetes operator.

[https://rook.io/](https://rook.io/)

Since 2020, we are now working on improving the automated operation of Ceph with Rook.io. Furthermore, it is planned to have the platform fully audited by various certification bodies.

After the acquisition of CoreOS by RedHat and the discontinuation of CoreOS support.

CoreOS Container Linux will reach its end of life on May 26, 2020 and will no longer receive updates.
(source: https://coreos.com/releases/)

We have decided together with our customer to provide an alternative to CoreOS before the 26.05.2020.

The current project name is Gardenlinux, although the name may not change until the public release.

Gardenlinux builds a full replacement for CoreOS based on Debian, without being biased by a target architecture.

Currently, the project supports the following platforms: BareMetal, AWS, GCP, Azure, VMWare, Openstack, and KVM and Docker.

Ports for AlibabaCloud are still under development.

In Produkton, Gardenlinux has already proven itself on BareMetal, KVM and AWS.

Even though microservices with Kubernetes are gaining traction in the cloud environment, we still have a high demand to serve for managing virtual machines.

To keep the technology stack as lean as possible, we are phasing out our Cloud Controller environments and managing Virtual Machines using Kubernetes.

In a further step, we are thereby able to use Kubernetes to map complete environments with microservices and virtual machines through one technology.

A customer approached us to help them build their new IaaS platform.

We helped him to evaluate the current situation and worked out a concept together with the customer from the selection of the location, the necessary hardware to the operation and implemented it together within a month.

On behalf of the customer, we provided the optimal Kubernetes platform for ONAP as a managed service.

ONAP is a comprehensive platform for orchestration, management, and automation of network and edge computing services for network operators,

cloud providers, and enterprises. Real-time, policy-driven orchestration and automation of physical and virtual network functions enables rapid automation of new services and complete lifecycle management critical for 5G and next-generation networks.