# Kubeeye | Tool To Find Various Problems On Kubernetes, Such As Application Misconfiguration, Unhealthy Cluster Components And Node Problems **URL:** \ **Category:** Tools & Scripts\ **Tags:** tools, programming, freebies\ **Created:** [November 27, 2022, 9:43pm UTC](https://onehack.st/t/kubeeye-tool-to-find-various-problems-on-kubernetes-such-as-application-misconfiguration-unhealthy-cluster-components-and-node-problems/228983 "2022-11-27T21:43:04Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![NoBody](https://onehack.st/user_avatar/onehack.st/nobody/32/163880_2.png) [@NoBody](https://onehack.st/u/NoBody)\ **Post date:** [November 27, 2022, 9:43pm UTC](https://onehack.st/t/kubeeye-tool-to-find-various-problems-on-kubernetes-such-as-application-misconfiguration-unhealthy-cluster-components-and-node-problems/228983/1 "2022-11-27T21:43:05Z")
[![](https://github.com/kubesphere/kubeeye/raw/main/docs/images/KubeEye-O.svg?raw=true)](https://github.com/kubesphere/kubeeye/blob/main/docs/images/KubeEye-O.svg?raw=true) [![](https://github.com/kubesphere/kubeeye/workflows/CI/badge.svg?branch=main&event=push)](https://github.com/kubesphere/kubeeye/actions?query=event%3Apush+branch%3Amain+workflow%3ACI+) [![](https://camo.githubusercontent.com/9f789c2e6184d4be4297fc84538d7a9cbb77fa71b0aedbb167d2b0488c21ef3b/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f616c6c5f636f6e7472696275746f72732d31302d6f72616e67652e7376673f7374796c653d666c61742d737175617265)](https://github.com/kubesphere/kubeeye#contributors-) > English | [中文](https://github.com/kubesphere/kubeeye/blob/main/README_zh.md) KubeEye is an inspection tool for Kubernetes. It discovers whether Kubernetes resources (by using [OPA](https://github.com/open-policy-agent/opa) ), cluster components, cluster nodes (by using [Node-Problem-Detector](https://github.com/kubernetes/node-problem-detector)), and other configurations comply with best practices and makes modification suggestions accordingly. KubeEye supports custom inspection rules and plugin installation. With [KubeEye Operator](https://github.com/kubesphere/kubeeye#kubeeye-operator), you can intuitively view the inspection results and modification suggestions on the web console. ## Architecture KubeEye obtains cluster resource details by using Kubernetes APIs, inspects resource configurations by using inspection rules and plugins, and generates inspection results. The architecture of KubeEye is as follows: [![kubeeye-architecture](https://github.com/kubesphere/kubeeye/raw/main/docs/images/kubeeye-architecture.svg?raw=true)](https://github.com/kubesphere/kubeeye/blob/main/docs/images/kubeeye-architecture.svg?raw=true) ## Install and use KubeEye 1. Install KubeEye on your machine. - Method 1: Download the pre-built executable file from [Releases](https://github.com/kubesphere/kubeeye/releases). - Method 2: Build from the source code. > Note: KubeEye files will be generated in `/usr/local/bin/` on your machine. git clone [https://github.com/kubesphere/kubeeye.git](https://github.com/kubesphere/kubeeye.git) cd kubeeye make installke 1. (Optional) Install [Node-problem-Detector](https://github.com/kubernetes/node-problem-detector). > Note: If you need detailed reports, run the following command, and then NPD will be installed on your cluster. kubeeye install npd 1. Run KubeEye to inspect clusters. > Note: The results of KubeEye are sorted by resource kind. kubeeye audit KIND NAMESPACE NAME REASON LEVEL MESSAGE Node docker-desktop kubelet has no sufficient memory available warning KubeletHasNoSufficientMemory Node docker-desktop kubelet has no sufficient PID available warning KubeletHasNoSufficientPID Node docker-desktop kubelet has disk pressure warning KubeletHasDiskPressure Deployment default testkubeeye NoCPULimits Deployment default testkubeeye NoReadinessProbe Deployment default testkubeeye NotRunAsNonRoot Deployment kube-system coredns NoCPULimits Deployment kube-system coredns ImagePullPolicyNotAlways Deployment kube-system coredns NotRunAsNonRoot Deployment kubeeye-system kubeeye-controller-manager ImagePullPolicyNotAlways Deployment kubeeye-system kubeeye-controller-manager NotRunAsNonRoot DaemonSet kube-system kube-proxy NoCPULimits DaemonSet k ube-system kube-proxy NotRunAsNonRoot Event kube-system coredns-558bd4d5db-c26j8.16d5fa3ddf56675f Unhealthy warning Readiness probe failed: Get “[http://10.1.0.87:8181/ready](http://10.1.0.87:8181/ready)”: dial tcp 10.1.0.87:8181: connect: connection refused Event kube-system coredns-558bd4d5db-c26j8.16d5fa3fbdc834c9 Unhealthy warning Readiness probe failed: HTTP probe failed with statuscode: 503 Event kube-system vpnkit-controller.16d5ac2b2b4fa1eb BackOff warning Back-off restarting failed container Event kube-system vpnkit-controller.16d5fa44d0502641 BackOff warning Back-off restarting failed container Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f5fc3229c Failed warning Failed to pull image “controller:latest”: rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f61b28527 Failed warning Error: ImagePullBackOff Role kubeeye-system kubeeye-leader-election-role CanDeleteResources ClusterRole kubeeye-manager-role CanDeleteResources ClusterRole kubeeye-manager-role CanModifyWorkloads ClusterRole vpnkit-controller CanImpersonateUser ClusterRole vpnkit-controller CanDeleteResources ## How KubeEye can help you - It inspects cluster resources according to Kubernetes best practices to ensure that clusters run stably. - It detects the control plane problems of the cluster, including kube-apiserver, kube-controller-manager, and etcd. - It detects node problems, including memory, CPU, disk pressure, and unexpected kernel error logs. ## Checklist | Yes/No | Check Item | Description | Severity | | --- | --- | --- | --- | | ✅ | PrivilegeEscalationAllowed | Privilege escalation is allowed. | danger | | ✅ | CanImpersonateUser | The Role/ClusterRole can impersonate users. | warning | | ✅ | CanModifyResources | The Role/ClusterRole can delete Kubernetes resources. | warning | | ✅ | CanModifyWorkloads | The Role/ClusterRole can modify Kubernetes resources. | warning | | ✅ | NoCPULimits | No CPU limits are set. | danger | | ✅ | NoCPURequests | No CPU resources are reserved. | danger | | ✅ | HighRiskCapabilities | High-risk features, such as ALL, SYS\_ADMIN, and NET\_ADMIN, are enabled. | danger | | ✅ | HostIPCAllowed | HostIPC is set to `true`. | danger | | ✅ | HostNetworkAllowed | HostNetwork is set to `true`. | danger | | ✅ | HostPIDAllowed | HostPID is set to `true`. | danger | | ✅ | HostPortAllowed | HostPort is set to `true`. | danger | | ✅ | ImagePullPolicyNotAlways | The image pull policy is not set to `always`. | warning | | ✅ | ImageTagIsLatest | The image tag is `latest`. | warning | | ✅ | ImageTagMiss | The image tag is missing. | danger | | ✅ | InsecureCapabilities | Insecure options are missing, such as KILL, SYS\_CHROOT, and CHOWN. | danger | | ✅ | NoLivenessProbe | Liveless probe is not set. | warning | | ✅ | NoMemoryLimits | No memory limits are set. | danger | | ✅ | NoMemoryRequests | No memory resources are reserved. | danger | | ✅ | NoPriorityClassName | Resource scheduling priority is not set. | ignore | | ✅ | PrivilegedAllowed | Pods are running in the privileged mode. | danger | | ✅ | NoReadinessProbe | Readiness probe is not set. | warning | | ✅ | NotReadOnlyRootFilesystem | readOnlyRootFilesystem is not set to `true`. | warning | | ✅ | NotRunAsNonRoot | runAsNonRoot is not set to `true`. | warning | | ✅ | CertificateExpiredPeriod | The certificate expiry date of the API Server is less than 30 days. | danger | | ✅ | EventAudit | Events need to be audited. | warning | | ✅ | NodeStatus | Node status needs to be checked. | warning | | ✅ | DockerStatus | Docker status needs to be checked. | warning | | ✅ | KubeletStatus | kubelet status needs to be checked. | warning | ## Add your own inspection rules ### Add custom OPA rules 1. Create a directory for storing OPA rules. mkdir opa 1. Add custom OPA rule files. > Note: - OPA rule for checking workloads: The package name must be _kubeeye\_workloads\_rego_. - OPA rule for checking RBAC settings: The package name must be _kubeeye\_RBAC\_rego_. - OPA rule for checking node settings: The package name must be _kubeeye\_nodes\_rego_. 1. To check whether the image registry address complies with rules, save the following rules to _imageRegistryRule.rego_ package kubeeye\_workloads\_rego deny[msg] { resource := input type := resource.Object.kind resourcename := resource.Object.metadata.name resourcenamespace := resource.Object.metadata.namespace workloadsType := {“Deployment”,“ReplicaSet”,“DaemonSet”,“StatefulSet”,“Job”} workloadsType[type] not workloadsImageRegistryRule(resource) msg := { “Name”: sprintf(“%v”, [resourcename]), “Namespace”: sprintf(“%v”, [resourcenamespace]), “Type”: sprintf(“%v”, [type]), “Message”: “ImageRegistryNotmyregistry” } } workloadsImageRegistryRule(resource) { regex.match(“^myregistry.public.kubesphere/basic/.+”, resource.Object.spec.template.spec.containers[\_].image) } 1. Run KubeEye with custom rules. > Note: Kubeeye will read all files ending with _.rego_ in the directory. root:# kubeeye audit -p ./opa NAMESPACE NAME KIND MESSAGE default nginx1 Deployment [ImageRegistryNotmyregistry NotReadOnlyRootFilesystem NotRunAsNonRoot] default nginx11 Deployment [ImageRegistryNotmyregistry PrivilegeEscalationAllowed HighRiskCapabilities HostIPCAllowed HostPortAllowed ImagePullPolicyNotAlways ImageTagIsLatest InsecureCapabilities NoPriorityClassName PrivilegedAllowed NotReadOnlyRootFilesystem NotRunAsNonRoot] default nginx111 Deployment [ImageRegistryNotmyregistry NoCPULimits NoCPURequests ImageTagMiss NoLivenessProbe NoMemoryLimits NoMemoryRequests NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot] ### Add custom NPD rules 1. Run the following command to change the ConfigMap: kubectl edit ConfigMap node-problem-detector-config -n kube-system 1. Run the following command to restart NPD: kubectl rollout restart DaemonSet node-problem-detector -n kube-system ## KubeEye Operator ### What is KubeEye Operator KubeEye Operator is an inspection platform for Kubernetes. It manages KubeEye to regularly inspect clusters and generate inspection results. ### How KubeEye Operator can help you - It records inspection results by using CR and provide a web page for you to intuitively view and compare cluster inspection results. - It provides more plugins. - It provides more detailed modification suggestions. ### Deploy KubeEye Operator ```auto kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml ``` ### Obtain the inspection results `kubectl get clusterinsight -o yaml` ```auto apiVersion: v1 items: - apiVersion: kubeeye.kubesphere.io/v1alpha1 kind: ClusterInsight metadata: name: clusterinsight-sample namespace: default spec: auditPeriod: 24h status: auditResults: auditResults: - resourcesType: Node resultInfos: - namespace: "" resourceInfos: - items: - level: warning message: KubeletHasNoSufficientMemory reason: kubelet has no sufficient memory available - level: warning message: KubeletHasNoSufficientPID reason: kubelet has no sufficient PID available - level: warning message: KubeletHasDiskPressure reason: kubelet has disk pressure name: kubeeyeNode ``` ## GitHub: [https://github.com/kubesphere/kubeeye](https://github.com/kubesphere/kubeeye)