<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>K8s-at-Home on Tsic&#39;s Blog</title>
    <link>https://tsic.top/categories/k8s-at-home/</link>
    <description>Recent content in K8s-at-Home on Tsic&#39;s Blog</description>
    <generator>Hugo -- 0.154.0</generator>
    <language>zh-CN</language>
    <lastBuildDate>Sun, 08 Oct 2023 11:12:10 +0800</lastBuildDate>
    <atom:link href="https://tsic.top/categories/k8s-at-home/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Change Scheduler for k3s</title>
      <link>https://tsic.top/post/change-scheduler/</link>
      <pubDate>Sun, 08 Oct 2023 11:12:10 +0800</pubDate>
      <guid>https://tsic.top/post/change-scheduler/</guid>
      <description>&lt;p&gt;本文主要介绍使用load-watcher + scheduler-plugins + descheduler 配合实现k3s根据资源占用平衡调度。&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>本文主要介绍使用load-watcher + scheduler-plugins + descheduler 配合实现k3s根据资源占用平衡调度。</p>
<h2 id="k3s-调度问题">k3s 调度问题</h2>
<p>k8s 默认调度器不会更具节点的实际负载进行调度，只会根据request中申请的资源来。所以会导致多数的部署都是往一台比较强劲的节点上，或者新加入的节点不会有任何的pod分配过来。</p>
<h2 id="metrics-server">metrics-server</h2>
<p>使用 k8s 默认的metrics-server 来获取节点资源使用情况</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
</span></span></code></pre></div><h2 id="load-watcher">load-watcher</h2>
<p>为 <a href="https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/kep/61-Trimaran-real-load-aware-scheduling/README.md">trimaran</a> 打分时提供当前node的资源使用情况数据。
上游项目地址: <a href="https://github.com/paypal/load-watcher">https://github.com/paypal/load-watcher</a>
镜像: <a href="https://github.com/tsic404/docker-images/pkgs/container/load-watcher">https://github.com/tsic404/docker-images/pkgs/container/load-watcher</a></p>
<p>为了 load-watcher 方便读取集群各个节点的资源使用信息，将kubeconfig作为configmap传递到容器中，并指定 KUBE_CONFIG 环境变量。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nn">---</span><span class="w"> </span><span class="c"># kube-config configmap</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ConfigMap</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">loadwatcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">immutable</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">data</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">kube-config</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">    xxxxx</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nn">---</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher-deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">loadwatcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">ghcr.io/tsic404/load-watcher:latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">         </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">KUBE_CONFIG</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">           </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l">/kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="m">2020</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l">/kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">subPath</span><span class="p">:</span><span class="w"> </span><span class="l">kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">configMap</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">items</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span>- <span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">kube-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nn">---</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Service</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">loadwatcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterIP</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">http</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">2020</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="m">2020</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">load-watcher</span><span class="w">
</span></span></span></code></pre></div><h2 id="scheduler-plugins">scheduler-plugins</h2>
<p>采用 helm install as a second scheduler
使用 trimaran 插件来根据当前 node 的实际资源占用情况来进行调度。
values.yaml</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scheduler</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">trimaran</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">registry.k8s.io/scheduler-plugins/kube-scheduler:v0.26.7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicaCount</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">leaderElect</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">controller</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">scheduler-plugins-controller</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">registry.k8s.io/scheduler-plugins/controller:v0.26.7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicaCount</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">plugins</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;TargetLoadPacking&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">disabled</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;NodeResourcesBalancedAllocation&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;NodeResourcesLeastAllocated&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">pluginConfig</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">TargetLoadPacking</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">args</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">watcherAddress</span><span class="p">:</span><span class="w"> </span><span class="l">http://load-watcher.loadwatcher.svc.cluster.local:2020</span><span class="w">
</span></span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">helm install scheduler-plugins as-a-second-scheduler/ --create-namespace --namespace scheduler-plugins -f values.yaml
</span></span></code></pre></div><h2 id="descheduler">descheduler</h2>
<p>k8s 中pod一旦绑定了节点就不会在进行调度，但是pod运行过程中，pod使用的资源可能会不断变化，这样分配时的平衡就会被打破。所以引入重平衡工具<code>descheduler</code>，让其找到可以可以移除的pod并驱逐他们，重新触发k8s的调度。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">helm install descheduler --namespace kube-system descheduler/descheduler --set <span class="nv">kind</span><span class="o">=</span>Deployment
</span></span></code></pre></div><h2 id="depolyment">depolyment</h2>
<p>由于采用的是将 trimaran 安装成第二调度器，所以需要在 deployment 中显示指明调度器名称</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">schedulerName</span><span class="p">:</span><span class="w"> </span><span class="l">trimaran</span><span class="w">
</span></span></span></code></pre></div>]]></content:encoded>
    </item>
    <item>
      <title>k3s with multus cni</title>
      <link>https://tsic.top/post/k3s-with-multus-cni/</link>
      <pubDate>Mon, 24 Jul 2023 11:38:08 +0800</pubDate>
      <guid>https://tsic.top/post/k3s-with-multus-cni/</guid>
      <description>&lt;p&gt;介绍一下如何配置k3s的multus cni。&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>介绍一下如何配置k3s的multus cni。</p>
<h1 id="k3s-with-multus-cni">k3s with multus cni</h1>
<p>最近在家部署了一个k3s，用于管理一些家用服务比如jellyfin，homeassistant等，在使用homeassistant时，遇到一些网络问题，在了解k8s一点网络架构后，需要在k8s中安装multus cni，为pod创建额外网口来解决我的问题。</p>
<h2 id="一安装multus">一、安装multus</h2>
<p>拉取官方git仓库<code>git clone https://github.com/k8snetworkplumbingwg/multus-cni.git</code></p>
<p>然后打开<code>code multus-cni</code></p>
<p>修改<code>deployments/multus-daemonset.yml</code></p>
<p>需要修改有三处</p>
<ol>
<li>cniVersion</li>
</ol>
<p>k3s 默认采用的 flannel 的 cniVersion 已经升级到 1.0.0 版本，为了 multus 可以正确生成 flannel 相关配置，需要把cniVersion提高到 1.0.0</p>
<ol start="2">
<li>kubeconfig</li>
</ol>
<p>multus 默认读取 kubeconfig 的路径为 <code>/etc/cni/net.d/multus.d/multus.kubeconfig</code>，但是 k3s 将相关配置保存在 <code>/var/lib/rancher/k3s/agent/etc/</code> 中，导致 cni 相关的配置（下面volume）会改动到此，所以需要修改 kubeconfig 路径到 <code>/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig</code></p>
<ol start="3">
<li>volume</li>
</ol>
<p>为了配置 k3s 资源的默认保存路径
将 <code>/etc/cni/net.d</code> 修改为 <code>/var/lib/rancher/k3s/agent/etc/cni/net.d</code>
将 <code>/opt/multus/bin</code> 修改为 <code>/var/lib/rancher/k3s/data/current/bin</code></p>
<p>生成的diff文件</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-diff" data-lang="diff"><span class="line"><span class="cl"><span class="gh">diff --git a/deployments/multus-daemonset.yml b/deployments/multus-daemonset.yml
</span></span></span><span class="line"><span class="cl"><span class="gh">index ab626a66..28eb48af 100644
</span></span></span><span class="line"><span class="cl"><span class="gd">--- a/deployments/multus-daemonset.yml
</span></span></span><span class="line"><span class="cl"><span class="gi">+++ b/deployments/multus-daemonset.yml
</span></span></span><span class="line"><span class="cl"><span class="gu">@@ -125,7 +125,7 @@ data:
</span></span></span><span class="line"><span class="cl">       },
</span></span><span class="line"><span class="cl">       &#34;delegates&#34;: [
</span></span><span class="line"><span class="cl">         {
</span></span><span class="line"><span class="cl"><span class="gd">-          &#34;cniVersion&#34;: &#34;0.3.1&#34;,
</span></span></span><span class="line"><span class="cl"><span class="gi">+          &#34;cniVersion&#34;: &#34;1.0.0&#34;,
</span></span></span><span class="line"><span class="cl">           &#34;name&#34;: &#34;default-cni-network&#34;,
</span></span><span class="line"><span class="cl">           &#34;plugins&#34;: [
</span></span><span class="line"><span class="cl">             {
</span></span><span class="line"><span class="cl"><span class="gu">@@ -145,7 +145,7 @@ data:
</span></span></span><span class="line"><span class="cl">           ]
</span></span><span class="line"><span class="cl">         }
</span></span><span class="line"><span class="cl">       ],
</span></span><span class="line"><span class="cl"><span class="gd">-      &#34;kubeconfig&#34;: &#34;/etc/cni/net.d/multus.d/multus.kubeconfig&#34;
</span></span></span><span class="line"><span class="cl"><span class="gi">+      &#34;kubeconfig&#34;: &#34;/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig&#34;
</span></span></span><span class="line"><span class="cl">     }
</span></span><span class="line"><span class="cl"> ---
</span></span><span class="line"><span class="cl"> apiVersion: apps/v1
</span></span><span class="line"><span class="cl"><span class="gu">@@ -183,8 +183,8 @@ spec:
</span></span></span><span class="line"><span class="cl">         command: [&#34;/thin_entrypoint&#34;]
</span></span><span class="line"><span class="cl">         args:
</span></span><span class="line"><span class="cl">         - &#34;--multus-conf-file=auto&#34;
</span></span><span class="line"><span class="cl"><span class="gd">-        - &#34;--multus-autoconfig-dir=/host/etc/cni/net.d&#34;
</span></span></span><span class="line"><span class="cl"><span class="gd">-        - &#34;--cni-conf-dir=/host/etc/cni/net.d&#34;
</span></span></span><span class="line"><span class="cl"><span class="gi">+        - &#34;--cni-version=1.0.0&#34;
</span></span></span><span class="line"><span class="cl"><span class="gi">+        - &#34;--multus-kubeconfig-file-host=/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig&#34;
</span></span></span><span class="line"><span class="cl">         resources:
</span></span><span class="line"><span class="cl">           requests:
</span></span><span class="line"><span class="cl">             cpu: &#34;100m&#34;
</span></span><span class="line"><span class="cl"><span class="gu">@@ -222,10 +222,10 @@ spec:
</span></span></span><span class="line"><span class="cl">       volumes:
</span></span><span class="line"><span class="cl">         - name: cni
</span></span><span class="line"><span class="cl">           hostPath:
</span></span><span class="line"><span class="cl"><span class="gd">-            path: /etc/cni/net.d
</span></span></span><span class="line"><span class="cl"><span class="gi">+            path: /var/lib/rancher/k3s/agent/etc/cni/net.d
</span></span></span><span class="line"><span class="cl">         - name: cnibin
</span></span><span class="line"><span class="cl">           hostPath:
</span></span><span class="line"><span class="cl"><span class="gd">-            path: /opt/cni/bin
</span></span></span><span class="line"><span class="cl"><span class="gi">+            path: /var/lib/rancher/k3s/data/current/bin
</span></span></span><span class="line"><span class="cl">         - name: multus-cfg
</span></span><span class="line"><span class="cl">           configMap:
</span></span><span class="line"><span class="cl">             name: multus-cni-config
</span></span></code></pre></div><p>修改完后，部署 multus cni 的 daemonset 即可。</p>
<h2 id="二配置-cni-网络">二、配置 cni 网络</h2>
<p>根据我家的网络情况，我需要在homeassistant中访问lan和iot两个网段。所以定义了两个NetworkAttachmentDefinition。
一个 macvlan 对 lan 的访问，一个 vlan 对 iot 的访问。
cni中更多网络介绍 <a href="https://www.cni.dev/plugins/current/main/">https://www.cni.dev/plugins/current/main/</a>。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">NetworkAttachmentDefinition</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w">  </span><span class="l">lan</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">homeassistant</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">config</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">    {
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;cniVersion&#34;: &#34;0.3.1&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;type&#34;: &#34;macvlan&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;master&#34;: &#34;eth0&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;ipam&#34;: {
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;type&#34;: &#34;host-local&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;subnet&#34;: &#34;192.168.18.0/24&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;rangeStart&#34;: &#34;192.168.18.32&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;rangeEnd&#34;: &#34;192.168.18.100&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;routes&#34;: [
</span></span></span><span class="line"><span class="cl"><span class="sd">          { &#34;dst&#34;: &#34;192.168.18.0/24&#34; }
</span></span></span><span class="line"><span class="cl"><span class="sd">        ]
</span></span></span><span class="line"><span class="cl"><span class="sd">      }
</span></span></span><span class="line"><span class="cl"><span class="sd">    }</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nn">---</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;k8s.cni.cncf.io/v1&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">NetworkAttachmentDefinition</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w">  </span><span class="l">iot</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">homeassistant</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">config</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">    {
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;cniVersion&#34;: &#34;0.3.1&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;type&#34;: &#34;vlan&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;master&#34;: &#34;eth0&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;vlanId&#34;: 2,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;linkInContainer&#34;: false,
</span></span></span><span class="line"><span class="cl"><span class="sd">      &#34;ipam&#34;: {
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;type&#34;: &#34;host-local&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;subnet&#34;: &#34;192.168.32.0/24&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;rangeStart&#34;: &#34;192.168.32.3&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;rangeEnd&#34;: &#34;192.168.32.100&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">        &#34;routes&#34;: [
</span></span></span><span class="line"><span class="cl"><span class="sd">          { &#34;dst&#34;: &#34;192.168.32.0/24&#34; }
</span></span></span><span class="line"><span class="cl"><span class="sd">        ]
</span></span></span><span class="line"><span class="cl"><span class="sd">      }
</span></span></span><span class="line"><span class="cl"><span class="sd">    }</span><span class="w">
</span></span></span></code></pre></div><h2 id="三使用-multus">三、使用 multus</h2>
<p>使用前需要注意，k3s 中不包含一些 cni 比如vlan，使用时会报vlan找不到的错误。
此时需要自己拉取 cni plugins 的仓库，编译对应的 cni。
<code>git clone https://github.com/containernetworking/plugins</code>
并将编译好的二进制放到 k3s data 下的 bin 里面。
然后在 pod 中加上下面内容即可。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">k8s.v1.cni.cncf.io/networks</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">        [
</span></span></span><span class="line"><span class="cl"><span class="sd">            {
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;name&#34;: &#34;lan&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;ips&#34;: [&#34;192.168.18.32/24&#34;],
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;namespace&#34;: &#34;homeassistant&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;interface&#34;: &#34;net1&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;mac&#34;: &#34;32:44:0b:5a:0e:2e&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            },
</span></span></span><span class="line"><span class="cl"><span class="sd">            {
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;name&#34;: &#34;iot&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;ips&#34;: [&#34;192.168.32.3/24&#34;],
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;namespace&#34;: &#34;homeassistant&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;interface&#34;: &#34;net1.2&#34;,
</span></span></span><span class="line"><span class="cl"><span class="sd">                &#34;mac&#34;: &#34;00:e0:4c:68:02:3d&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">            }
</span></span></span><span class="line"><span class="cl"><span class="sd">        ]</span><span class="w">
</span></span></span></code></pre></div><p>这样homeassistant 就会有 loopback，eth0，net1，net1.2 四个网口。</p>
<h2 id="四一些问题的处理方式">四、一些问题的处理方式</h2>
<ol>
<li>遇到不断加 ip 到 net1 和 eth0。从 rangeStart 直到 rangeEnd 为止。</li>
</ol>
<p>将 multus daemonset停下，清理各个节点 multus 相关的配置，重新部署 multus。</p>
<ol start="2">
<li>报 loopback 或者 macvlan 等非 链式调用。</li>
</ol>
<p>在对应节点上运行 <code>k3s check-config</code>。查看 <code>cni</code> 相关的(md5sum, link)是否正常。</p>
]]></content:encoded>
    </item>
    <item>
      <title>Rpi4 Diskless</title>
      <link>https://tsic.top/post/rpi4-diskless/</link>
      <pubDate>Mon, 24 Jul 2023 10:52:40 +0800</pubDate>
      <guid>https://tsic.top/post/rpi4-diskless/</guid>
      <description>&lt;p&gt;使用tftp + nfssroot/iscsi 实现树莓派无盘启动。&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>使用tftp + nfssroot/iscsi 实现树莓派无盘启动。</p>
<h1 id="树莓派使用网络启动">树莓派使用网络启动</h1>
<p>pi  通过 tftp 和 dhcp 拉取 boot 分区的内容。然后把拉取的 boot 分区中的 initramfs 加载到内存，通过 内核启动参数（nfsroot，iscsi）和iniramfs 完成网络启动过程。</p>
<p>其他比如 rock5b 不支持 tftp, 但是可以用后面 initramfs 和内核参数。所以理论上可以做到本地 sd 卡只存放 boot 分区，然后通过网络启动 rootfs。</p>
<h2 id="一-tftp-hpa">一、 tftp-hpa</h2>
<ol>
<li>安装 tftp-hpa</li>
</ol>
<p><code>sudo apt install tftp-hpa</code></p>
<ol start="2">
<li>修改配置文件</li>
</ol>
<p><code>sudo vim /etc/default/tftpd-hpa</code></p>
<p>修改一下配置</p>
<pre tabindex="0"><code># /etc/default/tftpd-hpa

TFTP_USERNAME=&#34;tftp&#34;
TFTP_DIRECTORY=&#34;/data/pxe/&#34;
TFTP_ADDRESS=&#34;:69&#34;
TFTP_OPTIONS=&#34;--secure&#34;
</code></pre><ol start="3">
<li>重启 tftp-hpa 服务</li>
</ol>
<p><code>sudo systemctl restart tftp-hpa</code></p>
<ol start="4">
<li>在 TFTP_DIRECTORY 放置对应的网络启动文件。</li>
</ol>
<p>将 pi img 中 boot 分区相关的文件放到对应的目录下，下文修改 boot 相关内容时，需要同步 tftp server中。</p>
<pre tabindex="0"><code>root@rock5b:~# ls /data/pxe/
rapi4-1_boot  rapi4-2_boot  rapi4-3_boot rpi4-1-root.img  rpi4-2-root.img  rpi4-3-root.img
</code></pre><h2 id="二-nfsroot">二、 nfsroot</h2>
<p>相关链接： <a href="https://docs.kernel.org/admin-guide/nfs/nfsroot.html">https://docs.kernel.org/admin-guide/nfs/nfsroot.html</a></p>
<ol>
<li>安装nfs服务</li>
</ol>
<p><code>sudo apt install nfs-kernel-server</code></p>
<ol start="2">
<li>生成nfs路径</li>
</ol>
<p><code>mkdir /data/pxe/nfsroot1</code></p>
<p><code>vim /etc/exportfs</code></p>
<p>加入以下内容</p>
<pre tabindex="0"><code># rootfs
/data/pxe/nfsroot1    192.168.28.1/24(rw,no_root_squash,async,insecure)

# boot
/data/pxe/ 192.168.28.1/24(rw,no_root_squash,async,insecure)
</code></pre><p>然后在 pi 上尝试挂载 nfs，并将 rootfs 同步到nfs中。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">mount 192.168.18.14:/data/pxe/nfsroot1 /mnt
</span></span><span class="line"><span class="cl">rsync -avz --exclude /mnt --exclude /sys --exclude /proc --exclude /run --exclude /boot / /mnt
</span></span></code></pre></div><p>修改 nfsroot 中的 fstab(/mnt/etc/fstab)，将一些 tmp 和 log 挂载为 tmpfs 节省网络资源。</p>
<pre tabindex="0"><code>proc            /proc           proc    defaults          0       0
192.168.18.14:/data/pxe/rapi4-1_boot/ /boot nfs defaults,vers=4.1,proto=tcp 0 0
tmpfs   /tmp         tmpfs   rw,nodev,nosuid,size=2G,mode=777          0  0
tmpfs   /var/log         tmpfs   rw,nodev,nosuid,size=2G,mode=777          0  0
tmpfs   /var/tmp         tmpfs   rw,nodev,nosuid,size=2G,mode=777          0  0
</code></pre><ol start="3">
<li>修改pi内核启动参数</li>
</ol>
<p>生成initramfs。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">update-initramfs -u -k <span class="sb">`</span>uname -r<span class="sb">`</span>
</span></span></code></pre></div><p>然后修改boot/config.txt, 加载initramfs。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">vim /boot/config.txt
</span></span></code></pre></div><p>加入以下内容，注意修改成对应的 iniramfs 文件。</p>
<pre tabindex="0"><code>initramfs initrd.img-6.1.21-v8+ followkernel
</code></pre><p>修改内核参数 /boot/cmdline.txt</p>
<pre tabindex="0"><code>console=serial0,115200 console=tty1 ip=192.168.18.15::192.168.18.1:255.255.255.0:rpi4-1:eth0:off root=/dev/nfs nfsroot=192.168.18.14:/data/pxe/nfsroot1,tcp,rw rootfstype=nfs rootwait elevator=deadline cgroup_memory=1 cgroup_enable=memory cgroup_enable=cpuset
</code></pre><p><img loading="lazy" src="https://s2.loli.net/2023/07/24/w3SYbkex57HVoID.png"></p>
<p>注意将修改后的 boot 同步到 tftp server pxe 上。</p>
<h2 id="三-bootcmd">三、 bootcmd</h2>
<ol>
<li>查看当前bootloader配置</li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">root@rpi4-1:~# vcgencmd bootloader_config
</span></span><span class="line"><span class="cl"><span class="o">[</span>all<span class="o">]</span>
</span></span><span class="line"><span class="cl"><span class="nv">BOOT_UART</span><span class="o">=</span><span class="m">0</span>
</span></span></code></pre></div><ol start="2">
<li>提取配置文件</li>
</ol>
<pre tabindex="0"><code>cp /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2023-05-11.bin pieeprom.bin
rpi-eeprom-config pieeprom.bin &gt; bootconf.txt
</code></pre><ol start="3">
<li>设置配置文件</li>
</ol>
<pre tabindex="0"><code># bootconf.txt

[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=4000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=192.168.18.14
TFTP_PREFIX=1
BOOT_ORDER=0x21
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
TFTP_PREFIX_STR=rapi4-1_boot/
</code></pre><ol start="4">
<li>一些配置说明</li>
</ol>
<ul>
<li>
<p>BOOT_UART</p>
<p>设置为1，表示使能GPIO 14和 15的输出，也就是我们可以连接串口打开信息。其串口参数为波特率115200，8位，无奇偶校验位，1位的停止位。</p>
</li>
<li>
<p>TFTP_PREFIX</p>
<p>0 - Use the serial number e.g. &ldquo;9ffefdef/&rdquo;</p>
<p>1 - Use the string specified by TFTP_PREFIX_STR</p>
<p>2 - Use the MAC address e.g. &ldquo;DC-A6-32-01-36-C2/&rdquo;</p>
<p>Default: 0</p>
<p>自己定义TFTP_PREFIX_STR，方便识别 pi，所以设置成 1。</p>
</li>
<li>
<p>TFTP_PREFIX_STR</p>
<p>当TFTP_PREFIX设置为1的时候，可以设置TFTP_PREFIX_STR的路径。这样每个 pi 通过有不同的 TFTP_PREFIX_STR 从而启动不同的 rootfs 中去。</p>
</li>
<li>
<p>TFTP_IP</p>
<p>设置TFTP服务器的IP地址，树莓派的IP地址是通过DHCP自动获取的。</p>
</li>
<li>
<p>BOOT_ORDER</p>
<p>该参数配置了不同的启动模式：</p>
<ul>
<li>0x0 - NONE (stop with error pattern)</li>
<li>0x1 - SD CARD</li>
<li>0x2 - NETWORK</li>
</ul>
</li>
</ul>
<ol start="5">
<li>将配置写回到<code>pieeprom.bin</code></li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">rpi-eeprom-config --out pieeprom-new.bin --config bootconf.txt pieeprom.bin
</span></span></code></pre></div><ol start="6">
<li>更新eeprom</li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">sudo rpi-eeprom-update -d -f ./pieeprom-new.bin
</span></span></code></pre></div><h2 id="四-iscsi">四、 iscsi</h2>
<p>nfs方式无法支持docker的overlyfs。</p>
<p>iscsi 搭建
<a href="https://www.cnblogs.com/pipci/p/11617680.html">https://www.cnblogs.com/pipci/p/11617680.html</a></p>
<ol>
<li>
<p>搭建iscsi
安装tgt
<code>sudo apt install -y tgt</code></p>
</li>
<li>
<p>创建 rootfs img 文件</p>
</li>
</ol>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">dd <span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/data/pxe/rpi4-1-root.img <span class="nv">bs</span><span class="o">=</span>1G <span class="nv">count</span><span class="o">=</span><span class="m">128</span> <span class="nv">status</span><span class="o">=</span>progress
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">root@rock5b:~# cat /etc/tgt/conf.d/rpi.conf 
</span></span><span class="line"><span class="cl">default-driver iscsi
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">&lt;target iqn.cluster.tsic.top:rpi4.rootfs1&gt;
</span></span><span class="line"><span class="cl">        backing-store /data/pxe/rpi4-1-root.img
</span></span><span class="line"><span class="cl">&lt;/target&gt;
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">&lt;target iqn.cluster.tsic.top:rpi4.rootfs2&gt;
</span></span><span class="line"><span class="cl">        backing-store /data/pxe/rpi4-2-root.img
</span></span><span class="line"><span class="cl">&lt;/target&gt;
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">&lt;target iqn.cluster.tsic.top:rpi4.rootfs3&gt;
</span></span><span class="line"><span class="cl">        backing-store /data/pxe/rpi4-3-root.img
</span></span><span class="line"><span class="cl">&lt;/target&gt;
</span></span></code></pre></div><p>然后使用 iscsiadm 加载iscsi 磁盘。</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">iscsiadm -m discovery -t st -p 192.168.18.14
</span></span><span class="line"><span class="cl">iscsiadm --mode node --targetname <span class="s2">&#34;iqn.cluster.tsic.top:rpi4.rootfs2&#34;</span> --portal 192.168.18.14 --login
</span></span></code></pre></div><p><code>lsblk</code>即可看到一个 <code>sdX</code> 磁盘。然后为该磁盘分区，将 <code>/</code> 内容同步到磁盘中即可。</p>
<ol start="2">
<li>修改pi内核启动参数</li>
</ol>
<p>分区时会有相关输出，或者使用 <code>blkid</code> 查询分区的 UUID。
root=UUID=xxxxxxxxxxxxxxxxxxx</p>
<pre tabindex="0"><code>console=serial0,115200 console=tty1 root=UUID=c48523b6-8262-416c-a5da-9dbe6656ac70 ip=192.168.18.15::192.168.18.1:255.255.255.0:rpi4-1:eth0:off ISCSI_INITIATOR=iqn.2016-04.com.open-iscsi:ca33685ee2a ISCSI_TARGET_NAME=iqn.cluster.tsic.top:rpi4.rootfs1 ISCSI_TARGET_IP=192.168.18.14 ISCSI_TARGET_PORT=3260 rw rootfstype=ext4 fsck.repair=yes rootwait elevator=deadline cgroup_memory=1 cgroup_enable=memory cgroup_enable=cpuset
</code></pre><p>其他一些参数
iscsi 认证相关
ISCSI_USERNAME=
ISCSI_PASSWORD=</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
