{"componentChunkName":"component---src-templates-post-template-jsx","path":"/works/posts/2022-10-29--001","result":{"data":{"site":{"siteMetadata":{"title":"Blog by Eunyoung","subtitle":"작업 기록 블로그","copyright":"© All rights reserved.","author":{"name":"EunYoung","twitter":"#"},"disqusShortname":"","url":"https://ssongey.github.io"}},"markdownRemark":{"id":"441c75bd-356f-52f9-a30e-4f0c7ea1da28","html":"<p>이번에 전사적으로 사용하고 있는 리소스 최적화 요청이 내려왔다.<br>\n때문에 내가 운영하고 있던 서비스들의 VM 스펙 변경 및 클러스터 노드를 줄이는 작업을 진행하게 되었다.</p>\n<br/>\n<br/>\n<p>클러스터 노드를 줄이는 작업을 진행할때, 삭제 대상이 된 노드에 서비스하고 있는 파드들을 어떻게 옮길지에 대한 고민을 하게 되었다.</p>\n<p>\n  <a\n    class=\"gatsby-resp-image-link\"\n    href=\"/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/a7115/001-01.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n  \n  <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 960px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 23.75%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA8klEQVQY01VQ2XKEMAzj//+whd3hJiSEKyHAcKgy2z70QTOyLR9ytK4r9n1/sG0bnHP/4vM8geuCGwb4ecK2LAhuxqwVhqYmd1gXj+DZFxZE1losFAm6rsP7/Yb3/oEMDyEA943s9UKrFDyHOQ62WiP5/sLY91w0YxpHTORRmqbI8xxVVaEsS8Rx/MTC67qGpeg6DhRZ9iwrigJ10yCjJkkS6qonJ7E1+nOh5raejQ2FilfIpQKpDdx8nweMbmGYEwy0X/IApVr05MYYaKLvDCL54R/EolgVLlYD3yC/vPnDmcP9NP5ioj2LWeySC9z4qf0AZzp7IINjqrEAAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    >\n      <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width: 100%; height: 100%; margin: 0; vertical-align: middle; position: absolute; top: 0; left: 0; box-shadow: inset 0px 0px 0px 400px white;\"\n        alt=\"001 01\"\n        title=\"\"\n        src=\"/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/d9199/001-01.png\"\n        srcset=\"/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/8ff5a/001-01.png 240w,\n/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/e85cb/001-01.png 480w,\n/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/d9199/001-01.png 960w,\n/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/07a9c/001-01.png 1440w,\n/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/29114/001-01.png 1920w,\n/devHistoryBlog/static/1a0decba31a526be34202bd47ff5a4bc/a7115/001-01.png 2186w\"\n        sizes=\"(max-width: 960px) 100vw, 960px\"\n      />\n    </span>\n  </span>\n  \n  </a>\n    </p>\n<p>처음엔 <code class=\"language-text\">nodeSelector</code> 를 이용하여 삭제 대상 노드에서 파드들을 빼려고 했는데,</p>\n<p>이를 쉽게 해주는 <strong>cordon, drain</strong> 기능이 있다는걸 알게 되었다..!</p>\n<br/>\n<br/>\n<h2>✔️ Cordon</h2>\n<ul>\n<li>지정된 노드에 더이상 Pod들이 스케줄링되지 않도록 한다.</li>\n</ul>\n<div class=\"gatsby-highlight\" data-language=\"bash\"><pre class=\"language-bash\"><code class=\"language-bash\">$ kubectl cordon <span class=\"token punctuation\">[</span>노드명<span class=\"token punctuation\">]</span></code></pre></div>\n<p>\n  <a\n    class=\"gatsby-resp-image-link\"\n    href=\"/devHistoryBlog/static/353075458c49085386b14e24328bd54e/ad00e/001-02.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n  \n  <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 960px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 36.25%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsTAAALEwEAmpwYAAABZklEQVQoz2VRa2+CQBBEEwVFAUUQ5ABFhePpuw+oiU3//2+a7l2aGNMPk7tsZmdndhW2XmMWRVBNC6OxAb3fx0RR4EynCJgHb+Viu90hIo5hGJhS3XVdxHGMPnEV4r7germgKUvskgS32xV5VWGf57BsG133BU0bgWccYRjC930Jz/NQUs8/MYHz+xvObYv6fMHj+wfXusZ704CRi4/PG8YTDRnfk0CFLMvAOceaUrXUM5vNJFRVfQqyXYLd6YxNUYJFCWpyWrAAxWYDe2EiSWPEmwg2ObYsC/P5XMYWQqImMBwOn4Jxuoe/TzFxXIx1A/ZoBEdTwYjIIp/EQixdD8vl8iWaaZry1TQNg8FA7lUMUTqyfjoekaapjFHRDkUsx3HweDyQkOMgCGRM8RfH0HUd9/td1ouikMNEX03rUgpablk34HSItu2Q8xwlkWx7gcPh8BrnD71eTw4VcVerlVyFcCgO9gtEorWRrVwuSgAAAABJRU5ErkJggg=='); background-size: cover; display: block;\"\n    >\n      <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width: 100%; height: 100%; margin: 0; vertical-align: middle; position: absolute; top: 0; left: 0; box-shadow: inset 0px 0px 0px 400px white;\"\n        alt=\"001 02\"\n        title=\"\"\n        src=\"/devHistoryBlog/static/353075458c49085386b14e24328bd54e/d9199/001-02.png\"\n        srcset=\"/devHistoryBlog/static/353075458c49085386b14e24328bd54e/8ff5a/001-02.png 240w,\n/devHistoryBlog/static/353075458c49085386b14e24328bd54e/e85cb/001-02.png 480w,\n/devHistoryBlog/static/353075458c49085386b14e24328bd54e/d9199/001-02.png 960w,\n/devHistoryBlog/static/353075458c49085386b14e24328bd54e/ad00e/001-02.png 1366w\"\n        sizes=\"(max-width: 960px) 100vw, 960px\"\n      />\n    </span>\n  </span>\n  \n  </a>\n    </p>\n<br/>\n<br/>\n<p>노드에 cordon 을 걸면 노드의 상태가 ScheduleingDisabled 가 된 것을 확인할 수 있다.<br>\n그럼 더이상 해당 노드에는 새로운 파드들이 들어오지 않게 된다.<br>\n다시 정상적으로 노드에 스케줄링을 하기 위해서는 <strong>uncordon</strong>을 수행하면 된다. </p>\n<div class=\"gatsby-highlight\" data-language=\"bash\"><pre class=\"language-bash\"><code class=\"language-bash\">$ kubectl uncordon <span class=\"token punctuation\">[</span>노드명<span class=\"token punctuation\">]</span></code></pre></div>\n<p>\n  <a\n    class=\"gatsby-resp-image-link\"\n    href=\"/devHistoryBlog/static/6bbbafbe2a9aeee4e92c33367b443e01/e4611/001-03.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n  \n  <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 960px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 26.25%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA/klEQVQY03WQTW+CQBCGMeVAgoAtKuwCCigFDOIuGHto+f8/6+lCDx5MD5M3mY9n5h0rTvaUXUdeN8gspykKzlJyTjOkjInElrquOZ1OBEGAEILNZkMURcRxvOictyzrL9LPiGqaKNRAUl64JQl6v2UwKqIPwp2PiMUy7DjOc/C/aLqK7eGIvV7j+u94tk1gv+EbLcsjh1ySJCnX65UwDBew53k0TUOWZQtktVo9gapXKNNcVxVfjwdd39NrjW9sDcNIaqy7rss4jqzN0hk0W2zblsK85+VCPSr0/U57ufD9M6H6G6NS7EJje9Dmj2IBarNkvmyGzJrnuanJF+AvB3J/U/18lNYAAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    >\n      <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width: 100%; height: 100%; margin: 0; vertical-align: middle; position: absolute; top: 0; left: 0; box-shadow: inset 0px 0px 0px 400px white;\"\n        alt=\"001 03\"\n        title=\"\"\n        src=\"/devHistoryBlog/static/6bbbafbe2a9aeee4e92c33367b443e01/d9199/001-03.png\"\n        srcset=\"/devHistoryBlog/static/6bbbafbe2a9aeee4e92c33367b443e01/8ff5a/001-03.png 240w,\n/devHistoryBlog/static/6bbbafbe2a9aeee4e92c33367b443e01/e85cb/001-03.png 480w,\n/devHistoryBlog/static/6bbbafbe2a9aeee4e92c33367b443e01/d9199/001-03.png 960w,\n/devHistoryBlog/static/6bbbafbe2a9aeee4e92c33367b443e01/e4611/001-03.png 1298w\"\n        sizes=\"(max-width: 960px) 100vw, 960px\"\n      />\n    </span>\n  </span>\n  \n  </a>\n    </p>\n<br/>\n<br/>\n<h2>✔️ Drain</h2>\n<ul>\n<li>지정된 노드에 있는 Pod들을 다른 노드들로 재생성 후, 지정된 노드에서 Pod 삭제</li>\n</ul>\n<div class=\"gatsby-highlight\" data-language=\"bash\"><pre class=\"language-bash\"><code class=\"language-bash\">$ kubectl drain <span class=\"token punctuation\">[</span>노드명<span class=\"token punctuation\">]</span> \n\n$ kubectl drain <span class=\"token punctuation\">[</span>노드명<span class=\"token punctuation\">]</span> --ignore-daemonsets</code></pre></div>\n<br/>\n<p>cordon 작업을 통해 삭제 대상 노드에 더이상 새로운 파드가 못들어오도록 막았다면, 이젠 삭제 대상의 노드들에 있는 Pod들을 삭제대상이 아닌 노드로 옮기는 작업을 할 차례이다.</p>\n<p>식제 대상 노드의 pod들을 확인해보니, daemonset 등도 포함되어 있는 것을 확인할 수 있다.\n\n  <a\n    class=\"gatsby-resp-image-link\"\n    href=\"/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/2e92a/001-04.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n  \n  <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 960px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 19.583333333333332%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAvElEQVQY0z2QyRKDIBBEcSmXCLgrWorxpCdP/v+vdexJJYcpoKf7MaDe+w6dJAiUgrEGzo0oXgXiOIZ6tF9VVYWu61CWJdq2RdM0slLnnnqWZVB+XREGgYSMsdj8hmEYxBiGIaIokp5zTvS+7wWwbV8fod578bPUdV1I01RCNN/3LbUsi9zKYv88T4FSJ5BnrfUfyOnGcYSa5xl5nguwrmsxHschwWmaBBg8L+A0DBFgrZVw8nwVoexxT+8H0tpgZae8r9YAAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    >\n      <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width: 100%; height: 100%; margin: 0; vertical-align: middle; position: absolute; top: 0; left: 0; box-shadow: inset 0px 0px 0px 400px white;\"\n        alt=\"001 04\"\n        title=\"\"\n        src=\"/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/d9199/001-04.png\"\n        srcset=\"/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/8ff5a/001-04.png 240w,\n/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/e85cb/001-04.png 480w,\n/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/d9199/001-04.png 960w,\n/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/07a9c/001-04.png 1440w,\n/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/29114/001-04.png 1920w,\n/devHistoryBlog/static/1b07df40dd7bf0adcc5685e76a4a1dcb/2e92a/001-04.png 2508w\"\n        sizes=\"(max-width: 960px) 100vw, 960px\"\n      />\n    </span>\n  </span>\n  \n  </a>\n    </p>\n<br/>\n<p>daemonset 의 특성상 pod가 죽게되면 즉시 다시 생성되기 때문에 노드에 daemonset 이 있는 경우 drain 명령어를 실행하면 error: cannot delete DaemonSet-managed Pods 오류가 발생한다.\n\n  <a\n    class=\"gatsby-resp-image-link\"\n    href=\"/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/10f47/001-05.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n  \n  <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 960px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 16.666666666666664%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAjklEQVQI133MTwuEIBCHYREiMitISsHW9NDaH+2w3/+7/XZ3DsJe9vAyMMM87LxvnCnjihHxuWHfI3LOCCHAe49pmjCOY5ld16Gua4pzDsbYb25Z4NcV7vP8cI4gay2MMZjnGVprQqqqKtBfcDsOHFfCKyWE1RGmlIKUkvpiQgiqbdtS0zR07/sewzCU/RsXZUiIEtQciQAAAABJRU5ErkJggg=='); background-size: cover; display: block;\"\n    >\n      <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width: 100%; height: 100%; margin: 0; vertical-align: middle; position: absolute; top: 0; left: 0; box-shadow: inset 0px 0px 0px 400px white;\"\n        alt=\"001 05\"\n        title=\"\"\n        src=\"/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/d9199/001-05.png\"\n        srcset=\"/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/8ff5a/001-05.png 240w,\n/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/e85cb/001-05.png 480w,\n/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/d9199/001-05.png 960w,\n/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/07a9c/001-05.png 1440w,\n/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/29114/001-05.png 1920w,\n/devHistoryBlog/static/87dc747f538ba462c6314ba961146532/10f47/001-05.png 2726w\"\n        sizes=\"(max-width: 960px) 100vw, 960px\"\n      />\n    </span>\n  </span>\n  \n  </a>\n    </p>\n<br/>\n<p>보통은 로깅, proxy, 네트웍 관련 데몬셋은 떠있기 때문에 꼭 <code class=\"language-text\">--ignore-daemonsets</code> 옵션을 주고 실행을 한다.\n그럼 데몬셋 파드는 무시하고 나머지 파드들만 옮긴다.\n\n  <a\n    class=\"gatsby-resp-image-link\"\n    href=\"/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/cb88c/001-06.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n  \n  <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 960px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 16.666666666666664%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAeUlEQVQI142MuwoDIQAEFURQ0BMU7Czik2iM+f+P29wJSX3FMNvskLE+aH2gl4ocH8g5o5SCEAK89zDGwFq70VrjOA5IKfdWSv39g7znxJwvPEdHbQ21VsQY4ZzbZyEECCH3mWew9461Fq59OaW0g5xzMMZAKb0d/AJ8lUgm2nNW+QAAAABJRU5ErkJggg=='); background-size: cover; display: block;\"\n    >\n      <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width: 100%; height: 100%; margin: 0; vertical-align: middle; position: absolute; top: 0; left: 0; box-shadow: inset 0px 0px 0px 400px white;\"\n        alt=\"001 06\"\n        title=\"\"\n        src=\"/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/d9199/001-06.png\"\n        srcset=\"/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/8ff5a/001-06.png 240w,\n/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/e85cb/001-06.png 480w,\n/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/d9199/001-06.png 960w,\n/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/07a9c/001-06.png 1440w,\n/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/29114/001-06.png 1920w,\n/devHistoryBlog/static/87e5c175e853bd0ccf95d66fe5a8ef6c/cb88c/001-06.png 2728w\"\n        sizes=\"(max-width: 960px) 100vw, 960px\"\n      />\n    </span>\n  </span>\n  \n  </a>\n    </p>\n<br/>\n<p>삭제 노드에는 데몬셋 파드만 남은 것을 확인할 수 있다.\n\n  <a\n    class=\"gatsby-resp-image-link\"\n    href=\"/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/0ddab/001-07.png\"\n    style=\"display: block\"\n    target=\"_blank\"\n    rel=\"noopener\"\n  >\n  \n  <span\n    class=\"gatsby-resp-image-wrapper\"\n    style=\"position: relative; display: block;  max-width: 960px; margin-left: auto; margin-right: auto;\"\n  >\n    <span\n      class=\"gatsby-resp-image-background-image\"\n      style=\"padding-bottom: 16.249999999999996%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAqklEQVQI1y2P2w6DIBBEoeIlUVoVFDHGGyRqG+v/f90UN304mYdJzuwyf+zwxwemLCE5RxwLGNugrmskSUKUoev7HkqpPzWeL4m2banL85yIogjsfRzwzoExRnD+gHMey7JgnmdkWYau6zBNE6qqwjAM0Epj27YwHmMcR+pucdM0YOf3xHVd0FoHGSdWt2Lf9yB2EELAWgtjDIqioJRSkug+4B67RWma0lc/nwJJ65Q8+XMAAAAASUVORK5CYII='); background-size: cover; display: block;\"\n    >\n      <img\n        class=\"gatsby-resp-image-image\"\n        style=\"width: 100%; height: 100%; margin: 0; vertical-align: middle; position: absolute; top: 0; left: 0; box-shadow: inset 0px 0px 0px 400px white;\"\n        alt=\"001 07\"\n        title=\"\"\n        src=\"/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/d9199/001-07.png\"\n        srcset=\"/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/8ff5a/001-07.png 240w,\n/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/e85cb/001-07.png 480w,\n/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/d9199/001-07.png 960w,\n/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/07a9c/001-07.png 1440w,\n/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/29114/001-07.png 1920w,\n/devHistoryBlog/static/425e65c7ef2b4c2b46de43dddc350bfe/0ddab/001-07.png 2220w\"\n        sizes=\"(max-width: 960px) 100vw, 960px\"\n      />\n    </span>\n  </span>\n  \n  </a>\n    </p>\n<br/>\n<p>이후 삭제 대상 노드 처리를 해주면 끝!!</p>","fields":{"tagSlugs":["/tags/k-8-s/"],"slug":"/works/posts/2022-10-29--001"},"frontmatter":{"title":"Cordon, Drain 명령어를 이용하여 Node 작업하기","tags":["k8s"],"date":"2022-10-29","description":null}}},"pageContext":{"slug":"/works/posts/2022-10-29--001"}},"staticQueryHashes":[]}