Kubernetes - 机器学习 - 问题锦集

▶ Pytorch 共享内存不足的问题

问题描述

k8s 中运行 Pytorch 程序，出现以下错误

ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)

问题分析

PyTorch 官方文档：Please note that PyTorch uses shared memory to share data between processes, so if torch multiprocessing is used (e.g. for multithreaded data loaders) the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host or --shm-size command line options to nvidia-docker run.

PyTorch 的 IPC 会利用共享内存，所以共享内存必须足够大。Docker 默认共享内存是 64M，并且可以通过 docker run --shm-size 进行修改。

解决方法

运行容器部署模板添加 shm 的挂载

      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
      containers:
      - name: ***
        volumeMounts:
          - mountPath: /dev/shm
            name: dshm

参考资料

• https://docs.docker.com/engine/reference/run/
• https://github.com/kubernetes/kubernetes/issues/28272

▶ CentOS 7 容器无法启用 service

问题描述

在 CentOS 7 容器下开启 sshd 服务（service sshd start），出现如下错误：

Failed to get D-Bus connection: Operation not permitted

问题分析

CentOS 官方镜像文档：Systemd is now included in both the centos:7 and centos:latest base containers, but it is not active by default.

解决方法

授予容器特权模式，默认启用 Systemd

      volumes:
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
      containers:
      - name: ***
        command:
          - /usr/sbin/init
        volumeMounts:
          - mountPath: /sys/fs/cgroup
            name: cgroup
        securityContext:
            privileged: true

参考资料

• https://github.com/docker-library/docs/tree/master/centos#dockerfile-for-systemd-base-image