infinity-emb 是个 embedding 推理服务, 支持 qwen3 embedding, bge-m3e 等主流 embedding模型. 然而在GPU上部署却有难以处理的包冲突问题. 记录调试结果如下.
环境
显卡: NVIDIA A40 *2 (只用了一张卡)
操作系统: Rocky Linux release 8.4 (Green Obsidian)
CUDA Version: 12.8
环境管理器: conda (miniforge3)
实测, 可支持的请求量级
- A40单卡, 可以同时加载 qwen3 0.6b 4b 8b 一起跑
- qwen3 0.6b 4b 可以按最大一批32稳定运行
- qwen3 8b 在 batch 为32时不稳定, batch 降低到 16 可以稳定运行
安装
conda create -n infinity python=3.12
conda activate infinity
pip install -r infinity-requirements.txt
实在有些包难以找到兼容的版本, 改源码注释掉不用的功能.
vim /your-conda-path/envs/infinity/lib/python3.12/site-packages/infinity_emb/transformer/utils_optimum.py
第 8 行
from huggingface_hub import HfApi, HfFolder # type: ignore
改为
from huggingface_hub import HfApi #, HfFolder # type: ignore
pip freeze一个可以跑起来的版本搭配如下
infinity-requirements.txt 内容
aiohappyeyeballs==2.6.1
aiohttp==3.13.3
aiosignal==1.4.0
annotated-doc==0.0.4
annotated-types==0.7.0
anyio==4.12.1
attrs==25.4.0
backoff==2.2.1
certifi==2026.2.25
charset-normalizer==3.4.4
click==8.1.7
coloredlogs==15.0.1
cuda-bindings==12.9.4
cuda-pathfinder==1.4.0
datasets==4.6.1
dill==0.4.0
distro==1.9.0
fastapi==0.135.1
filelock==3.25.0
frozenlist==1.8.0
fsspec==2026.2.0
h11==0.16.0
hf-xet==1.3.2
httpcore==1.0.9
httptools==0.7.1
httpx==0.28.1
huggingface_hub==0.36.2
humanfriendly==10.0
idna==3.11
infinity_emb==0.0.77
Jinja2==3.1.6
joblib==1.5.3
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mdurl==0.1.2
mpmath==1.3.0
multidict==6.7.1
multiprocess==0.70.18
networkx==3.6.1
numpy==1.26.4
nvidia-cublas-cu12==12.8.4.1
nvidia-cuda-cupti-cu12==12.8.90
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cudnn-cu12==9.10.2.21
nvidia-cufft-cu12==11.3.3.83
nvidia-cufile-cu12==1.13.1.3
nvidia-curand-cu12==10.3.9.90
nvidia-cusolver-cu12==11.7.3.90
nvidia-cusparse-cu12==12.5.8.93
nvidia-cusparselt-cu12==0.7.1
nvidia-nccl-cu12==2.27.5
nvidia-nvjitlink-cu12==12.8.93
nvidia-nvshmem-cu12==3.4.5
nvidia-nvtx-cu12==12.8.90
orjson==3.11.7
packaging @ file:///home/task_176104874243446/conda-bld/packaging_1761049080023/work
pandas==3.0.1
pillow==12.1.1
posthog==7.9.6
prometheus-fastapi-instrumentator==7.1.0
prometheus_client==0.24.1
propcache==0.4.1
protobuf==7.34.0
pyarrow==23.0.1
pydantic==2.12.5
pydantic_core==2.41.5
Pygments==2.19.2
python-dateutil==2.9.0.post0
python-dotenv==1.2.2
PyYAML==6.0.3
regex==2026.2.28
requests==2.32.5
rich==13.9.4
safetensors==0.7.0
scikit-learn==1.8.0
scipy==1.17.1
sentence-transformers==3.4.1
sentencepiece==0.2.1
setuptools==80.10.2
shellingham==1.5.4
six==1.17.0
starlette==0.52.1
sympy==1.14.0
threadpoolctl==3.6.0
tokenizers==0.22.2
torch==2.10.0
tqdm==4.67.3
transformers==4.57.6
triton==3.6.0
typer==0.11.1
typing-inspection==0.4.2
typing_extensions==4.15.0
urllib3==2.6.3
uvicorn==0.32.1
uvloop==0.22.1
watchfiles==1.1.1
websockets==16.0
wheel==0.46.3
xxhash==3.6.0
yarl==1.23.0
启动服务
下载模型, huggingface镜像: https://hf-mirror.com
启动服务
conda activate infinity
nohup \
infinity_emb v2 \
--port 11435 \
--model-id "./models/Qwen3-Embedding-0.6B" \
--served-model-name qwen3-embedding-0.6b \
--model-id "./models/Qwen3-Embedding-4B" \
--served-model-name qwen3-embedding-4b \
--model-id "./models/Qwen3-Embedding-8B" \
--served-model-name qwen3-embedding-8b \
--engine torch \
--device auto \
--device-id '0' \
--api-key your-password \
--no-bettertransformer \
> qwen3-embedding-log.log 2>&1 &
访问接口
curl --request POST 'http://127.0.0.1:11435/embeddings'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer your-password'
--data-raw '{
"model": "qwen3-embedding-8b",
"dimensions": 128,
"input": [
"Two cute cats.",
"天空为什么是蓝的?"
]
}'
