hugging face 的Tokenizers库目前在 mac 上必须源码安装，利用 rust安装
- python环境
conda create -n transformers python==3.8
conda activate transformers
- rust环境
brew install rust
确保环境变量成功配置cargo、rustc
cargo --version
rustc --version
- setuptools_rust
python -m pip install setuptools_rust
- torch
python -m pip install torch torchvision torchaudio
- tokenizers
python setup.py install
- transformers
python -m pip install git+https://github.com/huggingface/transformers
然后 mac 就可以使用transformers库和 hugging face预训练模型了，如果帮助到您请点个简书关注
