ARM-V8与Trustzone
ARM v8 AArch64架构
aarch64架构图
由上图可知：ARM处理器有两种状态：Normal world与 Secure world; ARM处理器有四个异常等级(Exception Level)：Elx (x = 0, 1, 2, 3). 由于el3拥有最高权限，不区分安全与非安全，所以ARM处理器总共有2 * 4 - 1 = 7个状态
重要的寄存器
通用寄存器：
ARMv8-A提供了31个64位的通用寄存器，始终可以访问，并且可以在所有异常级别访问。 在AArch64执行状态下，每个寄存器（X0-X30）都是64位宽度。 宽度增加有助于减少大部分应用程序中的寄存器压力。
每个64位通用寄存器（X0 - X30）也有一个32位的格式（W0 - W30）。
| Register | x | 功能 |
|---|---|---|
| CPSR | - | 记录当前处理器状态:NZCV IFTM… |
| SP_ELx | 0,1,2,3 | 指向堆栈顶部的指针寄存器 |
| ELR_ELx | 1,2,3 | 异常链接寄存器, 保存要在异常后返回的地址 |
| SPSR_ELx | 1,2,3 | 保存的处理器状态寄存器 |
| VBAR_ELx | 1,2,3 | 基于向量的地址寄存器, 保存异常基地址以便发送到ELn的任何异常。 |
| TTBR0_ELx/TTBR1_ELx | 0,1,2,3 | 保存页表基地址 |
| SCR_EL3 | - | 用于配置arm核上的安全属性：如指示当前核的安全状态NS位，指示当前中断或异常是否路由至el3，不同的中断将使用不同位段表示：https://developer.arm.com/documentation/ddi0595/2021-06/AArch64-Registers/SCR-EL3--Secure-Configuration-Register |
异常等级切换
系统调用异常：SVC, HVC, or SMC (vbar_elx, spsr_elx)
SVC通常被EL0（user mode）的软件用来申请 操作系统上EL1（OS service）请求特权操作或访问系统资源。
HVC主要被guest OS用来请求hypervisor的服务;
SMC表示: Secure monitor Call, 用于secure与non-secure切换, 本质是通过保存两个世界的寄存器上下文信息后，设置scr_el3.NS位来完成世界的切换：：
安全世界切换
高异常等级->低异常等级：ERET (elr_elx, cpsr)
Trustzone与TEE
GP(Glabol Platform)标准下的TEE
GP规范化定义了一个Trustzone下TEE软件与硬件架构。用白话说就是一整套TEE标准化的众多规则，本质上也是为了保证安全应用调用的接口命名一致，保持可移植性。
- 对于CA/TA开发者来说，我们主要需要在CA侧调用对应接口，在TA侧实现对应接口，以打通CA与TA的交互session.
CA侧调用：
TEEC_InitializeContext
TEEC_FinalizeContext
TEEC_RegisterSharedMemory
TEEC_AllocateSharedMemory
TEEC_ReleaseSharedMemory
TEEC_OpenSession
TEEC_CloseSession
TEEC_InvokeCommand
TEEC_RequestCancellation
TA侧实现：
TA_CreateEntryPoint
TA_DestroyEntryPoint
TA_InvokeCommandEntryPoint
TA_OpenSessionEntryPoint
- 对于TEE_OS开发者来说，则需要的实现GP规定下的各种component，也必须实现对应和TA交互的API，方便TA的移植。这个比较多就不在这里列举，可以参考官方网站 ：https://globalplatform.org/specs-library/?filter-committee=tee
硬件基础：
image.png
在支持trustzone技术的微处理器中，AMR实现了对cortex的虚拟化，将每个物理的处理器虚拟化成两个虚拟核，一个为安全核，一个普通安全核，也就是一个物理核存在secure wrold态和normal world态。
AXI总线上NS位的扩展：在原有的AXI基础上对每一个读写信道增加了一个额外的控制信号，用来表示当前的读写操作是安全操作还是非安全操作，被称为NS位或者Non-Secure位。当NS位信号为1时，拒绝访问安全资源
其他支持
- Secure bootrom：启动的rom，存储安全启动代码
- TZASC组件(Trustzone Address Space Controller)：配置所有设备地址的安全属性
- TZMA组件(Trustzone Memory Adapter)：配置片上静态ram地址的安全属性
- TZPC组件(Trustzone Protection Controller)：确定外部设备的发送的信号安全属性
- TZIC组件（Trustzone Interrupt Controller）：配置中断的安全属性(CPU处于安全态时，中断为安全中断)
- MMU的扩展：也就是每个世界都具有单独的MMU页表。存放在MMU中的每一条页表描述符都会包含一个NS位来表示被映射的内存安全属性。
- Cache的扩展：在cache中的每一项都会按照normal world和secure world的状态打上对应的tag，这就能实现在不同的world下面，处理器只要属性自己world的cache就可以了。
ARM TEE软件架构：
下面是GP标准下，TEE软件架构图，对应optee中的组件名
gp下tee软件架构
- TEE Cilent API: 提供给非安全用户态应用调用TA的接口，将参数下沉至tee内核驱动；在optee中位libteec.so
- TEE protocol Specs: 接受来自tee-driver的命令完成需要在el0才能进行的工作：如文件加载的；在optee中为tee-supplicant，是一个后台进程
- Driver & REE Communication Agent 1.管理非安全相关设备以及与安全世界通信的共享内存; 2.非安全世界rpc通信的端点; 3.触发smc异常，使arm核进入安全态. 在optee中为tee-driver.ko（目前已默认集成在linux kernel中）
- ATF runtime: 保存非安全世界cpu上下文，恢复安全世界上下文，调用eret 将cpu退回sel1.
- Trusted OS Component：即运行在安全世界的OS组件，功能在不断扩展，在optee则为optee编译好的镜像，通常为tee.bin
- TEE Internal Core API：通常为一个用于安全应用编译时的SDK，optee中有一个sdk文件夹，可以参考其安全应用编译文档
- TA (Trusted App)：运行在隔离环境下的安全应用
所以TEE是啥
TEE (Trusted Execution Environment):
在ARM下，由上述Trustzone硬件基础与软件组件共同构建的一个与非安全世界隔离的可信程序运行环境。
除了Trustzone之外，其他的硬件平台上的TEE有以下知名环境：
Intel SGX/AMD Secure Execution Environment/RISC-V MultiZone…
optee
为什么选择optee
先看optee官方是怎么给自己定义的：
OP-TEE is a Trusted Execution Environment (TEE) designed as companion to a non-secure Linux kernel running on Arm; Cortex-A cores using the TrustZone technology. OP-TEE implements TEE Internal Core API v1.1.x which is the API exposed to Trusted Applications and the TEE Client API v1.0, which is the API describing how to communicate with a TEE. Those APIs are defined in the GlobalPlatform API specifications.
The main design goals for OP-TEE are:
- Isolation - the TEE provides isolation from the non-secure OS and protects the loaded Trusted Applications (TAs) from each other using underlying hardware support,
- Small footprint - the TEE should remain small enough to reside in a reasonable amount of on-chip memory as found on Arm based systems,
- Portability - the TEE aims at being easily pluggable to different architectures and available HW and has to support various setups such as multiple client OSes or multiple TEEs
在我看来选择这个项目作为TEE学习更主要的原因是：开源。大部分TEE过于封闭，很难去窥探trustzone里奥秘，optee是一个所有组件都完全开源的项目，由linaro主导并维护，社区活跃度也很高，仅一年左右时间更新5个大版本
参考资料
源码
optee_os: https://github.com/OP-TEE/optee_os
optee_client: https://github.com/OP-TEE/optee_client
tee驱动默认在linux内核中, 更新在这: https://github.com/linaro-swg/linux/tree/optee
中文资料书籍
《手机安全和可信应用开发指南:TrustZone与OP-TEE技术详解》
作者博客：https://blog.csdn.net/shuaifengyun
OPTEE Core 架构参考
Architecture ‒ OP-TEE documentation documentation
本文不介绍optee os的具体架构，而是通过一个经典的例子来分析optee中一个完整的请求是如何从非安全世界的app发送到安全应用ta，并完成功能调用的，linaro官方的example仓，有许多功能调用的例子：https://github.com/linaro-swg/optee_examples
本文就以最简单的hello_world为例子
CA->TA
CA是调用TEE安全服务的入口程序，运行在非安全用户态el0。
TA是TEE服务的重点，大部分自定义的安全服务在TA中实现，运行在安全用户态SEL0.
从CA到TA的invoke command，完成一次服务调用，将会经历一个漫长的层级及CPU安全态切换，通常的通路如下：
(链接) (ioctl) (smc指令) (el3 eret sel1) (sel1 eret)
CA========>libteec.so=======>tee_driver=========>ATF==========>TEE_kernel==========>TA
若TEE/TA服务有部分工作需要返回非安全态执行，则需要返回tee_driver呼叫非安全侧守护进程tee_supplicant去完成工作：
TA/TEE_kernel====>ATF====>tee_driver====>tee_supplicant
由于ATF主要工作是在启动阶段完成，运行态时主要用于切换CPU安全/非安全状态，保存/恢复彼此的寄存器状态上下文，不在此处细聊。下面我们将通过一次TA加载过程的数据流来细讲每一个组件的主要功能。
CA
非安全用户态程序，动态链接libteec.so，调用其接口。CA是调用安全世界TA服务的起点。
linaro官方的example仓，有许多功能调用的例子：
https://github.com/linaro-swg/optee_examples
简单看一下hello_world:
#define TA_HELLO_WORLD_UUID \
{ 0x8aaaf200, 0x2450, 0x11e4, \
{ 0xab, 0xe2, 0x00, 0x02, 0xa5, 0xd5, 0xc5, 0x1b} }
int main(void)
{
TEEC_Result res;
TEEC_Context ctx;
TEEC_Session sess;
TEEC_Operation op;
TEEC_UUID uuid = TA_HELLO_WORLD_UUID;
uint32_t err_origin;
/* Initialize a context connecting us to the TEE */
res = TEEC_InitializeContext(NULL, &ctx);
if (res != TEEC_SUCCESS)
errx(1, "TEEC_InitializeContext failed with code 0x%x", res);
/*
* Open a session to the "hello world" TA, the TA will print "hello
* world!" in the log when the session is created.
*/
res = TEEC_OpenSession(&ctx, &sess, &uuid,
TEEC_LOGIN_PUBLIC, NULL, NULL, &err_origin);
if (res != TEEC_SUCCESS)
errx(1, "TEEC_Opensession failed with code 0x%x origin 0x%x",
res, err_origin);
/*
* Execute a function in the TA by invoking it, in this case
* we're incrementing a number.
*
* The value of command ID part and how the parameters are
* interpreted is part of the interface provided by the TA.
*/
/* Clear the TEEC_Operation struct */
memset(&op, 0, sizeof(op));
/*
* Prepare the argument. Pass a value in the first parameter,
* the remaining three parameters are unused.
*/
op.paramTypes = TEEC_PARAM_TYPES(TEEC_VALUE_INOUT, TEEC_NONE,
TEEC_NONE, TEEC_NONE);
op.params[0].value.a = 42;
/*
* TA_HELLO_WORLD_CMD_INC_VALUE is the actual function in the TA to be
* called.
*/
printf("Invoking TA to increment %d\n", op.params[0].value.a);
res = TEEC_InvokeCommand(&sess, TA_HELLO_WORLD_CMD_INC_VALUE, &op,
&err_origin);
if (res != TEEC_SUCCESS)
errx(1, "TEEC_InvokeCommand failed with code 0x%x origin 0x%x",
res, err_origin);
printf("TA incremented value to %d\n", op.params[0].value.a);
/*
* We're done with the TA, close the session and
* destroy the context.
*
* The TA will print "Goodbye!" in the log when the
* session is closed.
*/
TEEC_CloseSession(&sess);
TEEC_FinalizeContext(&ctx);
return 0;
}
libteec.so(el0)
libteec.so中包含了GP标准下提供的五个供CA调用与TA通讯的接口：
TEEC_Result TEEC_InitializeContext(
const char* name,
TEEC_Context* context)
void TEEC_FinalizeContext(
TEEC_Context* context)
TEEC_Result TEEC_OpenSession (
TEEC_Context* context,
TEEC_Session* session,
const TEEC_UUID* destination,
uint32_t connectionMethod,
const void* connectionData,
TEEC_Operation* operation,
uint32_t* returnOrigin)
void TEEC_CloseSession (
TEEC_Session* session)
TEEC_Result TEEC_InvokeCommand(
TEEC_Session* session,
uint32_t commandID,
TEEC_Operation* operation,
uint32_t* returnOrigin)
可以看到成对出现的ctx和session操作以及服务调用的TEEC_InvokeCommand。
调用TEE中TA的服务第一步是打开TEE驱动：可以看到TEEC_InitializeContext操作是很简单的，尝试去打开tee_driver的字符设备，teec的上下文中将驱动的描述符存下来，供后续使用; 并查询了一些驱动中保存的TEE_OS的一些版本信息.
TEEC_Result TEEC_InitializeContext(const char *name, TEEC_Context *ctx)
{
char devname[PATH_MAX] = { 0 };
int fd = 0;
size_t n = 0;
if (!ctx)
return TEEC_ERROR_BAD_PARAMETERS;
for (n = 0; n < TEEC_MAX_DEV_SEQ; n++) {
uint32_t gen_caps = 0;
snprintf(devname, sizeof(devname), "/dev/tee%zu", n); //通常tee驱动只会创建一个设备，自定义条件下可能有多个设备。
fd = teec_open_dev(devname, name, &gen_caps);//尝试去打开tee_driver的驱动
if (fd >= 0) {
ctx->fd = fd; //将设备的文件描述符存储在上下文中
ctx->reg_mem = gen_caps & TEE_GEN_CAP_REG_MEM; //reg_mem TEE状态 动态注册共享内存
ctx->memref_null = gen_caps & TEE_GEN_CAP_MEMREF_NULL;
return TEEC_SUCCESS;
}
}
return TEEC_ERROR_ITEM_NOT_FOUND;
}
static int teec_open_dev(const char *devname, const char *capabilities,
uint32_t *gen_caps)
{
int fd = 0;
struct tee_ioctl_version_data vers;
memset(&vers, 0, sizeof(vers));
fd = open(devname, O_RDWR);
if (fd < 0)
return -1;
if (ioctl(fd, TEE_IOC_VERSION, &vers)) {
EMSG("TEE_IOC_VERSION failed");
goto err;
}
/* We can only handle GP TEEs */
if (!(vers.gen_caps & TEE_GEN_CAP_GP))
goto err;
...
}
TEEC_OpenSession提供了从文件系统加载TA到TEE中，并打开TA-CA的安全通道session的功能。
libteec通过结构体tee_ioctl_open_session_arg和tee_ioctl_param传递session的信息
struct tee_ioctl_open_session_arg {
__u8 uuid[TEE_IOCTL_UUID_LEN];
__u8 clnt_uuid[TEE_IOCTL_UUID_LEN];
__u32 clnt_login;
__u32 cancel_id;
__u32 session;
__u32 ret;
__u32 ret_origin;
__u32 num_params;
} __aligned(8);
struct tee_ioctl_param_memref {
__u64 shm_offs;
__u64 size;
__s64 shm_id;
};
struct tee_ioctl_param_value {
__u64 a;
__u64 b;
__u64 c;
};
struct tee_ioctl_param {
__u64 attr;
union {
struct tee_ioctl_param_memref memref;
struct tee_ioctl_param_value value;
} u;
};
/* optee_client/libteec/src/tee_client_api.c */
TEEC_Result TEEC_OpenSession(TEEC_Context *ctx, TEEC_Session *session,
const TEEC_UUID *destination,
uint32_t connection_method, const void *connection_data,
TEEC_Operation *operation, uint32_t *ret_origin)
{
struct tee_ioctl_open_session_arg *arg = NULL;
struct tee_ioctl_param *params = NULL;
TEEC_Result res = TEEC_ERROR_GENERIC;
uint32_t eorig = 0;
int rc = 0;
const size_t arg_size = sizeof(struct tee_ioctl_open_session_arg) +
TEEC_CONFIG_PAYLOAD_REF_COUNT *
sizeof(struct tee_ioctl_param);
union {
struct tee_ioctl_open_session_arg arg;
uint8_t data[arg_size];
} buf;
struct tee_ioctl_buf_data buf_data;
TEEC_SharedMemory shm[TEEC_CONFIG_PAYLOAD_REF_COUNT]; //当teec输入的operation的param中含有数据流buffer时，需要申请将buffer拷贝到一块有tee驱动申请的安全与非安全交互的share_memory。
memset(&buf, 0, sizeof(buf));
memset(&shm, 0, sizeof(shm));
memset(&buf_data, 0, sizeof(buf_data));
(void)&connection_data;
if (!ctx || !session) {
eorig = TEEC_ORIGIN_API;
res = TEEC_ERROR_BAD_PARAMETERS;
goto out;
}
buf_data.buf_ptr = (uintptr_t)&buf;
buf_data.buf_len = sizeof(buf);
arg = &buf.arg;
arg->num_params = TEEC_CONFIG_PAYLOAD_REF_COUNT;
params = (struct tee_ioctl_param *)(arg + 1);
uuid_to_octets(arg->uuid, destination);
arg->clnt_login = connection_method;
res = teec_pre_process_operation(ctx, operation, params, shm); //此处完成share_memory从内核到用户态的映射以及数据流的拷贝
if (res != TEEC_SUCCESS) {
eorig = TEEC_ORIGIN_API;
goto out_free_temp_refs;
}
rc = ioctl(ctx->fd, TEE_IOC_OPEN_SESSION, &buf_data); //进入驱动ioctl函数,cmd决定了
if (rc) {
EMSG("TEE_IOC_OPEN_SESSION failed");
eorig = TEEC_ORIGIN_COMMS;
res = ioctl_errno_to_res(errno);
goto out_free_temp_refs;
}
res = arg->ret; //驱动回写的返回值
eorig = arg->ret_origin;
if (res == TEEC_SUCCESS) {
session->ctx = ctx;
session->session_id = arg->session;
}
teec_post_process_operation(operation, params, shm);
out_free_temp_refs:
teec_free_temp_refs(operation, shm);
out:
if (ret_origin)
*ret_origin = eorig;
return res;
}
此处主要是将open_session用户态入参结构体初始化并通过ioctl传入内核中, 注意此处关键函数teec_pre_process_operation，注释中有描述大概的功能。后文中共享内存都指安全与非安全交互的share_memory。我们主要查看对用户态buffer的处理：
static TEEC_Result teec_pre_process_operation(TEEC_Context *ctx,
TEEC_Operation *operation,
struct tee_ioctl_param *params,
TEEC_SharedMemory *shms)
{
TEEC_Result res = TEEC_ERROR_GENERIC;
size_t n = 0;
memset(shms, 0, sizeof(TEEC_SharedMemory) *
TEEC_CONFIG_PAYLOAD_REF_COUNT);
...
for (n = 0; n < TEEC_CONFIG_PAYLOAD_REF_COUNT; n++) {
uint32_t param_type = 0;
/* 遍历操作中的params */
param_type = TEEC_PARAM_TYPE_GET(operation->paramTypes, n);
switch (param_type) {
...
/* in_buffer中有两种类型，tmpref是还没写进share_memory的buffer */
case TEEC_MEMREF_TEMP_INPUT:
case TEEC_MEMREF_TEMP_OUTPUT:
case TEEC_MEMREF_TEMP_INOUT:
res = teec_pre_process_tmpref(ctx, param_type,
&operation->params[n].tmpref, params + n,
shms + n); //主要关心的是对temref处理，见下面实现
if (res != TEEC_SUCCESS)
return res;
break;
/* memref需要先申请一块share_memory，在CA中就直接写入共享内存 */
/* 在CA申请共享内存有两种方式
1. TEEC_AllocateSharedMemory
若上下文中reg_mem为true则将申请一块新内存，注册进内核作为新增的共享内存。
否则将从mmap一块内核中的共享内存，供CA使用。
2. TEEC_RegisterSharedMemory
与TEEC_AllocateSharedMemory基本一样，区别在于此时注册的进共享内存需要在外面申请好。
*/
case TEEC_MEMREF_WHOLE:
res = teec_pre_process_whole(
&operation->params[n].memref,
params + n);
if (res != TEEC_SUCCESS)
return res;
break;
case TEEC_MEMREF_PARTIAL_INPUT:
case TEEC_MEMREF_PARTIAL_OUTPUT:
case TEEC_MEMREF_PARTIAL_INOUT:
res = teec_pre_process_partial(param_type,
&operation->params[n].memref, params + n);
if (res != TEEC_SUCCESS)
return res;
break;
default:
return TEEC_ERROR_BAD_PARAMETERS;
}
}
return TEEC_SUCCESS;
}
static TEEC_Result teec_pre_process_tmpref(TEEC_Context *ctx,
uint32_t param_type, TEEC_TempMemoryReference *tmpref,
struct tee_ioctl_param *param,
TEEC_SharedMemory *shm)
{
TEEC_Result res = TEEC_ERROR_GENERIC;
/* 设置一些param和share_memory的属性，通常是读写属性 */
switch (param_type) {
/*input tee预计做读操作， output tee预计做写操作， inout tee将读写整块区域*/
case TEEC_MEMREF_TEMP_INPUT:
param->attr = TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INPUT;
shm->flags = TEEC_MEM_INPUT;
break;
...
}
/* 入参buffer的长度，后面去申请shm将会以这个长度去申请 */
shm->size = tmpref->size;
if (!tmpref->buffer && ctx->memref_null) {
if (tmpref->size)
return TEEC_ERROR_BAD_PARAMETERS;
/* Null pointer, indicate no shared memory attached */
param->u.memref.shm_id = TEE_MEMREF_NULL;
shm->id = -1;
} else {
res = TEEC_AllocateSharedMemory(ctx, shm); // 申请share_memory
if (res != TEEC_SUCCESS)
return res;
//将入参buffer拷入申请share_memory
memcpy(shm->buffer, tmpref->buffer, tmpref->size);
param->u.memref.shm_id = shm->id;
}
param->u.memref.size = tmpref->size;
return TEEC_SUCCESS;
}
内核tee_driver(el1):
用户态到内核态的中间发生了啥
驱动的初始化相关分析见：https://blog.csdn.net/shuaifengyun/article/details/72934531
首先我们要有一个概念，tee的driver是一个两层结构，第一层在开源内核代码driver/tee/tee_core.c中，其提供了一个设备创建的框架，提供了基础的open/ioctl的操作的外层封装。第二层才是真正的具体的实现, 具体挂载方式在下面代码分析可以看到。optee中是实现在driver/tee/optee中。当然厂商可以根据自己的需求，去挂载自己的实现，这也是二层结构的主要意义，厂商开发自己的驱动不必过于关心设备的创建问题，只要关心驱动的具体实现就好。
此处主要分析ioctl到open_session的流程。 分析内核首先得找到f_ops：
/* driver/tee/tee_core.c */
struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
struct device *dev,
struct tee_shm_pool *pool,
void *driver_data)
{
struct tee_device *teedev; /* 驱动的全局变量，包含了驱动中各个驱动的结构 */
void *ret;
int rc, max_id;
int offs = 0;
teedev = kzalloc(sizeof(*teedev), GFP_KERNEL);
...
cdev_init(&teedev->cdev, &tee_fops); // tee的驱动为字符设备，tee_fops here!
...
dev_set_drvdata(&teedev->dev, driver_data);
device_initialize(&teedev->dev);
...
/* 挂载上的二级结构，看下面的分析 */
teedev->desc = teedesc;
}
EXPORT_SYMBOL_GPL(tee_device_alloc);
static const struct file_operations tee_fops = {
.owner = THIS_MODULE,
.open = tee_open, // open入口
.release = tee_release,
.unlocked_ioctl = tee_ioctl, // ioctl入口
.compat_ioctl = compat_ptr_ioctl,
};
首先看看tee_open的操作，由TEEC_InitializeContext调用：
/* driver/tee/tee_core.c */
static int tee_open(struct inode *inode, struct file *filp)
{
struct tee_context *ctx;
/* 初始化好打开设备的上下文, 通过inode中的i_cdev找到驱动全局变量teedev */
ctx = teedev_open(container_of(inode->i_cdev, struct tee_device, cdev));
if (IS_ERR(ctx))
return PTR_ERR(ctx);
/*
* Default user-space behaviour is to wait for tee-supplicant
* if not present for any requests in this context.
*/
ctx->supp_nowait = false;
filp->private_data = ctx;
return 0;
}
static struct tee_context *teedev_open(struct tee_device *teedev)
{
int rc;
struct tee_context *ctx;
if (!tee_device_get(teedev))
return ERR_PTR(-EINVAL);
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); // 创建上下文
if (!ctx) {
rc = -ENOMEM;
goto err;
}
kref_init(&ctx->refcount);
ctx->teedev = teedev;
INIT_LIST_HEAD(&ctx->list_shm);
/*
真正的open操作，由optee驱动的desc->op提供.
如上文所说，tee_driver是一个外层框架，其驱动真正实现的挂载点正是此处
*/
rc = teedev->desc->ops->open(ctx);
if (rc)
goto err;
return ctx;
err:
kfree(ctx);
tee_device_put(teedev);
return ERR_PTR(rc);
}
/* desc 挂载的位置： driver/tee/optee/core.c*/
static struct optee *optee_probe(struct device_node *np)
{
...
invoke_fn = get_invoke_func(np); // 此处将为触发smc的函数挂载
...
optee->invoke_fn = invoke_fn;
...
/* 调用了上面的tee_device_alloc，注册上了具体的tee驱动描述和op*/
teedev = tee_device_alloc(&optee_desc, NULL, pool, optee);
if (IS_ERR(teedev)) {
rc = PTR_ERR(teedev);
goto err;
}
optee->teedev = teedev;
teedev = tee_device_alloc(&optee_supp_desc, NULL, pool, optee);
if (IS_ERR(teedev)) {
rc = PTR_ERR(teedev);
goto err;
}
optee->supp_teedev = teedev;
...
}
static optee_invoke_fn *get_invoke_func(struct device_node *np)
{
...
else if (!strcmp("smc", method))
return optee_smccc_smc; //optee->invoke_fn = optee_smccc_smc；
...
}
static const struct tee_driver_ops optee_ops = {
.get_version = optee_get_version,
.open = optee_open, // open函数， 不具体分析了，主要还是用于初始化打开tee设备的上下文
.release = optee_release,
//ioctl下的操作，如open_session:
.open_session = optee_open_session,
.close_session = optee_close_session,
.invoke_func = optee_invoke_func,
.cancel_req = optee_cancel_req,
.shm_register = optee_shm_register,
.shm_unregister = optee_shm_unregister,
};
static const struct tee_desc optee_desc = {
.name = DRIVER_NAME "-clnt",
.ops = &optee_ops,
.owner = THIS_MODULE,
};
有了对teedev_open的分析，分析ioctl相关操作也就并不复杂了，直接看tee_ioctl：
/* driver/tee/tee_core.c */
static long tee_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
struct tee_context *ctx = filp->private_data;
void __user *uarg = (void __user *)arg;
switch (cmd) {
...
/* 主要看open_session */
case TEE_IOC_OPEN_SESSION:
return tee_ioctl_open_session(ctx, uarg);
...
}
}
static int tee_ioctl_open_session(struct tee_context *ctx,
struct tee_ioctl_buf_data __user *ubuf)
{
int rc;
size_t n;
struct tee_ioctl_buf_data buf;
struct tee_ioctl_open_session_arg __user *uarg;
struct tee_ioctl_open_session_arg arg;
struct tee_ioctl_param __user *uparams = NULL;
struct tee_param *params = NULL;
bool have_session = false;
if (!ctx->teedev->desc->ops->open_session)
return -EINVAL;
if (copy_from_user(&buf, ubuf, sizeof(buf)))
return -EFAULT;
if (buf.buf_len > TEE_MAX_ARG_SIZE ||
buf.buf_len < sizeof(struct tee_ioctl_open_session_arg))
return -EINVAL;
/* 这一段操作都是将用户态的args拷贝到内核态，恢复对应的结构体；和普通驱动操作差不多 */
uarg = u64_to_user_ptr(buf.buf_ptr);
if (copy_from_user(&arg, uarg, sizeof(arg)))
return -EFAULT;
if (sizeof(arg) + TEE_IOCTL_PARAM_SIZE(arg.num_params) != buf.buf_len)
return -EINVAL;
if (arg.num_params) {
params = kcalloc(arg.num_params, sizeof(struct tee_param),
GFP_KERNEL);
if (!params)
return -ENOMEM;
uparams = uarg->params;
rc = params_from_user(ctx, params, arg.num_params, uparams);
if (rc)
goto out;
}
// 从上一段源码可看到挂载的具体处理函数为optee_open_session,
rc = ctx->teedev->desc->ops->open_session(ctx, &arg, params);
if (rc)
goto out;
have_session = true;
/* 将结果拷贝回用户态空间 */
if (put_user(arg.session, &uarg->session) ||
put_user(arg.ret, &uarg->ret) ||
put_user(arg.ret_origin, &uarg->ret_origin)) {
rc = -EFAULT;
goto out;
}
rc = params_to_user(uparams, arg.num_params, params);
out:
/*
* If we've succeeded to open the session but failed to communicate
* it back to user space, close the session again to avoid leakage.
*/
if (rc && have_session && ctx->teedev->desc->ops->close_session)
ctx->teedev->desc->ops->close_session(ctx, arg.session);
if (params) {
/* Decrease ref count for all valid shared memory pointers */
for (n = 0; n < arg.num_params; n++)
if (tee_param_is_memref(params + n) &&
params[n].u.memref.shm)
tee_shm_put(params[n].u.memref.shm);
kfree(params);
}
return rc;
}
内核切换到安全世界做了啥
之后ioctl将进入cpu切换安全世界的准备工作：optee_msg_arg是最重要的结构体，他是安全世界与非安全世界信息的结构体。
struct optee_msg_arg {
uint32_t cmd;
uint32_t func;
uint32_t session;
uint32_t cancel_id;
uint32_t pad;
uint32_t ret;
uint32_t ret_origin;
uint32_t num_params;
/* num_params tells the actual number of element in params */
struct optee_msg_param params[];
};
/* driver/tee/optee/call.c */
int optee_open_session(struct tee_context *ctx,
struct tee_ioctl_open_session_arg *arg,
struct tee_param *param)
{
struct optee_context_data *ctxdata = ctx->data;
int rc;
struct tee_shm *shm;
struct optee_msg_arg *msg_arg;
phys_addr_t msg_parg;
struct optee_session *sess = NULL;
/* +2 for the meta parameters added below */
/* 这个函数用来申请传递smc的具体msg的共享内存空间，安全与非安全通过该结构体完成通信
msg_parg是具体的物理地址，通过该物理地址传递具体cmd参数
（安全态和非安全态维护着不同的页表） */
shm = get_msg_arg(ctx, arg->num_params + 2, &msg_arg, &msg_parg);
if (IS_ERR(shm))
return PTR_ERR(shm);
/* 指定上具体的smc msg中的cmd，此处将去完成open_session操作 */
msg_arg->cmd = OPTEE_MSG_CMD_OPEN_SESSION;
msg_arg->cancel_id = arg->cancel_id;
/*
* Initialize and add the meta parameters needed when opening a
* session.
* 主要包含上uuid和client_uuid：
* uuid是一个TA服务的唯一标识符
*/
msg_arg->params[0].attr = OPTEE_MSG_ATTR_TYPE_VALUE_INPUT |
OPTEE_MSG_ATTR_META;
msg_arg->params[1].attr = OPTEE_MSG_ATTR_TYPE_VALUE_INPUT |
OPTEE_MSG_ATTR_META;
memcpy(&msg_arg->params[0].u.value, arg->uuid, sizeof(arg->uuid));
memcpy(&msg_arg->params[1].u.value, arg->uuid, sizeof(arg->clnt_uuid));
msg_arg->params[1].u.value.c = arg->clnt_login;
/* 我们主要关心的函数：将用户态传下来的数据拷入上面申请的smc通讯共享内存中params的字段 */
rc = optee_to_msg_param(msg_arg->params + 2, arg->num_params, param);
if (rc)
goto out;
sess = kzalloc(sizeof(*sess), GFP_KERNEL);
if (!sess) {
rc = -ENOMEM;
goto out;
}
/* 此处将触发smc指令，cpu进入el3的atf中*/
if (optee_do_call_with_arg(ctx, msg_parg)) {
msg_arg->ret = TEEC_ERROR_COMMUNICATION;
msg_arg->ret_origin = TEEC_ORIGIN_COMMS;
}
if (msg_arg->ret == TEEC_SUCCESS) {
/* A new session has been created, add it to the list. */
sess->session_id = msg_arg->session;
mutex_lock(&ctxdata->mutex);
list_add(&sess->list_node, &ctxdata->sess_list);
mutex_unlock(&ctxdata->mutex);
} else {
kfree(sess);
}
/* 将拷贝数据从msg共享内存拷贝回非安全入参的共享内存中 */
...
}
int optee_to_msg_param(struct optee_msg_param *msg_params, size_t num_params,
const struct tee_param *params)
{
int rc;
size_t n;
for (n = 0; n < num_params; n++) {
/*value类型的传参，赋值即可，memref的buffer入参如下处理：*/
...
case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INPUT:
case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_OUTPUT:
case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INOUT:
/* 将shm注册进msg_param结构体中
* tmp需要写入具体物理地址，ref直接写入shm结构体即可
*/
if (tee_shm_is_registered(p->u.memref.shm))
rc = to_msg_param_reg_mem(mp, p);
else
rc = to_msg_param_tmp_mem(mp, p);
if (rc)
return rc;
break;
default:
return -EINVAL;
}
}
return 0;
}
参数准备好之后，开始配置寄存器，触发smc指令，完成切换到el3，准备进行CPU安全态与非安全态切换：
/* driver/tee/optee/call.c */
u32 optee_do_call_with_arg(struct tee_context *ctx, phys_addr_t parg)
{
struct optee *optee = tee_get_drvdata(ctx->teedev);
struct optee_call_waiter w;
struct optee_rpc_param param = { };
struct optee_call_ctx call_ctx = { };
u32 ret;
/* 准备好寄存器的值 */
/* r0: smc_call的命令 */
param.a0 = OPTEE_SMC_CALL_WITH_ARG;
/* r1, r2: 上面准备好的msg结构体的物理地址, r1低32位，r2高32位，通过寄存器传递给安全侧 */
reg_pair_from_64(¶m.a1, ¶m.a2, parg);
/* Initialize waiter */
optee_cq_wait_init(&optee->call_queue, &w);
while (true) {
struct arm_smccc_res res;
optee_bm_timestamp();
/*如上文optee_probe所示， 此处将挂载上optee_smccc_smc */
optee->invoke_fn(param.a0, param.a1, param.a2, param.a3,
param.a4, param.a5, param.a6, param.a7,
&res);
optee_bm_timestamp();
if (res.a0 == OPTEE_SMC_RETURN_ETHREAD_LIMIT) {
/*
* Out of threads in secure world, wait for a thread
* become available.
*/
optee_cq_wait_for_completion(&optee->call_queue, &w);
} else if (OPTEE_SMC_RETURN_IS_RPC(res.a0)) {
might_sleep();
param.a0 = res.a0;
param.a1 = res.a1;
param.a2 = res.a2;
param.a3 = res.a3;
optee_handle_rpc(ctx, ¶m, &call_ctx);
} else {
ret = res.a0;
break;
}
}
optee_rpc_finalize_call(&call_ctx);
/*
* We're done with our thread in secure world, if there's any
* thread waiters wake up one.
*/
optee_cq_wait_final(&optee->call_queue, &w);
return ret;
}
/* driver/tee/optee/core
/* Simple wrapper functions to be able to use a function pointer */
static void optee_smccc_smc(unsigned long a0, unsigned long a1,
unsigned long a2, unsigned long a3,
unsigned long a4, unsigned long a5,
unsigned long a6, unsigned long a7,
struct arm_smccc_res *res)
{
arm_smccc_smc(a0, a1, a2, a3, a4, a5, a6, a7, res);
}
触发smc处：
/* include/linux/arm-smccc.h */
#define arm_smccc_smc(...) __arm_smccc_smc(__VA_ARGS__, NULL)
...
asmlinkage void __arm_smccc_smc(unsigned long a0, unsigned long a1,
unsigned long a2, unsigned long a3, unsigned long a4,
unsigned long a5, unsigned long a6, unsigned long a7,
struct arm_smccc_res *res, struct arm_smccc_quirk *quirk);
/* arch/arm64/kernel/smccc-call.S */
.macro SMCCC instr
.cfi_startproc
\instr #0 /* smc 0, 此处触发smc指令进入el3 atf */
ldr x4, [sp]
stp x0, x1, [x4, #ARM_SMCCC_RES_X0_OFFS] /* smc指令回来之后，保存结果寄存器状态 */
stp x2, x3, [x4, #ARM_SMCCC_RES_X2_OFFS]
ldr x4, [sp, #8]
cbz x4, 1f /* no quirk structure */
ldr x9, [x4, #ARM_SMCCC_QUIRK_ID_OFFS]
cmp x9, #ARM_SMCCC_QUIRK_QCOM_A6
b.ne 1f
str x6, [x4, ARM_SMCCC_QUIRK_STATE_OFFS]
1: ret
.cfi_endproc
.endm
/*
* void arm_smccc_smc(unsigned long a0, unsigned long a1, unsigned long a2,
* unsigned long a3, unsigned long a4, unsigned long a5,
* unsigned long a6, unsigned long a7, struct arm_smccc_res *res,
* struct arm_smccc_quirk *quirk)
*/
ENTRY(__arm_smccc_smc)
SMCCC smc
ENDPROC(__arm_smccc_smc)
EXPORT_SYMBOL(__arm_smccc_smc)
具体流程参考下图：
image.png
Arm Trusted firmware(ATF, el3)
之后CPU陷入EL3态，进入ATF，ATF主要确认TEE kernel入口(std_smc或者fast_smc模式，有不同的入口)，非安全世界寄存器上下文的保存工作，修改CPU系统寄存器SCR的NS状态为安全位等，此处不细讲，具体可以看：
https://blog.csdn.net/shuaifengyun/article/details/73118852
TEE_kernel
kernel为smc_call创建线程并启动线程
ATF完成好工作后, CPU进入sel1状态, 代码运行的入口由ATF复原并选择，以arm64为例：
/* optee-src/core/arch/arm/kernel/thread_optee_smc_a64.S */
/*
* Vector table supplied to ARM Trusted Firmware (ARM-TF) at
* initialization.
*
* Note that ARM-TF depends on the layout of this vector table, any change
* in layout has to be synced with ARM-TF.
*/
FUNC thread_vector_table , : , .identity_map
b vector_std_smc_entry
b vector_fast_smc_entry
b vector_cpu_on_entry
b vector_cpu_off_entry
b vector_cpu_resume_entry
b vector_cpu_suspend_entry
b vector_fiq_entry
b vector_system_off_entry
b vector_system_reset_entry
END_FUNC thread_vector_table
KEEP_PAGER thread_vector_table
LOCAL_FUNC vector_std_smc_entry , : , .identity_map
readjust_pc
bl thread_handle_std_smc
/*
* Normally thread_handle_std_smc() should return via
* thread_exit(), thread_rpc(), but if thread_handle_std_smc()
* hasn't switched stack (error detected) it will do a normal "C"
* return.
*/
mov w1, w0
ldr x0, =TEESMC_OPTEED_RETURN_CALL_DONE
smc #0
b . /* SMC should not return */
END_FUNC vector_std_smc_entry
open_session将进入vector_std_smc_entry->thread_handle_std_smc，进一步分析TEE的内核如何处理一条smc_call:
/* optee-src/core/arch/arm/kernel/thread.c */
uint32_t thread_handle_std_smc(uint32_t a0, uint32_t a1, uint32_t a2,
uint32_t a3, uint32_t a4, uint32_t a5,
uint32_t a6 __unused, uint32_t a7 __maybe_unused)
{
uint32_t rv = OPTEE_SMC_RETURN_OK;
/* 检查tee堆栈的canary，是否有溢出情况 */
thread_check_canaries();
...
/*
* thread_resume_from_rpc() and thread_alloc_and_run() only return
* on error. Successful return is done via thread_exit() or
* thread_rpc().
*/
/* smc_call来自rpc(非安全侧响应tee的命令的守护进程)分支 */
if (a0 == OPTEE_SMC_CALL_RETURN_FROM_RPC) {
thread_resume_from_rpc(a3, a1, a2, a4, a5);
rv = OPTEE_SMC_RETURN_ERESUME;
} else { /* smc_call来自Libteec，主要包括open session, close session, invoke等 */
/*a0-a3对应寄存器r0-r3*/
thread_alloc_and_run(a0, a1, a2, a3);
rv = OPTEE_SMC_RETURN_ETHREAD_LIMIT;
}
...
}
thread_alloc_and_run将对于来自libteec的smc_call创建一个thread:
/* optee-src/core/arch/arm/thread.c */
void thread_alloc_and_run(uint32_t a0, uint32_t a1, uint32_t a2, uint32_t a3)
{
size_t n;
struct thread_core_local *l = thread_get_core_local();
bool found_thread = false;
/* 确定当前核上没有线程占据，
* 特别注意，OPTEE每一个核只能有一个线程运行
*/
assert(l->curr_thread == -1);
thread_lock_global();
/* 从全局变量的thread中，拿出一个分配给当前的请求 */
for (n = 0; n < CFG_NUM_THREADS; n++) {
if (threads[n].state == THREAD_STATE_FREE) {
threads[n].state = THREAD_STATE_ACTIVE;
found_thread = true;
break;
}
}
thread_unlock_global();
if (!found_thread)
return;
l->curr_thread = n;
threads[n].flags = 0;
/* 初始化thread的各种寄存器上下文 */
init_regs(threads + n, a0, a1, a2, a3);
thread_lazy_save_ns_vfp();
/* 线程启动 */
thread_resume(&threads[n].regs);
/*NOTREACHED*/
panic();
}
static void init_regs(struct thread_ctx *thread, uint32_t a0, uint32_t a1,
uint32_t a2, uint32_t a3)
{
/* 指定该线程上下文中PC指针的地址，当该thread resume回来之后就会开始执行regs.pc执
* 行的函数
*/
thread->regs.pc = (uint64_t)thread_std_smc_entry;
/*
* Stdcalls starts in SVC mode with masked foreign interrupts, masked
* Asynchronous abort and unmasked native interrupts.
*/
thread->regs.cpsr = SPSR_64(SPSR_64_MODE_EL1, SPSR_64_MODE_SP_EL0,
THREAD_EXCP_FOREIGN_INTR | DAIFBIT_ABT);
/* Reinitialize stack pointer */
thread->regs.sp = thread->stack_va_end;
/*
* Copy arguments into context. This will make the
* arguments appear in x0-x7 when thread is started.
*/
thread->regs.x[0] = a0;
thread->regs.x[1] = a1;
thread->regs.x[2] = a2;
thread->regs.x[3] = a3;
thread->regs.x[4] = 0;
thread->regs.x[5] = 0;
thread->regs.x[6] = 0;
thread->regs.x[7] = 0;
/* Set up frame pointer as per the Aarch64 AAPCS */
thread->regs.x[29] = 0;
}
thread_resume将启动线程， 该函数为汇编代码，主要是保存一些寄存器状态，并且指定thread运行在什么模式。
/* optee-src/core/arch/arm/kernel/arm64_macros.S*/
/* void thread_resume(struct thread_ctx_regs *regs) */
FUNC thread_resume , :
/* 恢复线程寄存器状态 */
/* 加载结构体中前三个寄存器sp, pc, cpsr到x1, x2, x3，见下thread_ctx_regs结构体 */
load_xregs x0, THREAD_CTX_REGS_SP, 1, 3
/* 加载结构体x4-x30 */
load_xregs x0, THREAD_CTX_REGS_X4, 4, 30
/* 恢复sp, pc, cpsr, pc加载进elr_el1，即触发返回时将执行到该地址：
* thread_std_smc_entry */
mov sp, x1
msr elr_el1, x2
msr spsr_el1, x3
b_if_spsr_is_el0 w3, 1f /* 处理由ta触发svc下陷到sel1的操作，CA过来的操作不走这条路 */
/* 恢复 x0 - x3 */
load_xregs x0, THREAD_CTX_REGS_X1, 1, 3
ldr x0, [x0, THREAD_CTX_REGS_X0]
/* 触发函数返回， 进入thread_std_smc_entry */
eret
1: load_xregs x0, THREAD_CTX_REGS_X1, 1, 3
ldr x0, [x0, THREAD_CTX_REGS_X0]
msr spsel, #1
store_xregs sp, THREAD_CORE_LOCAL_X0, 0, 1
b eret_to_el0
END_FUNC thread_resume
/* optee-src/core/arch/arm/include/kernel/thread.h */
/* 线程寄存器上下文 */
struct thread_ctx_regs {
uint64_t sp;
uint64_t pc;
uint64_t cpsr;
uint64_t x[31];
};
在init_regs中的regs.pc中已经制定了该thread被resume回来之后的pc指针为thread_std_smc_entry，当thread被resume之后就会去执行该函数：
/* optee-src/core/arch/arm/kernel/thread_optee_smc_a64.S */
FUNC thread_std_smc_entry , :
bl __thread_std_smc_entry /* 线程入口，此时线程真正启动起来 */
mov w20, w0 /* Save return value for later */
/* Mask all maskable exceptions before switching to temporary stack */
msr daifset, #DAIFBIT_ALL
bl thread_get_tmp_sp
mov sp, x0
bl thread_state_free
ldr x0, =TEESMC_OPTEED_RETURN_CALL_DONE
mov w1, w20
mov x2, #0
mov x3, #0
mov x4, #0
/* 触发smc，退出安全世界，进入el3 */
smc #0
b . /* SMC should not return */
END_FUNC thread_std_smc_entry
用一张图表示如下：
image.png
kernel对msg的处理
__thread_std_smc_entry对于不同的硬件环境可能有一些不同的处理，是一个weak函数，但最终都将进入函数
tee_entry_std，注意r0带着smc_call的cmd，r1，r2两个寄存器保存着optee_msg的物理地址，这些都是来自于非安全世界的传入：
static uint32_t std_smc_entry(uint32_t a0, uint32_t a1, uint32_t a2,
uint32_t a3 __unused)
{
paddr_t parg = 0;
struct optee_msg_arg *arg = NULL;
uint32_t num_params = 0;
struct mobj *mobj = NULL;
uint32_t rv = 0;
/* 此处可以看到来自CA的请求,r0只能是OPTEE_SMC_CALL_WITH_ARG */
if (a0 != OPTEE_SMC_CALL_WITH_ARG) {
EMSG("Unknown SMC 0x%"PRIx32, a0);
DMSG("Expected 0x%x", OPTEE_SMC_CALL_WITH_ARG);
return OPTEE_SMC_RETURN_EBADCMD;
}
/* 通过r1-r2恢复msg的物理地址 */
parg = reg_pair_to_64(a1, a2);
/* Check if this region is in static shared space */
if (core_pbuf_is(CORE_MEM_NSEC_SHM, parg,
sizeof(struct optee_msg_arg))) {
/* 此处将物理地址转成安全世界中的虚拟地址 */
mobj = get_cmd_buffer(parg, &num_params);
} else {
if (parg & SMALL_PAGE_MASK)
return OPTEE_SMC_RETURN_EBADADDR;
mobj = map_cmd_buffer(parg, &num_params);
}
if (!mobj || !ALIGNMENT_IS_OK(parg, struct optee_msg_arg)) {
EMSG("Bad arg address 0x%" PRIxPA, parg);
mobj_put(mobj);
return OPTEE_SMC_RETURN_EBADADDR;
}
/* mobj_shm_alloc函数完成了mobj的申请，拿到va即可，获得msg的虚拟地址 */
arg = mobj_get_va(mobj, 0);
assert(arg && mobj_is_nonsec(mobj));
/* 获取msg，开始处理请求 */
rv = tee_entry_std(arg, num_params);
mobj_put(mobj);
return rv;
}
static struct mobj *get_cmd_buffer(paddr_t parg, uint32_t *num_params)
{
struct optee_msg_arg *arg;
size_t args_size;
/* arg的地址在此 */
arg = phys_to_virt(parg, MEM_AREA_NSEC_SHM);
if (!arg)
return NULL;
/* 读取params的数量 */
*num_params = READ_ONCE(arg->num_params);
if (*num_params > OPTEE_MSG_MAX_NUM_PARAMS)
return NULL;
args_size = OPTEE_MSG_GET_ARG_SIZE(*num_params);
/* optee中的对于虚拟内存分解成mobj(memory object)结构体，将保存虚拟地址，物理地址等 */
return mobj_shm_alloc(parg, args_size, 0);
}
Open_session 内核的处理流程
过了这么久的流程，终于来到了TEE真正处理请求操作的代码:
open_session主要完成的工作就是确定TA处理服务cmd(invoke_command)的代码入口，其3.5版本之后有一个巨大的改版，3.5之后版本的实现后续将简单提到一下，此处我们先看3.5版本的实现方式：
其流程非常复杂，先贴一张大神总结的流程图，顺着这个往下看，后面我们解读一些关键步骤的源码：
image
图太大，看起来很复杂，我们总结几个重点：
- 构造session与寻找目标TA是否已经加载：所有的session会以一个链表的形式存在一个结构体
struct tee_ta_session_head中，主要 在src/core/kernel/tee_ta_manager.c中的tee_ta_init_session函数中。同时确定目标TA已打开过，打开过则将直接将该ta的上下文返回，进入TA中的打开session操作中。 - 加载TA：从tee发起的对ree侧的服务请求RPC，呼叫守护进程
tee_supplicant将TA加载进一段共享内存中，并 触发smc返回安全侧 - TEE将恢复线程，并对TA进行elf格式校验与签名校验(验签的公钥被编入了tee_kernel中，需要在编译时放在指定文件夹)
- 若验签成功，TEE将加载TA进入安全内存中，通过TA头中对各函数地址偏移的指定，进入TA的open_session 的entry，完成TA中指定的处理后，返回REE侧
TEE_kernel 处理入口
简单分析一下：
/* optee-src/core/arch/arm/entry_std.c */
uint32_t __tee_entry_std(struct optee_msg_arg *arg, uint32_t num_params)
{
uint32_t rv = OPTEE_SMC_RETURN_OK;
/* Enable foreign interrupts for STD calls */
thread_set_foreign_intr(true);
switch (arg->cmd) {
/* 回忆起来, tee_driver填的cmd为OPTEE_MSG_CMD_OPEN_SESSION */
case OPTEE_MSG_CMD_OPEN_SESSION:
entry_open_session(arg, num_params);
break;
...
return rv;
}
static void entry_open_session(struct optee_msg_arg *arg, uint32_t num_params)
{
TEE_Result res;
TEE_ErrorOrigin err_orig = TEE_ORIGIN_TEE;
struct tee_ta_session *s = NULL;
TEE_Identity clnt_id;
TEE_UUID uuid;
struct tee_ta_param param;
size_t num_meta;
uint64_t saved_attr[TEE_NUM_PARAMS] = { 0 };
/* 获取一些基本的参数，在驱动侧我们能看到前两个params是驱动填充的，是一些meta数据：
* 1. uuid，用来确定加载的TA的名字
* 2. 参数个数
* 3. clnt_id (这个具体其实没有具体使用，可能后续将用于客户端鉴权)
*/
res = get_open_session_meta(num_params, arg->params, &num_meta, &uuid,
&clnt_id);
if (res != TEE_SUCCESS)
goto out;
/* 此处是非常关键的安全操作：将共享内存中的param拷贝到安全内存中 */
res = copy_in_params(arg->params + num_meta, num_params - num_meta,
¶m, saved_attr);
if (res != TEE_SUCCESS)
goto cleanup_shm_refs;
/* 之后所有的操作都在此函数中完成，以下流程都在该函数中进行 */
res = tee_ta_open_session(&err_orig, &s, &tee_open_sessions, &uuid,
&clnt_id, TEE_TIMEOUT_INFINITE, ¶m);
if (res != TEE_SUCCESS)
s = NULL;
...
}
构造session与寻找目标TA是否已经加载
tee_ta_open_session第一步就是去初始化session：
/* core/kernel/tee_ta_manager.c */
TEE_Result tee_ta_open_session(TEE_ErrorOrigin *err,
struct tee_ta_session **sess,
struct tee_ta_session_head *open_sessions,
const TEE_UUID *uuid,
const TEE_Identity *clnt_id,
uint32_t cancel_req_to,
struct tee_ta_param *param)
{
TEE_Result res;
struct tee_ta_session *s = NULL;
struct tee_ta_ctx *ctx;
bool panicked;
bool was_busy = false;
res = tee_ta_init_session(err, open_sessions, uuid, &s);
if (res != TEE_SUCCESS) {
DMSG("init session failed 0x%x", res);
return res;
}
...
}
static TEE_Result tee_ta_init_session(TEE_ErrorOrigin *err,
struct tee_ta_session_head *open_sessions,
const TEE_UUID *uuid,
struct tee_ta_session **sess)
{
TEE_Result res;
struct tee_ta_ctx *ctx;
/* session结构体的初始化，ta_ctx上下文也将注册进此结构体 */
struct tee_ta_session *s = calloc(1, sizeof(struct tee_ta_session));
...
/* Look for already loaded TA */
ctx = tee_ta_context_find(uuid);
if (ctx) {
res = tee_ta_init_session_with_context(ctx, s);
if (res == TEE_SUCCESS || res != TEE_ERROR_ITEM_NOT_FOUND)
goto out;
}
/* Look for pseudo TA */
/* 静态 pseudo TA是指已经一起连编进optee内核的一些TA。
* 这种TA打开比较简单，搜索内核中PTA的段，匹配对应的uuid，若匹配上，则直接进入open_session处理
* 由于是与optee内核连编，无需验签
*/
res = tee_ta_init_pseudo_ta_session(uuid, s);
if (res == TEE_SUCCESS || res != TEE_ERROR_ITEM_NOT_FOUND)
goto out;
/* Look for user TA */
/* 关键函数，假如我们要加载从文件系统/安全存储上的TA，走这个分支，我们这里主要观察 */
res = tee_ta_init_user_ta_session(uuid, s);
...
}
加载TA与签名认证
tee_ta_init_user_ta_session 核心处理函数就是load_elf函数，他将发起一个从TEE到REE的请求，请求去加载文件系统上指定UUID的TA。
首先看一下TA的结构：
image.png
可以看到，TA是分段的，而加载过程中，只需要加载TA raw image即可，但加载前需要对其进行校验，其校验方法参看下文：
/* core/arch/arm/kernel/user_ta.c */
static TEE_Result load_elf(const TEE_UUID *uuid, struct user_ta_ctx *utc)
{
TEE_Result res = TEE_ERROR_ITEM_NOT_FOUND;
const struct user_ta_store_ops *op = NULL;
/* ta_stores总共有三个，我们主管关心runtime时从文件系统上加载TA，其注册的op在底下 */
SCATTERED_ARRAY_FOREACH(op, ta_stores, struct user_ta_store_ops) {
DMSG("Lookup user TA ELF %pUl (%s)", (void *)uuid,
op->description);
res = load_elf_from_store(uuid, op, utc);
...
}
/* core/arch/arm/kernel/ree_fs_ta.c */
TEE_TA_REGISTER_TA_STORE(9) = {
.description = "REE",
.open = ree_fs_ta_open, /*此处注册上了open函数*/
.get_size = ree_fs_ta_get_size,
.read = ree_fs_ta_read,
.close = ree_fs_ta_close,
};
load_elf_from_store将处理将文件系统中的TA加载进安全内存中，并把entry_func入口确定下来，整个流程非常复杂，牵扯到多次TEE到REE的交互（TEE到REE的交互部分，下个部分tee_supplicant相关章节将着重关注，当前只阐述流程，源码分析可以后续补上。TA实际上还是一个前面有一个签名头的elf文件，其加载过程如下：
ta_store->open(uuid, &handle):TEE让REE首先加载TA头部, 验证头部中hash值的签名，确保头部hash值的可靠性. 并且比较被签名的UUID时候与文件系统上ta命名的UUID相同；elf_load_init：准备好加载elf所需要的状态和处理函数；elf_load_head：加载elf头，updata哈希计算；为TA分配栈空间;
get_elf_segments：解析elf段表；elf_load_body: 分段拷贝段并update哈希值，最后一段计算最后hash值与第一步哈希值是否一致；一致的话完成TA加载。
之后我们查看TA的入口，入口依然在load_elf_from_store中被定义：我们由TA raw image可以看到，第一个section为.ta_head, 我们看一下这一段的结构体：
/* ta/arch/arm/user_ta_header.c */
const struct ta_head ta_head __section(".ta_head") = {
/* UUID, unique to each TA */
.uuid = TA_UUID,
/*
* According to GP Internal API, TA_FRAMEWORK_STACK_SIZE corresponds to
* the stack size used by the TA code itself and does not include stack
* space possibly used by the Trusted Core Framework.
* Hence, stack_size which is the size of the stack to use,
* must be enlarged
*/
.stack_size = TA_STACK_SIZE + TA_FRAMEWORK_STACK_SIZE,
.flags = TA_FLAGS,
#ifdef __ILP32__
/*
* This workaround is neded on 32-bit because it seems we can't
* initialize a 64-bit integer from the address of a function.
*/
.entry.ptr32 = { .lo = (uint32_t)__utee_entry },
#else
.entry.ptr64 = (uint64_t)__utee_entry,
#endif
};
这一个.section将通过链接的脚本，在TA的编译链接过程中作为raw image的第一个section:
/* ta/arch/arm/ta.ld.S */
SECTIONS {
.ta_head : {*(.ta_head)}
.text : {
__text_start = .;
*(.text .text.*)
*(.stub)
*(.glue_7)
*(.glue_7t)
*(.gnu.linkonce.t.*)
/* Workaround for an erratum in ARM's VFP11 coprocessor */
*(.vfp11_veneer)
__text_end = .;
}
...
}
/* core/arch/arm/kernel/user_ta.c */
static TEE_Result load_elf_from_store(const TEE_UUID *uuid,
const struct user_ta_store_ops *ta_store,
struct user_ta_ctx *utc)
{
...
res = get_elf_segments(elf, &segs, &num_segs);
for (n = 0; n < num_segs; n++) {
...
/* n == 0时进入该分支 */
if (!n) {
elf->load_addr = segs[0].va; /*elf->load_addr将指向.ta_head段*/
DMSG("ELF load address %#" PRIxVA, elf->load_addr);
}
}
...
}
TEE_Result tee_ta_init_user_ta_session(const TEE_UUID *uuid,
struct tee_ta_session *s)
{
ta_head = (struct ta_head *)(vaddr_t)utc->load_addr;
/* 此处比较elf中被签名UUID与TA文件名UUID，保证文件名UUID没有被篡改 */
if (memcmp(&ta_head->uuid, uuid, sizeof(TEE_UUID)) != 0) {
res = TEE_ERROR_SECURITY;
goto err;
}
...
utc->ctx.flags = ta_head->flags;
utc->ctx.uuid = ta_head->uuid;
utc->entry_func = ta_head->entry.ptr64; /* 即__utee_entry */
...
}
可以看到TA的入口为__utee_entry，之后我们将关注TEE_kernel怎么进行层级切换进入sel0的TA.
TEE_kernel -> TA
我们回到tee_ta_open_session函数，完成session建立和TA加载后，开始准备进入sel0
/* core/kernel/tee_ta_manager.c */
TEE_Result tee_ta_open_session(TEE_ErrorOrigin *err,
struct tee_ta_session **sess,
struct tee_ta_session_head *open_sessions,
const TEE_UUID *uuid,
const TEE_Identity *clnt_id,
uint32_t cancel_req_to,
struct tee_ta_param *param)
{
TEE_Result res;
struct tee_ta_session *s = NULL;
struct tee_ta_ctx *ctx;
bool panicked;
bool was_busy = false;
res = tee_ta_init_session(err, open_sessions, uuid, &s);
if (res != TEE_SUCCESS) {
DMSG("init session failed 0x%x", res);
return res;
}
...
if (tee_ta_try_set_busy(ctx)) {
set_invoke_timeout(s, cancel_req_to);
res = ctx->ops->enter_open_session(s, param, err); /* 此处开始准备切换层级 */
tee_ta_clear_busy(ctx);
} else {
/* Deadlock avoided */
res = TEE_ERROR_BUSY;
was_busy = true;
}
...
}
ctx->ops在tee_ta_init_user_ta_session中已经完成挂载:
/* core/arch/arm/kernel/user_ta.c */
TEE_Result tee_ta_init_user_ta_session(const TEE_UUID *uuid,
struct tee_ta_session *s)
{
...
/*
* Set context TA operation structure. It is required by generic
* implementation to identify userland TA versus pseudo TA contexts.
*/
set_ta_ctx_ops(&utc->ctx);
...
}
static const struct tee_ta_ops user_ta_ops __rodata_unpaged = {
.enter_open_session = user_ta_enter_open_session, /* enter_open_session 挂载点 */
.enter_invoke_cmd = user_ta_enter_invoke_cmd,
.enter_close_session = user_ta_enter_close_session,
.dump_state = user_ta_dump_state,
.destroy = user_ta_ctx_destroy,
.get_instance_id = user_ta_get_instance_id,
};
/*
* Break unpaged attribute dependency propagation to user_ta_ops structure
* content thanks to a runtime initialization of the ops reference.
*/
static struct tee_ta_ops const *_user_ta_ops;
static TEE_Result init_user_ta(void)
{
_user_ta_ops = &user_ta_ops;
return TEE_SUCCESS;
}
service_init(init_user_ta); /* optee启动时将初始化user_ta_ops */
static void set_ta_ctx_ops(struct tee_ta_ctx *ctx)
{
ctx->ops = _user_ta_ops;
}
/* res = ctx->ops->enter_open_session(s, param, err)的入口 */
static TEE_Result user_ta_enter_open_session(struct tee_ta_session *s,
struct tee_ta_param *param, TEE_ErrorOrigin *eo)
{
return user_ta_enter(eo, s, UTEE_ENTRY_FUNC_OPEN_SESSION, 0, param);
}
static TEE_Result user_ta_enter(TEE_ErrorOrigin *err,
struct tee_ta_session *session,
enum utee_entry_func func, uint32_t cmd,
struct tee_ta_param *param)
{
TEE_Result res;
struct utee_params *usr_params;
uaddr_t usr_stack;
struct user_ta_ctx *utc = to_user_ta_ctx(session->ctx);
TEE_ErrorOrigin serr = TEE_ORIGIN_TEE;
struct tee_ta_session *s __maybe_unused;
void *param_va[TEE_NUM_PARAMS] = { NULL };
/* Map user space memory */
res = tee_mmu_map_param(utc, param, param_va);
if (res != TEE_SUCCESS)
goto cleanup_return;
/* Switch to user ctx */
tee_ta_push_current_session(session);
/* Make room for usr_params at top of stack */
usr_stack = utc->stack_addr + utc->mobj_stack->size;
usr_stack -= ROUNDUP(sizeof(struct utee_params), STACK_ALIGNMENT);
usr_params = (struct utee_params *)usr_stack;
init_utee_param(usr_params, param, param_va);
/* 将用户空间和内核空间都准备好后，开始层级切换 */
res = thread_enter_user_mode(func, tee_svc_kaddr_to_uref(session),
(vaddr_t)usr_params, cmd, usr_stack,
utc->entry_func, utc->is_32bit,
&utc->ctx.panicked, &utc->ctx.panic_code);
...
}
uint32_t thread_enter_user_mode(unsigned long a0, unsigned long a1,
unsigned long a2, unsigned long a3, unsigned long user_sp,
unsigned long entry_func, bool is_32bit,
uint32_t *exit_status0, uint32_t *exit_status1)
{
uint32_t spsr;
tee_ta_update_session_utime_resume();
if (!get_spsr(is_32bit, entry_func, &spsr)) {
*exit_status0 = 1; /* panic */
*exit_status1 = 0xbadbadba;
return 0;
}
/* a0: 指示进入TA的操作，如open_Session就是UTEE_ENTRY_FUNC_OPEN_SESSION
* a1: session结构体地址
* a2: 用户态传递过来的param
* a3: invoke_cmd使用的cmd， open_session时为0
* sp: kernel给TA分配的栈空间
* entry_func: sel0 TA入口
* spsr:之前保存下来的sel0的CPU状态寄存器的值
*/
return __thread_enter_user_mode(a0, a1, a2, a3, user_sp, entry_func,
spsr, exit_status0, exit_status1);
}
__thread_enter_user_mode进入一段汇编，层级切换基本思路相同，保存当前CPU状态寄存器，加载sel0的寄存器上下文，同时指定eret的地址，触发eret
/* core/arch/arm/kernel/thread_a64.S */
FUNC __thread_enter_user_mode , :
ldr x8, [sp]
/*
* Create the and fill in the struct thread_user_mode_rec
*/
sub sp, sp, #THREAD_USER_MODE_REC_SIZE
store_xregs sp, THREAD_USER_MODE_REC_EXIT_STATUS0_PTR, 7, 8
store_xregs sp, THREAD_USER_MODE_REC_X19, 19, 30
/*
* Switch to SP_EL1
* Disable exceptions
* Save kern sp in x19
*/
msr daifset, #DAIFBIT_ALL
mov x19, sp
msr spsel, #1
/*
* Save the kernel stack pointer in the thread context
*/
/* get pointer to current thread context */
get_thread_ctx sp, 21, 20, 22
/*
* Save kernel stack pointer to ensure that el0_svc() uses
* correct stack pointer
*/
str x19, [x21, #THREAD_CTX_KERN_SP]
/*
* Initialize SPSR, ELR_EL1, and SP_EL0 to enter user mode
*/
msr spsr_el1, x6
/* Set user sp */
mov x13, x4 /* Used when running TA in Aarch32 */
msr sp_el0, x4 /* Used when running TA in Aarch64 */
/* Set user function */
/* ta entry_func */
msr elr_el1, x5
/* Set frame pointer (user stack can't be unwound past this point) */
mov x29, #0
/* Jump into user mode */
store_xregs sp, THREAD_CORE_LOCAL_X0, 0, 1
/* 切换页表，TEE为每个TA提供ttbr0中的一段作为内存映射空间(L1); eret层级切换 */
b eret_to_el0
/* 进入TA， 即上文所说ta entry_func： __utee_entry */
END_FUNC __thread_enter_user_mode
TA入口
这一段就非常简单了，直接看代码：
/* lib/libutee/arch/arm/user_ta_entry.c */
void __noreturn __utee_entry(unsigned long func, unsigned long session_id,
struct utee_params *up, unsigned long cmd_id)
{
TEE_Result res;
switch (func) {
/* 即寄存器r0的值， UTEE_ENTRY_FUNC_OPEN_SESSION*/
case UTEE_ENTRY_FUNC_OPEN_SESSION:
res = entry_open_session(session_id, up);
break;
case UTEE_ENTRY_FUNC_CLOSE_SESSION:
res = entry_close_session(session_id);
break;
case UTEE_ENTRY_FUNC_INVOKE_COMMAND:
res = entry_invoke_command(session_id, up, cmd_id);
break;
default:
res = 0xffffffff;
TEE_Panic(0);
break;
}
ta_header_save_params(0, NULL);
utee_return(res);
}
static TEE_Result entry_open_session(unsigned long session_id,
struct utee_params *up)
{
TEE_Result res;
struct ta_session *session;
uint32_t param_types;
TEE_Param params[TEE_NUM_PARAMS];
/* 为TA指定上session， 注意一个TA可能被多个CA调用，有多个session */
res = ta_header_add_session(session_id);
if (res != TEE_SUCCESS)
return res;
session = ta_header_get_session(session_id);
if (!session)
return TEE_ERROR_BAD_STATE;
/* 经过漫长过程终于传递到TA的， CA写下的param */
__utee_to_param(params, ¶m_types, up);
ta_header_save_params(param_types, params);
/* 最上层所说的GP接口， 在TA的SDK中， 由用户实现此接口 */
res = TA_OpenSessionEntryPoint(param_types, params,
&session->session_ctx);
__utee_from_param(up, param_types, params);
if (res != TEE_SUCCESS)
ta_header_remove_session(session_id);
return res;
}
完成操作后，将一步步返回到CA，基本原路返回所以构成类似不细说了。 终于一段漫长的open_session的过程结束了。
TA中GP标准实现接口
参考官方文档：https://optee.readthedocs.io/en/latest/architecture/globalplatform_api.html
TEE->tee-supplicant->TEE(RPC请求)
首先我们要知道tee-supplicant是啥：
tee_supplicant的主要作用是使OP-TEE能够通过tee_supplicant来访问REE端文件系统中的资源，例如加载存放在文件系统中的TA镜像到TEE中，对REE端数据库的操作，对EMMC中RPMB分区的操作，提供socket通信等。 其源代码optee_client/tee-supplicant目录中。编译之后会生成一个名字为tee_supplicant的可执行文件，该可执行文件在REE启动的时候会作为一个后台程序被自动启动，而且常驻于系统中。
原文链接：https://blog.csdn.net/shuaifengyun/java/article/details/72912238
其实就是一个常驻在非安全侧，用于响应TEE一些需要在非安全侧完成的请求的一个用户态服务守护进程。既然是在运行在用户态，想必中间又要经过ATF->tee_driver->tee_supplicant的流程。
TEE的官方文档有一个TEE发送请求加载TA的流程图（OPTEE中通常把这种请求叫做RPC），我们首先留下一个完整的印象：
image.png
我们还是从TA加载部分开始说起，第一步探查TEE发送rpc命令到非安全侧tee_driver的操作：
TEE->tee_driver
一切要从ta_store->open(uuid, &handle)上挂载的ree_fs_ta_open函数说起：他将调用rpc_load, 这是TEE发送rpc请求到非安全侧的起点：
/* core/arch/arm/kernel/ree_fs_ta.c */
static TEE_Result ree_fs_ta_open(const TEE_UUID *uuid,
struct user_ta_store_handle **h)
{
...
/* Request TA from tee-supplicant */
res = rpc_load(uuid, &ta, &ta_size, &mobj);
if (res != TEE_SUCCESS)
goto error;
...
}
static TEE_Result rpc_load(const TEE_UUID *uuid, struct shdr **ta,
size_t *ta_size, struct mobj **mobj)
{
TEE_Result res;
struct thread_param params[2];
if (!uuid || !ta || !mobj || !ta_size)
return TEE_ERROR_BAD_PARAMETERS;
memset(params, 0, sizeof(params));
params[0].attr = THREAD_PARAM_ATTR_VALUE_IN;
/* 将请求的UUID写入参数中 */
tee_uuid_to_octets((void *)¶ms[0].u.value, uuid);
params[1].attr = THREAD_PARAM_ATTR_MEMREF_OUT;
/* 发送第一个rpc，有下面可以看到请求到的是TA文件的size */
res = thread_rpc_cmd(OPTEE_RPC_CMD_LOAD_TA, 2, params);
if (res != TEE_SUCCESS)
return res;
/* 发送第二个rpc，根据读取到的TA size申请共享内存 */
*mobj = thread_rpc_alloc_payload(params[1].u.memref.size);
if (!*mobj)
return TEE_ERROR_OUT_OF_MEMORY;
if ((*mobj)->size < params[1].u.memref.size) {
res = TEE_ERROR_SHORT_BUFFER;
goto exit;
}
*ta = mobj_get_va(*mobj, 0);
/* We don't expect NULL as thread_rpc_alloc_payload() was successful */
assert(*ta);
*ta_size = params[1].u.memref.size;
params[0].attr = THREAD_PARAM_ATTR_VALUE_IN;
tee_uuid_to_octets((void *)¶ms[0].u.value, uuid);
params[1].attr = THREAD_PARAM_ATTR_MEMREF_OUT;
params[1].u.memref.offs = 0;
params[1].u.memref.mobj = *mobj;
/* 发送第三个rpc，将TA首先加载如共享内存当中 */
res = thread_rpc_cmd(OPTEE_RPC_CMD_LOAD_TA, 2, params);
exit:
if (res != TEE_SUCCESS)
thread_rpc_free_payload(*mobj);
return res;
}
可以看到thread_rpc_cmd是发送rpc请求的接口，直接分析：
/* core/arch/arm/kernel/thread.c */
uint32_t thread_rpc_cmd(uint32_t cmd, size_t num_params,
struct thread_param *params)
{
uint32_t rpc_args[THREAD_RPC_NUM_ARGS] = { OPTEE_SMC_RETURN_RPC_CMD };
void *arg = NULL;
uint64_t carg = 0;
uint32_t ret = 0;
/* 套路和driver进入TEE驱动基本一致，还是使用optee_msg_arg结构体，
* 准备好cmd填在msg->cmd， params复制到msg->params;
* 然后获取结构体的地址的mobj描述块的物理地址，后续将填入r1, r2, r0填入OPTEE_SMC_RETURN_RPC_CMD */
/* The source CRYPTO_RNG_SRC_JITTER_RPC is safe to use here */
plat_prng_add_jitter_entropy(CRYPTO_RNG_SRC_JITTER_RPC,
&thread_rpc_pnum);
/* 事实上，当第一次调用带有params的thread_rpc_cmd需要完成两次安全与非安全的切换
* 首先需要获取一块共享内存描述块的物理地址，用于params的通讯。
*/
ret = get_rpc_arg(cmd, num_params, params, &arg, &carg);
if (ret)
return ret;
reg_pair_from_64(carg, rpc_args + 1, rpc_args + 2);
/* 很明显thread_rpc是一段汇编，将触发smc，进入ATF后切换回tee_driver */
thread_rpc(rpc_args);
return get_rpc_arg_res(arg, num_params, params);
}
/* core/arch/arm/kernel/thread_a64.S */
/* 操作也是类似， 保存好寄存器上下文，触发smc, 进入ATF， ATF将引导进入tee_driver入口 */
/* void thread_rpc(uint32_t rv[THREAD_RPC_NUM_ARGS]) */
FUNC thread_rpc , :
/* Read daif and create an SPSR */
mrs x1, daif
orr x1, x1, #(SPSR_64_MODE_EL1 << SPSR_64_MODE_EL_SHIFT)
/* Mask all maskable exceptions before switching to temporary stack */
msr daifset, #DAIFBIT_ALL
push x0, xzr
push x1, x30
bl thread_get_ctx_regs
ldr x30, [sp, #8] /* 恢复rpc返回的地址 */
store_xregs x0, THREAD_CTX_REGS_X19, 19, 30 /* 返回地址在thread.reg[30] */
mov x19, x0
bl thread_get_tmp_sp
pop x1, xzr /* Match "push x1, x30" above */
mov x2, sp
str x2, [x19, #THREAD_CTX_REGS_SP]
ldr x20, [sp] /* Get pointer to rv[] */
mov sp, x0 /* Switch to tmp stack */
/*
* We need to read rv[] early, because thread_state_suspend
* can invoke virt_unset_guest() which will unmap pages,
* where rv[] resides
*/
load_wregs x20, 0, 21, 23 /* Load rv[] into w20-w22 */
adr x2, .thread_rpc_return /* 注意，此处有一处重要的点，这里指定上的rpc返回tee后thread的入口 */
mov w0, #THREAD_FLAGS_COPY_ARGS_ON_RETURN
bl thread_state_suspend
mov x4, x0 /* Supply thread index */
ldr w0, =TEESMC_OPTEED_RETURN_CALL_DONE
mov x1, x21
mov x2, x22
mov x3, x23
smc #0 /* CPU进入el3，之后切回非安全侧el1，进入tee_driver */
b . /* SMC should not return */
/* core/arch/arm/kernel/thread.c */
int thread_state_suspend(uint32_t flags, uint32_t cpsr, vaddr_t pc)
{
struct thread_core_local *l = thread_get_core_local();
int ct = l->curr_thread;
...
assert(threads[ct].state == THREAD_STATE_ACTIVE);
/* 恢复thread后的pc此处指定：thread_rpc_return*/
threads[ct].flags |= flags;
threads[ct].regs.cpsr = cpsr;
threads[ct].regs.pc = pc;
threads[ct].state = THREAD_STATE_SUSPENDED;
...
}
tee-supplicant与tee_driver的协同
当libteec调用驱动来与OP-TEE进行数据的交互的时候，最终会调用optee_do_call_with_arg函数完成完成smc的操作，而该函数中有一个loop循环，每次触发smc操作之后会对从secure world中返回的参数res.a0进行判断，判定当前从secure world返回的数据是要执行RPC操作还是直接返回到CA。如果是来自TEE的RPC请求，则会将请求存放到请求队列req中。然后block住，直到tee_supplicant处理完请求并将req->c标记为完成之后才会进入下一个loop，重新出发smc操作，将处理结果返回给TEE。
原文链接：https://blog.csdn.net/shuaifengyun/java/article/details/73061002
image.png
所以第一步我们找到第一次发送smc_call的地方，看他之后做了啥：
/* driver/tee/optee/call.c */
u32 optee_do_call_with_arg(struct tee_context *ctx, phys_addr_t parg)
{
...
while (true) {
struct arm_smccc_res res;
optee_bm_timestamp();
/* smc调用点 */
optee->invoke_fn(param.a0, param.a1, param.a2, param.a3,
param.a4, param.a5, param.a6, param.a7,
&res);
optee_bm_timestamp();
if (res.a0 == OPTEE_SMC_RETURN_ETHREAD_LIMIT) {
/*
* Out of threads in secure world, wait for a thread
* become available.
*/
optee_cq_wait_for_completion(&optee->call_queue, &w);
} else if (OPTEE_SMC_RETURN_IS_RPC(res.a0)) { /* 此处的r0由tee填写，处理rpc进入*/
might_sleep();
param.a0 = res.a0;
param.a1 = res.a1;
param.a2 = res.a2;
param.a3 = res.a3;
/* 处理rpc函数 */
optee_handle_rpc(ctx, ¶m, &call_ctx);
} else {
ret = res.a0;
break;
}
}
}
后续流程就不上源码了，通过几个很简单的几个switch找到对应的处理函数, 并且通过物理r1和r2传递过来的物理地址获取的共享内存块，结构体中可以取到params写入的在非安全侧映射的虚拟地址：
optee_handle_rpc->handle_rpc_func_cmd(在内核能解决掉的请求都在这里处理完成了)->handle_rpc_supp_cmd（需要动用到tee-supplicant的请求到这里）
/* driver/tee/optee/rpc.c */
static void handle_rpc_supp_cmd(struct tee_context *ctx,
struct optee_msg_arg *arg)
{
struct tee_param *params;
arg->ret_origin = TEEC_ORIGIN_COMMS;
params = kmalloc_array(arg->num_params, sizeof(struct tee_param),
GFP_KERNEL);
if (!params) {
arg->ret = TEEC_ERROR_OUT_OF_MEMORY;
return;
}
/* 将params从共享内存中拷贝到非安全侧driver的内存中 */
if (optee_from_msg_param(params, arg->num_params, arg->params)) {
arg->ret = TEEC_ERROR_BAD_PARAMETERS;
goto out;
}
/* 将cmd送入tee-supplicant的处理队列中 */
arg->ret = optee_supp_thrd_req(ctx, arg->cmd, arg->num_params, params);
if (optee_to_msg_param(arg->params, arg->num_params, params))
arg->ret = TEEC_ERROR_BAD_PARAMETERS;
out:
kfree(params);
}
/* driver/tee/optee/supp.c */
u32 optee_supp_thrd_req(struct tee_context *ctx, u32 func, size_t num_params,
struct tee_param *param)
{
struct optee *optee = tee_get_drvdata(ctx->teedev);
struct optee_supp *supp = &optee->supp;
struct optee_supp_req *req = kzalloc(sizeof(*req), GFP_KERNEL);
bool interruptable;
u32 ret;
if (!req)
return TEEC_ERROR_OUT_OF_MEMORY;
/* 初始化该请求消息的c成员并配置请求数据 */
init_completion(&req->c);
req->func = func;
req->num_params = num_params;
req->param = param;
/* Insert the request in the request list */
/* 将接受到的请求添加到驱动的TEE请求消息队列中 */
mutex_lock(&supp->mutex);
list_add_tail(&req->link, &supp->reqs);
mutex_unlock(&supp->mutex);
/* Tell an eventual waiter there's a new request */
/* 将supp->reqs_c置位，通知tee_supplicant的receive操作，当前驱动中
有一个来自TEE的请求 */
complete(&supp->reqs_c);
/*
* Wait for supplicant to process and return result, once we've
* returned from wait_for_completion(&req->c) successfully we have
* exclusive access again.
*/
/* block在这里，通过判定req->c是否被置位来判定当前请求是否被处理完毕，
而req->c的置位是有tee_supplicant的send调用来完成的，如果被置位，则进入到
while循环中进行返回值的设定并跳出while*/
while (wait_for_completion_interruptible(&req->c)) {
mutex_lock(&supp->mutex);
interruptable = !supp->ctx;
if (interruptable) {
/*
* There's no supplicant available and since the
* supp->mutex currently is held none can
* become available until the mutex released
* again.
*
* Interrupting an RPC to supplicant is only
* allowed as a way of slightly improving the user
* experience in case the supplicant hasn't been
* started yet. During normal operation the supplicant
* will serve all requests in a timely manner and
* interrupting then wouldn't make sense.
*/
interruptable = !req->busy;
if (!req->busy)
list_del(&req->link);
}
mutex_unlock(&supp->mutex);
if (interruptable) {
req->ret = TEEC_ERROR_COMMUNICATION;
break;
}
}
ret = req->ret;
kfree(req);
return ret;
}
来看看tee-supplicant是如何接收到这个request的，先来一个tee-supplicant的总体流程：
image.png
tee-supplicant在系统启动时就将启动起来，常驻在用户态，陷入一个无限循环等待接收request：
/* optee_client/tee-supplicant/src/tee_supplicant.c */
int main(int argc, char *argv[])
{
struct thread_arg arg = { .fd = -1 };
int e;
/* 初始化互斥体 */
e = pthread_mutex_init(&arg.mutex, NULL);
if (e) {
EMSG("pthread_mutex_init: %s", strerror(e));
EMSG("terminating...");
exit(EXIT_FAILURE);
}
/* 判定是否带有启动参数，如果带有启动参数，则打开对应的驱动文件
如果没有带参数，则打开默认的驱动文件 */
if (argc > 2)
return usage();
if (argc == 2) {
arg.fd = open_dev(argv[1]);
if (arg.fd < 0) {
EMSG("failed to open \"%s\"", argv[1]);
exit(EXIT_FAILURE);
}
} else {
/*打开/dev/teepriv0设备，该设备为tee驱动设备文件，返回操作句柄*/
arg.fd = get_dev_fd();
if (arg.fd < 0) {
EMSG("failed to find an OP-TEE supplicant device");
exit(EXIT_FAILURE);
}
}
if (tee_supp_fs_init() != 0) {
EMSG("error tee_supp_fs_init");
exit(EXIT_FAILURE);
}
if (sql_fs_init() != 0) {
EMSG("sql_fs_init() failed ");
exit(EXIT_FAILURE);
}
/* 调用process_one_request函数接收来自TEE的请求，并加以处理 */
while (!arg.abort) {
if (!process_one_request(&arg))
arg.abort = true;
}
close(arg.fd);
return EXIT_FAILURE;
}
之后的处理，大佬的博客总结的已经很完整了，直接自取吧：
https://blog.csdn.net/shuaifengyun/article/details/72912238
我们主要看write_response之后，tee_driver是怎么做的:
/* optee_client/tee-supplicant/src/tee_supplicant.c */
static bool write_response(int fd, union tee_rpc_invoke *request)
{
struct tee_ioctl_buf_data data;
/* 将需要返回给TA的数据存放在buffer中 */
data.buf_ptr = (uintptr_t)&request->send;
data.buf_len = sizeof(struct tee_iocl_supp_send_arg) +
sizeof(struct tee_ioctl_param) *
request->send.num_params;
/* 调用驱动中ioctl函数的TEE_IOC_SUPPL_SEND功能，进数据发送给TA */
if (ioctl(fd, TEE_IOC_SUPPL_SEND, &data)) {
EMSG("TEE_IOC_SUPPL_SEND: %s", strerror(errno));
return false;
}
return true;
/* driver/tee/tee_core.c */
static long tee_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
struct tee_context *ctx = filp->private_data;
void __user *uarg = (void __user *)arg;
switch (cmd) {
...
case TEE_IOC_SUPPL_SEND:
return tee_ioctl_supp_send(ctx, uarg);
...
}
}
static int tee_ioctl_supp_send(struct tee_context *ctx,
struct tee_ioctl_buf_data __user *ubuf)
{
long rc;
struct tee_ioctl_buf_data buf;
struct tee_iocl_supp_send_arg __user *uarg;
struct tee_param *params;
u32 num_params;
u32 ret;
/* Not valid for this driver */
...
rc = ctx->teedev->desc->ops->supp_send(ctx, ret, num_params, params);
out:
kfree(params);
return rc;
}
/* driver/tee/optee/core.c */
static const struct tee_driver_ops optee_supp_ops = {
.get_version = optee_get_version,
.open = optee_open,
.release = optee_release,
.supp_recv = optee_supp_recv,
.supp_send = optee_supp_send, /* 挂载的函数 */
.shm_register = optee_shm_register_supp,
.shm_unregister = optee_shm_unregister_supp,
};
/* driver/tee/optee/supp.c */
int optee_supp_send(struct tee_context *ctx, u32 ret, u32 num_params,
struct tee_param *param)
{
struct tee_device *teedev = ctx->teedev;
struct optee *optee = tee_get_drvdata(teedev);
struct optee_supp *supp = &optee->supp;
struct optee_supp_req *req;
size_t n;
size_t num_meta;
mutex_lock(&supp->mutex);
/* 驱动中请求队列的pop操作 */
req = supp_pop_req(supp, num_params, param, &num_meta);
mutex_unlock(&supp->mutex);
if (IS_ERR(req)) {
/* Something is wrong, let supplicant restart. */
return PTR_ERR(req);
}
/* Update out and in/out parameters */
/* 使用传入的参数，更新请求的参数区域，将需要返回给TEE侧的数据填入到对应的位置 */
for (n = 0; n < req->num_params; n++) {
struct tee_param *p = req->param + n;
switch (p->attr & TEE_IOCTL_PARAM_ATTR_TYPE_MASK) {
case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_OUTPUT:
case TEE_IOCTL_PARAM_ATTR_TYPE_VALUE_INOUT:
p->u.value.a = param[n + num_meta].u.value.a;
p->u.value.b = param[n + num_meta].u.value.b;
p->u.value.c = param[n + num_meta].u.value.c;
break;
case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_OUTPUT:
case TEE_IOCTL_PARAM_ATTR_TYPE_MEMREF_INOUT:
p->u.memref.size = param[n + num_meta].u.memref.size;
break;
default:
break;
}
}
req->ret = ret;
/* Let the requesting thread continue */
/* 通知optee_supp_thrd_req函数，一个来自TEE侧的请求已经被处理完毕，
可以继续往下执行 */
complete(&req->c);
return 0;
}
optee_supp_thrd_reqblock将结束，继续往下执行，返回TEE
tee-supplicant
我们还是以TA_load这个命令为例：
process_one_request中的read_request从队列中读取到请求和TEE过来的param后，获取请求cmd
/* optee_client/tee-supplicant/src/tee_supplicant.c */
static bool process_one_request(struct thread_arg *arg)
{
size_t num_params = 0;
size_t num_meta = 0;
struct tee_ioctl_param *params = NULL;
uint32_t func = 0;
uint32_t ret = 0;
union tee_rpc_invoke request;
memset(&request, 0, sizeof(request));
DMSG("looping");
request.recv.num_params = RPC_NUM_PARAMS;
/* Let it be known that we can deal with meta parameters */
params = (struct tee_ioctl_param *)(&request.send + 1);
params->attr = TEE_IOCTL_PARAM_ATTR_META;
num_waiters_inc(arg);
/* 阻塞点，陷入内核中等待事件 */
if (!read_request(arg->fd, &request))
return false;
if (!find_params(&request, &func, &num_params, ¶ms, &num_meta))
return false;
if (num_meta && !num_waiters_dec(arg) && !spawn_thread(arg))
return false;
switch (func) {
case OPTEE_MSG_RPC_CMD_LOAD_TA:
/* load_ta使我们的目标函数 */
ret = load_ta(num_params, params);
break;
...
}
static uint32_t load_ta(size_t num_params, struct tee_ioctl_param *params)
{
int ta_found = 0;
size_t size = 0;
TEEC_UUID uuid;
struct tee_ioctl_param_value *val_cmd;
TEEC_SharedMemory shm_ta;
memset(&shm_ta, 0, sizeof(shm_ta));
/* 解析出需要加载的TA镜像的UUID以及配置将读取到的TA镜像的内容存放位置 */
if (num_params != 2 || get_value(num_params, params, 0, &val_cmd) ||
get_param(num_params, params, 1, &shm_ta))
return TEEC_ERROR_BAD_PARAMETERS;
/* 将UUID的值转换成TEEC_UUID格式 */
uuid_from_octets(&uuid, (void *)val_cmd);
/* 从ta_dir变量指定的目录中查找与UUID相符的TA镜像，并将其内容读取到共享内存中 */
size = shm_ta.size;
/* 此函数在一次加载TA流程中将经过两次调用，
* 第一次在shm_ta.buffer为空时，将只获取ta文件的大小，回给TEE侧
* 第二次才会去将TA加载进共享内存中
*/
ta_found = TEECI_LoadSecureModule(ta_dir, &uuid, shm_ta.buffer, &size);
if (ta_found != TA_BINARY_FOUND) {
EMSG(" TA not found");
return TEEC_ERROR_ITEM_NOT_FOUND;
}
/* 设定读取到的TA镜像的大小到返回参数的size成员中 */
params[1].u.memref.size = size;
return TEEC_SUCCESS;
}
tee_driver->TEE
调用点没有变化，还是optee->invoke, 进入ATF，确定入口，进入thread_resume恢复线程，以前的流程区别不大，只是恢复线程之后的pc指针是上文提到的thread_rpc_return
void thread_handle_std_smc(struct thread_smc_args *args)
{
thread_check_canaries();
#ifdef CFG_VIRTUALIZATION
if (!virt_set_guest(args->a7)) {
args->a0 = OPTEE_SMC_RETURN_ENOTAVAIL;
return;
}
#endif
if (args->a0 == OPTEE_SMC_CALL_RETURN_FROM_RPC)
/* 走这个分支 */
thread_resume_from_rpc(args);
else
thread_alloc_and_run(args);
#ifdef CFG_VIRTUALIZATION
virt_unset_guest();
#endif
...
}
static void thread_resume_from_rpc(struct thread_smc_args *args)
{
size_t n = args->a3; /* thread id */
struct thread_core_local *l = thread_get_core_local();
uint32_t rv = 0;
assert(l->curr_thread == -1);
lock_global();
/* 挂起状态的线程恢复active */
if (n < CFG_NUM_THREADS &&
threads[n].state == THREAD_STATE_SUSPENDED &&
args->a7 == threads[n].hyp_clnt_id)
threads[n].state = THREAD_STATE_ACTIVE;
else
rv = OPTEE_SMC_RETURN_ERESUME;
unlock_global();
if (rv) {
args->a0 = rv;
return;
}
l->curr_thread = n;
if (is_user_mode(&threads[n].regs))
tee_ta_update_session_utime_resume();
if (threads[n].have_user_map)
core_mmu_set_user_map(&threads[n].user_map);
/*
* Return from RPC to request service of a foreign interrupt must not
* get parameters from non-secure world.
*/
if (threads[n].flags & THREAD_FLAGS_COPY_ARGS_ON_RETURN) {
copy_a0_to_a5(&threads[n].regs, args);
threads[n].flags &= ~THREAD_FLAGS_COPY_ARGS_ON_RETURN;
}
thread_lazy_save_ns_vfp();
/* 线程恢复，返回thread_rpc_return */
thread_resume(&threads[n].regs);
}
thread_resume先eret回到加载回的el1_elr中，对应thread.regs.pc, 即thread_rpc_return，同时此时ret指令对应的跳转地址寄存器为r30, 也从thread.regs.x[30]中回复(具体可以参看上文thread_resume，有注释)
.thread_rpc_return:
/*
* At this point has the stack pointer been restored to the value
* stored in THREAD_CTX above.
*
* Jumps here from thread_resume above when RPC has returned. The
* IRQ and FIQ bits are restored to what they where when this
* function was originally entered.
*/
pop x16, xzr /* Get pointer to rv[] */
store_wregs x16, 0, 0, 5 /* Store w0-w5 into rv[] */
/* 此处调用ret， 返回x30地址， 即thread_rpc函数的下一句地址，整个rpc流程走完 */
ret
END_FUNC thread_rpc
至此rpc流程历经一大堆状态切换终于完成，重新回到业务流程thread_rpc_cmd函数中。
其他补充
以上流程看起来已经相当复杂，然而这才只是冰山一角, 还有很多有关内存管理，安全存储相关的资料，在此处总结一下
QEMU环境搭建
官方v7: https://optee.readthedocs.io/en/latest/building/devices/qemu.html
博客关于v8：https://blog.csdn.net/shuaifengyun/article/details/99855105
optee的密码库
主要使用的是libtomcrypt与libmbed，都运行在TEE_kernel中
安全存储相关
https://blog.csdn.net/shuaifengyun/article/list/1 36-42
optee内部system call
https://blog.csdn.net/shuaifengyun/article/details/73326870
内存管理
暂无比较完整的资料，后续自己补充。
最后
本文图片主要来自GP和optee官方文档，以及博客：https://icyshuai.blog.csdn.net/
水平很菜，欢迎指正
