权限认证

一、kubernetes权限RBAC

<font color=#999AAA >kubernetes使用的RBAC的示意图如下：user1 可以使用role1和role2所允许的资源，user2只可以使用 role1 所允许的资源。我们有时候会称user为subject

image

在这里插入图片描述

在<code>kubernetes，serviceaccount、user、group</code>为<code>user。、clusterrole和role代表role。clusterrolebinding</code>和<code>rolebinding代表rolebinding。

• GlobalRole（CRD资源）和ClusterRole、Role：全局角色，集群角色，项目角色，这三个都包含Rules，定义了这个Role所能允许的资源。三者的区别在于，在Role中是有namespace的，属于单个命名空间，ClusterRole是单个集群的，GlobalRole是多个集群的。
• Subject ：里面对应相应的用户如上图所示，Subject里面包含：apiGroup、Kind、Name（User的Name）
• RoleBinding、ClusterRoleBinding、GlobalRoleBinding：将Role与Subject绑定。确定每一个用户都有确定的Rules（权限）。

二、验证权限的过程

1. 填充请求。确定请求的范围，单集群还是多集群，是否是发给k8s的请求
2. 验证用户，密码、token等方式，如果是无用户则定义为anonymous
3. 如果是多集群，则需要转发。
4. 确定该资源是否能被用户获取
   1. 通过user，globalRoleBinding获取到globalRole，查看GlobalRole对该资源是否允许。
   2. 通过user，ClusterroleBinding获取到Clusterrole
   3. 通过user，Rolebinding获取到Role，查看Role对资源是否允许
   为了减少查询次数，通过if来过滤
5. 审计
6. 如果是k8s资源，类似于kubectl get pod -A的资源，则转发给kube-apiserver

三、代码分析

kubesphere使用的是iam模块来进行权限和权限获取，kubesphere使用go-restful开发 web应用框架，权限认证和权限获取都是在filter进行的
权限代码如下： 注意：filter和handle的方式一样，先被封装后被运行。所以下面请求的执行过程是：
<font color=#999AAA >WithRequestInfo--->WithAuthentication--->WithMultipleClusterDispatcher--->WithAuthorization--->WithAuditing---->WithKubeAPIServer

func (s *APIServer) buildHandlerChain(stopCh <-chan struct{}) {
    //.....
    handler := s.Server.Handler
    handler = filters.WithKubeAPIServer(handler, s.KubernetesClient.Config(), &errorResponder{})
//开启审计
    if s.Config.AuditingOptions.Enable {
        handler = filters.WithAuditing(handler,
            audit.NewAuditing(s.InformerFactory, s.Config.AuditingOptions.WebhookUrl, stopCh))
    }
    //授权
    handler = filters.WithAuthorization(handler, authorizers)
    if s.Config.MultiClusterOptions.Enable {
        //多集群转发
        handler = filters.WithMultipleClusterDispatcher(handler, clusterDispatcher)
    }

    loginRecorder := im.NewLoginRecorder(s.KubernetesClient.KubeSphere())
    //认证
    handler = filters.WithAuthentication(handler, authn, loginRecorder)
    //WithRequestInfo
    handler = filters.WithRequestInfo(handler, requestInfoResolver)
    s.Server.Handler = handler
}

主要分为以下步骤

1. WithRequestInfo ：填充请求信息
2. WithAuthentication：权限认证
3. WithMultipleClusterDispatcher：多集群路径填补和转发，多集群时根据requestinfo，将信息发送给对应的集群。
4. WithAuthorization：权限获取，判断是否对应集群资源有获取权限。
   5.WithAuditing：审计
   6.WithKubeAPIServer：转发，如果是直接对k8s的请求，则直接转发给k8s。

四、填充请求信息

请求信息如下：

type RequestInfo struct {
    // import k8s.io/apiserver/pkg/endpoints/request
    *k8srequest.RequestInfo
    //是否是k8s的请求，如果是k8s的请求，则会直接转发到kube-apiserver
    IsKubernetesRequest bool

    //请求资源所在的Workspace，可以为空
    Workspace string

    //请求资源所在的Cluster，如果是单集群则为空
    Cluster string

    //请求资源所在的devops项目
    DevOps string

    // 资源请求范围
    ResourceScope string
}

认证
有三个认证方式，anoymous、password、token三个方法任意一个通过校验，就立刻返回。

1. anoymous的实现方法如下

 func (a *Authenticator) AuthenticateRequest(req *http.Request) (*authenticator.Response, bool, error) {
    auth := strings.TrimSpace(req.Header.Get("Authorization"))
    if auth == "" {
        return &authenticator.Response{
            User: &user.DefaultInfo{
                Name:   user.Anonymous,
                UID:    "",
                Groups: []string{user.AllUnauthenticated},
            },
        }, true, nil
    }
    return nil, false, nil
   }

2. password的实现方法如下：从user资源中获取信息

func (im *passwordAuthenticator) Authenticate(username, password string) (authuser.Info, error) {
    //从k8s 集群中查询到user资源
    user, err := im.searchUser(username)
    
    providerOptions, ldapProvider := im.getLdapProvider()
    //....
       //通过ldapProvider来判断账号密码是否正确
    if ldapProvider != nil && username != constants.AdminUserName {
        //
        authenticated, err := ldapProvider.Authenticate(username, password)
        //....
        if authenticated != nil {
            return &authuser.DefaultInfo{
                Name: authenticated.Name,
                UID:  string(authenticated.UID),
            }, nil
        }
    }
    //也可以通过use资源的EncryptedPassword，验证密码
    if checkPasswordHash(password, user.Spec.EncryptedPassword) {
        return &authuser.DefaultInfo{
            Name: user.Name,
            UID:  string(user.UID),
        }, nil
    }
   
    return nil, AuthFailedIncorrectPassword
   }

3. token：使用jwt验证

  func (t tokenOperator) Verify(tokenStr string) (user.Info, error) {
       //调用jwt包验证
    authenticated, tokenType, err := t.issuer.Verify(tokenStr)
    //这里应该是过期时间，0为永不过期。这个为配置设置
    if t.options.OAuthOptions.AccessTokenMaxAge == 0 ||
        tokenType == token.StaticToken {
        return authenticated, nil
    }
       //redis验证
    if err := t.tokenCacheValidate(authenticated.GetName(), tokenStr); err != nil {
        klog.Error(err)
        return nil, err
    }
    return authenticated, nil
   }

五、多集群路由转发和协议升级

只在多集群下运行此函数

func (c *clusterDispatch) Dispatch(w http.ResponseWriter, req *http.Request, handler http.Handler) {
    info, _ := request.RequestInfoFrom(req.Context())
    //cluster 是crd 资源。获取集群所有cluster信息
    cluster, err := c.clusterLister.Get(info.Cluster)
    //请求集群是主机集群，不需要通过代理
    if isClusterHostCluster(cluster) {
        req.URL.Path = strings.Replace(req.URL.Path, fmt.Sprintf("/clusters/%s", info.Cluster), "", 1)
        handler.ServeHTTP(w, req)
        return
    }
    //查询集群
    innCluster := c.getInnerCluster(cluster.Name)
    transport := http.DefaultTransport
    //替换url
    u := *req.URL
    u.Path = strings.Replace(u.Path, fmt.Sprintf("/clusters/%s", info.Cluster), "", 1)
    if cluster.Spec.Connection.Type == clusterv1alpha1.ConnectionTypeDirect &&
        len(cluster.Spec.Connection.KubeSphereAPIEndpoint) == 0 {
        u.Scheme = innCluster.kubernetesURL.Scheme
        u.Host = innCluster.kubernetesURL.Host
        u.Path = fmt.Sprintf(proxyURLFormat, u.Path)
        transport = innCluster.transport
         //...
    } else {
        u.Host = innCluster.kubesphereURL.Host
        u.Scheme = innCluster.kubesphereURL.Scheme
    }
    
    httpProxy := proxy.NewUpgradeAwareHandler(&u, transport, false, false, c)
    httpProxy.ServeHTTP(w, req)
}

六、权限获取

首先在apiserver层中初始化Authorizer，然后进行授权模式的判断 代码如下：

var authorizers authorizer.Authorizer

    switch a.Config.AuthorizationOptions.Mode{
    case authorizationoptions.AlwaysAllow:
        authorizers = authorizerfactory.NewAlwaysAllowAuthorizer()
    case authorizationoptions.AlwaysDeny:
        authorizers = authorizerfactory.NewAlwaysDenyAuthorizer()
    default:
        fallthrough
    case authorizationoptions.RBAC:
    //放行的所有路径：如登录等
        excludePaths := []string{"/login/*"}
        pathAuthorizer,_ :=path.NewAuthorizer(excludePaths)
//RBAC权限判断，url鉴权
        RBACAuthorizer:=rbac.NewRBACAuthorizer(a.KubernetesClient.Kubernetes(),a.KubernetesClient.Kubesphere())
        //将放行的路径和RBAC权限授权放到unionauthorizer里面，然后进行首先判断，是不是放行的路径，根据pathAuthorizer返回的Authorizer，来判断如果是放行的路径，直接放行，否则进行RBAC权限判定
        authorizers = unionauthorizer.New(pathAuthorizer,RBACAuthorizer)
    }
//将授权加到handler里面
    handler = middleware.WithAuthorization(handler,authorizers)

path放行路径的截取
代码如下：

    var prefixes []string
    paths := sets.NewString()
    for _, p := range alwaysAllowPaths {
    //将放行的URL的前缀/去掉，如：/login/*去掉后为login/*
        p = strings.TrimPrefix(p, "/")
        if len(p) == 0 {
            paths.Insert(p)
            continue
        }
        //判断放行的URL是不是写的为/**
        if strings.ContainsRune(p[:len(p)-1], '*') {
            return nil, fmt.Errorf("only trailing * allowed in %q", p)
        }
        //去掉后面的*
        if strings.HasSuffix(p, "*") {
            prefixes = append(prefixes, p[:len(p)-1])
        } else {
            paths.Insert(p)
        }
    }
    return authorizer.AuthorizerFunc(func(a authorizer.Attributes) (authorizer.Decision, string, error) {
        pth := strings.TrimPrefix(a.GetPath(), "/")
        //如果是不带*的则需要用户请求的路径和放行的路径完全相同
        if paths.Has(pth) {
            return authorizer.DecisionAllow, "", nil
        }
        //如果是/*的则只要前缀为例如/login的即可
        for _, prefix := range prefixes {
            if strings.HasPrefix(pth, prefix) {
            //放行
                return authorizer.DecisionAllow, "", nil
            }
        }

        return authorizer.DecisionNoOpinion, "", nil
    }), nil

Union来判断是放行的路径，还是进行RBAC授权

//...authorizer.Authorizer 代表可以存放多个类型为Authorizer的结构体（即：放行的路径、RBAC授权）
func New(authorizationHandlers ...authorizer.Authorizer) authorizer.Authorizer {
    return unionAuthzHandler(authorizationHandlers)
}

func (authzHandler unionAuthzHandler) Authorize(a authorizer.Attributes) (authorizer.Decision, string, error) {
    var (
        errlist    []error
        reasonlist []string
    )

    for _, currAuthzHandler := range authzHandler {
        decision, reason, err := currAuthzHandler.Authorize(a)

        if err != nil {
            errlist = append(errlist, err)
        }
        if len(reason) != 0 {
            reasonlist = append(reasonlist, reason)
        }
        //根据截取放行的路径返回的放行DecisionAllow，进行放行
        switch decision {
        case authorizer.DecisionAllow, authorizer.DecisionDeny:
            return decision, reason, err
        case authorizer.DecisionNoOpinion:
            // continue to the next authorizer
        }
    }

    return authorizer.DecisionNoOpinion, strings.Join(reasonlist, "\n"), utilerrors.NewAggregate(errlist)
}

RBAC用户授权

func appliesTo(user user.Info, bindingSubjects []rbacv1.Subject, namespace string) (int, bool) {
    for i, bindingSubject := range bindingSubjects {
        if appliesToUser(user, bindingSubject, namespace) {
            return i, true
        }
    }
    return 0, false
}
//根据请求的用户去查看是否有该用户
func appliesToUser(user user.Info, subject rbacv1.Subject, namespace string) bool {
    switch subject.Kind {
    case rbacv1.UserKind:
        a :=strings.Compare(user.GetName(),subject.Name)
        return a == 0
    default:
        return false
    }
}

RBAC权限匹配

func ruleAllows(requestAttributes authorizer.Attributes, rule *rbacv1.PolicyRule) bool {
    //判断是否是资源请求
    if requestAttributes.IsResourceRequest() {
        //如果含有子资源的请求，则进行拼接
        combinedResource := requestAttributes.GetResource()
        if len(requestAttributes.GetSubresource()) > 0 {
            combinedResource = requestAttributes.GetResource() + "/" + requestAttributes.GetSubresource()
        }
        //进行权限、APIGroup、资源、资源名字进行判断用户请求的URL和该用户拥有的权限是否一致
        return VerbMatches(rule, requestAttributes.GetVerb()) &&
            APIGroupMatches(rule, requestAttributes.GetAPIGroup()) &&
            ResourceMatches(rule, combinedResource, requestAttributes.GetSubresource()) &&
            ResourceNameMatches(rule, requestAttributes.GetName())
    }
    return VerbMatches(rule, requestAttributes.GetVerb())
}

权限获取是通过RBAC方式。先调用Authorize

func (r *RBACAuthorizer) Authorize(requestAttributes authorizer.Attributes) (authorizer.Decision, string, error) {
    //调用visitRulesFor检查权限
    r.visitRulesFor(requestAttributes, ruleCheckingVisitor.visit)
    if ruleCheckingVisitor.allowed {
        return authorizer.DecisionAllow, ruleCheckingVisitor.reason, nil
    }
    //...
    //记录日志
    klog.Infof("...")
    //...
    return authorizer.DecisionNoOpinion, reason, nil
}

在调用vistRulesFor

func (r *RBACAuthorizer) visitRulesFor(requestAttributes authorizer.Attributes, visitor func(source fmt.Stringer, regoPolicy string, rule *rbacv1.PolicyRule, err error) bool) {
    //查看globalRole是否有对此资源的获取权限
    if globalRoleBindings, err := r.am.ListGlobalRoleBindings(""); err != nil {
        //...
    } else {
        sourceDescriber := &globalRoleBindingDescriber{}
        //循环globalRoleBindings。找到该用户的globalRoleBindings。
        for _, globalRoleBinding := range globalRoleBindings {
            subjectIndex, applies := appliesTo(requestAttributes.GetUser(), globalRoleBinding.Subjects, "")
            if !applies {
                //如果不是这个找到该用户的globalRoleBindings，则continue
                continue
            }
            //根据globalRoleBinding上的roleref，获取到role。进一步获取到regoPolicy和rules。这是两个规则，只要有一个规则符合即可
            regoPolicy, rules, err := r.am.GetRoleReferenceRules(globalRoleBinding.RoleRef, "")
            //...
             //根据regoPolicy验证是否符合规则
            if !visitor(sourceDescriber, regoPolicy, nil, nil) {
                return
            }
             //根据rules验证是否符合规则
            for i := range rules {
                if !visitor(sourceDescriber, "", &rules[i], nil) {
                    return
                }
            }
        }
        //...
    }
    
    if requestAttributes.GetResourceScope() == request.WorkspaceScope ||
        requestAttributes.GetResourceScope() == request.NamespaceScope ||
        requestAttributes.GetResourceScope() == request.DevOpsScope {
        //...
         // 代码部分删除，这部分主要是获取workspace
        workspace, err = r.am.GetNamespaceControlledWorkspace(requestAttributes.GetNamespace()); err != nil 
        // 通过workspace获取workspaceRoleBindings。
        if workspaceRoleBindings, err := r.am.ListWorkspaceRoleBindings("", workspace); ...{
             //...
        } else {
            //...轮训workspaceRoleBindings，找到workspaceRole，在进行权限检测。这部分和grobalrole一样
            for _, workspaceRoleBinding := range workspaceRoleBindings {
                subjectIndex, applies := appliesTo(requestAttributes.GetUser(), workspaceRoleBinding.Subjects, "")
                if !applies {
                    continue
                }
                regoPolicy, rules, err := r.am.GetRoleReferenceRules(workspaceRoleBinding.RoleRef, "")
                //...
                if !visitor(sourceDescriber, regoPolicy, nil, nil) {
                    return
                }
                for i := range rules {
                    if !visitor(sourceDescriber, "", &rules[i], nil) {
                        return
                    }
                }
            }
        }
    }

    if requestAttributes.GetResourceScope() == request.NamespaceScope ||
        requestAttributes.GetResourceScope() == request.DevOpsScope {
        
        namespace := requestAttributes.GetNamespace()
        // 直接获取namespace，或者根据DevOps获取namespace
        if requestAttributes.GetResourceScope() == request.DevOpsScope {
            if relatedNamespace, err := r.am.GetDevOpsRelatedNamespace(requestAttributes.GetDevOps()); err != nil {
                if !visitor(nil, "", nil, err) {
                    return
                }
            } else {
                namespace = relatedNamespace
            }
        }
        //根据namespace获取rolebinding
        if roleBindings, err := r.am.ListRoleBindings("", namespace); err != nil {
            if !visitor(nil, "", nil, err) {
                return
            }
        } else {
            sourceDescriber := &roleBindingDescriber{}
             //轮训roleBindings，找到role,检查role
            for _, roleBinding := range roleBindings {
                subjectIndex, applies := appliesTo(requestAttributes.GetUser(), roleBinding.Subjects, namespace)
                if !applies {
                    continue
                }
                regoPolicy, rules, err := r.am.GetRoleReferenceRules(roleBinding.RoleRef, namespace)
                if err != nil {
                    visitor(nil, "", nil, err)
                    continue
                }
                sourceDescriber.binding = roleBinding
                sourceDescriber.subject = &roleBinding.Subjects[subjectIndex]
                if !visitor(sourceDescriber, regoPolicy, nil, nil) {
                    return
                }
                for i := range rules {
                    if !visitor(sourceDescriber, "", &rules[i], nil) {
                        return
                    }
                }
            }
        }
    }
    //获取所有clusterRoleBindings，轮训查找到clusterRole。检查规则
    if clusterRoleBindings, err := r.am.ListClusterRoleBindings(""); err != nil {
        if !visitor(nil, "", nil, err) {
            return
        }
    } else {
        sourceDescriber := &clusterRoleBindingDescriber{}
        for _, clusterRoleBinding := range clusterRoleBindings {
            subjectIndex, applies := appliesTo(requestAttributes.GetUser(), clusterRoleBinding.Subjects, "")
            if !applies {
                continue
            }
            regoPolicy, rules, err := r.am.GetRoleReferenceRules(clusterRoleBinding.RoleRef, "")
            if err != nil {
                visitor(nil, "", nil, err)
                continue
            }
            sourceDescriber.binding = clusterRoleBinding
            sourceDescriber.subject = &clusterRoleBinding.Subjects[subjectIndex]
            if !visitor(sourceDescriber, regoPolicy, nil, nil) {
                return
            }
            for i := range rules {
                if !visitor(sourceDescriber, "", &rules[i], nil) {
                    return
                }
            }
        }
    }
}

检查规则的函数是visitor，如果regoPolicy符合就不检查rule

func (v *authorizingVisitor) visit(source fmt.Stringer, regoPolicy string, rule *rbacv1.PolicyRule, err error) bool {
    //调用open-policy-agent库检查权限
    if regoPolicy != "" && regoPolicyAllows(v.requestAttributes, regoPolicy) {
        v.allowed = true
        v.reason = fmt.Sprintf("RBAC: allowed by %s", source.String())
        return false
    }
    //调用k8s 接口实现rbac检查权限
    if rule != nil && ruleAllows(v.requestAttributes, rule) {
        v.allowed = true
        v.reason = fmt.Sprintf("RBAC: allowed by %s", source.String())
        return false
    }
    if err != nil {
        v.errors = append(v.errors, err)
    }
    return true
}

七、k8s集群资源转发

如果这个资源是向k8s请求的，则直接向k8s请求

func WithKubeAPIServer(handler http.Handler, config *rest.Config, failed proxy.ErrorResponder) http.Handler {
    //...
    //这部分为初始化内容，在请求时不执行。
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        info, ok := request.RequestInfoFrom(req.Context())
        if !ok {
            err := errors.New("Unable to retrieve request info from request")
            klog.Error(err)
            responsewriters.InternalError(w, req, err)
        }
        // 确认是否为k8s请求
        if info.IsKubernetesRequest {
            s := *req.URL
            s.Host = kubernetes.Host
            s.Scheme = kubernetes.Scheme

            // make sure we don't override kubernetes's authorization
            req.Header.Del("Authorization")
            //转发
            httpProxy := proxy.NewUpgradeAwareHandler(&s, defaultTransport, true, false, failed)
            httpProxy.UpgradeTransport = proxy.NewUpgradeRequestRoundTripper(defaultTransport, defaultTransport)
            httpProxy.ServeHTTP(w, req)
            return
        }

        handler.ServeHTTP(w, req)
    })
}

总结

希望与大佬们进行交流学习 。如有侵权部分请联系删除侵权内容