Random tech thoughts

I better write some things down before there're forgotten

Private and Public Apis on Different Ports With Spring

I recently had to create a (micro-) service, offering APIs to a user frontend and to other internal services. The frontend apis were supposed to require authentication by the user/browser and the internal apis were to work without authentication. But because the internal apis represented sensitive functionality, they should only by available to internal services.

When creating a HTTP/REST microservice with Spring, the service usually binds to one tcp port. So every client which is able to connect to this port is also able call all the endpoints of the services. This poses a problem, when one needs to offer private and publics apis on the same service.

There are several options how to control access to different endpoints. One could create different services and bind them to different hosts or ports and have a firewall control the access. Or one might have a reverse proxy in front of the service, and regulate access with it.

One easy way is to have the same service offer different apis on different ports. It requires some more advanced configuration of spring, but has the advantage of keeping all functionality in one service. The only external requirement for limitting access to the private apis is some kind of firewalling to restrict access to the private tcp port.

Here is how it’s done with Spring.

Different path-prefixes for internal apis

It all starts with two controller methods, one offering an internal endpoint, the other one offering an external endpoint.

1
2
3
4
5
6
7

@Controller
public class ExternalApiController {
    @GetMapping("/external/hello")
    public ResponseEntity<String> hello() {
        return ResponseEntity.ok("Hello stranger");
    }
}

1
2
3
4
5
6
7

@Controller
public class InternalApiController {
    @GetMapping("/internal/hello")
    public ResponseEntity<String> hello() {
        return ResponseEntity.ok("Hello friend");
    }
}

Note, that one controller serves under a path starting with /external, the other under /internal. This will be used later to systematically distinguish requests.

By default, these two endpoints are available at the same port and therefor it’s difficult to restrict access to the internal endpoint.

Listen on multiple ports

To change this, we first have to make spring listen on a second port. The internal tomcat server allows to listen on additional ports through a WebServerFactoryCustomizer, which can be provided as a bean.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

@Configuration
public class TrustedPortConfiguration {

    @Value("${server.port:8080}")
    private String serverPort;


    @Value("${management.port:${server.port:8080}}")
    private String managementPort;


    @Value("${server.trustedPort:null}")
    private String trustedPort;

    @Bean
    public WebServerFactoryCustomizer servletContainer() {

        Connector[] additionalConnectors = this.additionalConnector();

        ServerProperties serverProperties = new ServerProperties();
        return new TomcatMultiConnectorServletWebServerFactoryCustomizer(serverProperties, additionalConnectors);
    }


    private Connector[] additionalConnector() {

        if (StringUtils.isEmpty(this.trustedPort) || "null".equals(trustedPort)) {
            return null;
        }

        Set<String> defaultPorts = new HashSet<>();
        defaultPorts.add(serverPort);
        defaultPorts.add(managementPort);

        if (!defaultPorts.contains(trustedPort)) {
            Connector connector = new Connector("org.apache.coyote.http11.Http11NioProtocol");
            connector.setScheme("http");
            connector.setPort(Integer.valueOf(trustedPort));
            return new Connector[]{connector};
        } else {
            return new Connector[]{};
        }
    }

    private class TomcatMultiConnectorServletWebServerFactoryCustomizer extends TomcatServletWebServerFactoryCustomizer {
        private final Connector[] additionalConnectors;

        TomcatMultiConnectorServletWebServerFactoryCustomizer(ServerProperties serverProperties, Connector[] additionalConnectors) {
            super(serverProperties);
            this.additionalConnectors = additionalConnectors;
        }

        @Override
        public void customize(TomcatServletWebServerFactory factory) {
            super.customize(factory);

            if (additionalConnectors != null && additionalConnectors.length > 0) {
                factory.addAdditionalTomcatConnectors(additionalConnectors);
            }
        }
    }
}

With this configuration, the internal tomcat listens on two ports, and all endpoints are available on them. Half way there. All that’s left to do is restricting endpoints to a certain port.

Filter requests based on path and port

With spring, all incoming requests can be filtered, before they reach a controller. Filtering means, requests can be modified and answered, and the filter can decide, if the request either should be passed on for further processing, or if processing ends with it.

To write a filter, one must implement Filter and provide public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain). A Response can be generated and the request can be analyzed. If the processing of the request should continue, filterChain.doFilter must be called. Otherwise, the filter just returns.

Here’s one way how to implement a filter in order to filter requests to a internal api based on internal/external ports.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

public class TrustedEndpointsFilter implements Filter {

    private int trustedPortNum = 0;
    private String trustedPathPrefix;
    private final Logger log = LoggerFactory.getLogger(getClass().getName());

    TrustedEndpointsFilter(String trustedPort, String trustedPathPrefix) {
        if (trustedPort != null && trustedPathPrefix != null && !"null".equals(trustedPathPrefix)) {
            trustedPortNum = Integer.valueOf(trustedPort);
            this.trustedPathPrefix = trustedPathPrefix;
        }
    }

    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
        if (trustedPortNum != 0) {

            if (isRequestForTrustedEndpoint(servletRequest) && servletRequest.getLocalPort() != trustedPortNum) {
                log.warn("denying request for trusted endpoint on untrusted port");
                ((ResponseFacade) servletResponse).setStatus(404);
                servletResponse.getOutputStream().close();
                return;
            }

            if (!isRequestForTrustedEndpoint(servletRequest) && servletRequest.getLocalPort() == trustedPortNum) {
                log.warn("denying request for untrusted endpoint on trusted port");
                ((ResponseFacade) servletResponse).setStatus(404);
                servletResponse.getOutputStream().close();
                return;
            }
        }

        filterChain.doFilter(servletRequest, servletResponse);
    }

    private boolean isRequestForTrustedEndpoint(ServletRequest servletRequest) {
        return ((RequestFacade) servletRequest).getRequestURI().startsWith(trustedPathPrefix);
    }
}

The filter decides based on a configurable prefix of the request path, if this request is for an internal or an external endpoint and checks, if it arrived on the right port. If so, the request is passed on. If not, it is rejected with a status 404.

To activate the filter, it must be provided as a bean.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

@Configuration
public class WebConfig implements WebMvcConfigurer {

    @Value("${server.trustedPort:null}")
    private String trustedPort;

    @Value("${server.trustedPathPrefix:null}")
    private String trustedPathPrefix;

    @Bean
    public FilterRegistrationBean<TrustedEndpointsFilter> trustedEndpointsFilter() {
        return new FilterRegistrationBean<>(new TrustedEndpointsFilter(trustedPort, trustedPathPrefix));
    }
}

Configuration

Finally, some configuration has to be provided, to define ports and the paths.

1
2
3

server.port=8002
server.trustedPort=8003
server.trustedPathPrefix=/internal/

All requests to a path starting with server.trustedPathPrefix are served on server.trustedPort and all other requests are served on server.port.

Conclusion

Spring is incredible flexible and allows quite sophisticated configurations. It is easily possible to configure it to serve different endpoints on different tcp ports. With that one can serve internal apis and external apis using the same spring server instance and control access to them using firewalling.

You can find the complete example at my Github.

Kubernetes Persistent Volumes With CIFS and Loop Devices

Here’s an idea for a poor man’s NAS for Kubernetes PVs: All you need is a fileserver running samba and some bash scripting.

But let’s begin at how Kubernetes handles storage. The key philosophy here is, that storage is something different than computation, so storage get’s it’s own abstraction (i.e. Kubernetes resource). While computation is basically handled by pods, persistent storage is offered by Persistent Volumes (PV), which can be mounted inside a pod’s container as a volume. In order to mount a PV in a pod, the pod has to reference a Persistent Volume Claim (PVC).

The Persistent Volume Claim defines the requirements for the storage, e.g. capacity, and acts like a bridge between pod and PV, bringing them together. After a PVC is deployed it is pending and the controller-manager tries to match it to a PV. When that succeeds, the PVC is bound to the PV and available to be mounted in a pod as a volume.

graph LR; pvc[Persistent Volume Claim] -- controller manager binds --> pv[Persistent Volume] pod[Pod] -- mounts --> pvc

PVCs have a lifecycle separate from pods, so a pod can be created, deleted and recreated, while the PVC and with it the persistent storage (the stored data) stays alive and bound. When the PVC gets deleted, the stored data becomes unavailble for good and the PV can be reclaimed.

pod.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
    volumeMounts:
            - name: my-pv
              mountPath: /volume
  restartPolicy: Always

  volumes:
  - name: my-pv
    persistentVolumeClaim:
      claimName: my-pv-claim

pvc.yaml

1
2
3
4
5
6
7
8
9
10

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi

Because PV is only an abstract concept of storage, there has to be something, that implements how files are persisted. This is the storage provisioner. It’s job is to mount some kind of storage on the so it’s available for the pod. There are several provisioners shipped within Kubernetes, most of them for proprietary cloud providers or datacenter-scale storage systems. Luckily there is also FlexVolume, which allows you to implement your own provisioner easily.

pv.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0003
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  flexVolume:
    driver: "fnordian/cv"
    readOnly: false
    options:
      source: "//192.168.121.225/kubvolumes"
      mountOptions: "dir_mode=0700,file_mode=0600"
      cifsuser: "nobody"
      cifspass: "nobody"

Using FlexVolume

So FlexVolume does not implement any storage mounting itself, but it gives you the opportunity to do it yourself in form of a driver. The driver’s API is not too complicated and often implementing 3 actions (init, mount, unmount) is enough.

The details of a driver are explained here. The gist of creating a most basic driver is: It needs to have a name (vendor-drivername) and must reside as an executable in a certain path on every node. It defaults to:

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/vendor~drivername/drivername

The driver is then called with command line arguments, specifying the action and is expected to respond with json written to stdout. Here is an example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/fnordian~cv/cv init

# response: {"status": "Success", "capabilities": {"attach": false}}

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/fnordian~cv/cv mount \
    /var/lib/kubelet/pods/ce4b5af0-16de-11e8-bcfd-5254002cac04/volumes/fnordian~cv/pv0003 \
    {"cifspass":"nobody","cifsuser":"nobody","kubernetes.io/fsType":"","kubernetes.io/pod.name":"busybox",\
    "kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"ce4b5af0-16de-11e8-bcfd-5254002cac04",\
    "kubernetes.io/pvOrVolumeName":"pv0003","kubernetes.io/readwrite":"rw","kubernetes.io/serviceAccount.name":"default",\
    "mountOptions":"dir_mode=0700,file_mode=0600","source":"//192.168.121.159/kubvolumes"}

# response: {'status': 'Success'}

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/fnordian~cv/cv unmount /var/lib/kubelet/pods/a80184b0-16df-11e8-bcfd-5254002cac04/volumes/fnordian~cv/pv0003

# response: {'status': 'Success'}

As you can see, the driver’s mount action gets passed a target directory and parameters specifying the volume as JSON. After it returned with success, it is expected to have mounted a volume to the target directory. Unmounting follows the same principle but only needs the target directory as the argument.

Mount storage on CIFS

Now that we have the kubernetes specifics out of the way, all that’s left is setting up a cifs-share and writing the driver. The idea for the driver is, that the actual volume is a filesystem inside a plain file, which is made available to the nodes via cifs and mounted for the pod using loop.

I agree, that it is a bit complex, but the advantage of a filesystem inside a loop-device over e.g. a cifs-share mounted directly to the pod is, that it can store files for multiple users (with different uids) and also selinux context labels.

You can find the running driver on github. It expects name, username and password of a cifs-share as arguments, so be sure to have specified them in the PV. Please note the lack of error- and corner-case-handling.

Setting up the cifs share with samba is also straight forward. I’ve included a sample smb.conf in the repository. Be sure to also set a password using smbpasswd.

Setting Kubernetes Resource Quotas With Python

In an RBAC based kubernetes system, users' access to the cluster can be limitted using namespaces, roles and rules. These limits consists of resource-types and methods/verbs a user can apply on those. E.g. a user may create, list and delete pods.

While these limits already enable a quite effective isolation in a way, that one user may not modify the resources of another user (or the system), it is often necessary to constrain usage even more.

So here is another tool to further control cluster usage: resource quotas. Resource quotas let you limit the following resources on a per user-basis.

  • pods
  • services
  • replicationcontrollers
  • resourcequotas
  • secrets
  • configmaps
  • persistentvolumeclaims
  • services.nodeports
  • services.loadbalancers

You can find more documentation here

Let’s see how to create and apply them with python:

1
2
3
4
5
6
7
8
9
10
11

import kubernetes
v1 = kubernetes.client.CoreV1Api()

resource_quota = kubernetes.client.V1ResourceQuota(
        spec=kubernetes.client.V1ResourceQuotaSpec(
            hard={"requests.cpu": "1", "requests.memory": "512M", "limits.cpu": "2", "limits.memory": "512M",
                "requests.storage": "1Gi", "services.nodeports": "0"}))
resource_quota.metadata = kubernetes.client.V1ObjectMeta(namespace="user-namespace",
        name="user-quota")
v1.create_namespaced_resource_quota("user-namespace", resource_quota)

As with roles, resource quotas are applied to namespaces. So to set limits for a user, the quotas have to be configured with the user’s namespace.

Limitted Authorization for a Kubernetes User With Python

While last post’s was about authenticating a user to kubernetes, this one handles authorization.

Different levels of authorization in kubernetes can be achieved through namespaces, roles and rolebindings. In order to limit a user to his/her own space in the cluster, we create a personal namespace and bind a role to it. The role describes the types of access, the user will have. This kind of authorization is called RBAC, role-based access control.

A namespace in kubernetes is defined just by a name. So creating it is easy:

1
2
3

from kubernetes import client
v1 = client.CoreV1Api()
v1.create_namespace("user-namespace")

Roles are a bit more complex. They specify the access rights for the user. These rights are defined as a list of rules, which are additive and declare the resources and the type of access the role will give.

Besides roles, there are also cluster roles. The difference between those is, that a role is specific to a namespace, while a cluster role is not.

The example below gives a user enough rights to create, manage and delete deployments. Note that as a role is specific to a namespace and that a user-namespace is specific to a user, there will be a role for every user.

1
2
3
4
5
6
7
8
9
10
11
12

import kubernetes
rules = [
        kubernetes.client.V1PolicyRule([""], resources=["pods"], verbs=["get", "list", "create", "delete", "update"], ),
        kubernetes.client.V1PolicyRule(["extensions"], resources=["deployments", "replicasets"],
                                       verbs=["get", "list", "create", "delete", "update"], )
    ]
role = kubernetes.client.V1Role(rules=rules)
role.metadata = kubernetes.client.V1ObjectMeta(namespace="user-namespace",
                                               name="user-role")

rbac = kubernetes.client.RbacAuthorizationV1Api()
rbac.create_namespaced_role("user-namespace", role)

The created role must be bound to a user, otherwise it is not effective. Binding happens through role bindings.

1
2
3
4
5
6
7
8
9
10
11
12

import kubernetes

role_binding = kubernetes.client.V1RoleBinding(
        metadata=kubernetes.client.V1ObjectMeta(namespace="user-namespace",
                                                name="user-role-binding"),
        subjects=[kubernetes.client.V1Subject(name="user", kind="User", api_group="rbac.authorization.k8s.io")],
        role_ref=kubernetes.client.V1RoleRef(kind="Role", api_group="rbac.authorization.k8s.io",
                                             name="user-role"))

rbac = kubernetes.client.RbacAuthorizationV1Api()
rbac.create_namespaced_role_binding(namespace="user-namespace",
                                        body=role_binding)

And that’s it. Now, the user ‘user’ has access to it’s own namespace ‘user-namespace’ and may manage deployments.

There’s one more thing. What if you want to make sure, a user does not deplete all the cluster’s resources? With the current configuration, he or she is able to spawn new pods until all the nodes are full. It turns out, kubernetes has a way of handling this. It’s called resource quota, and my next post will be about it.

Creating Client Certificates for Kubernetes Apiserver With Python

Kubernetes features a complex, multi-user permissions system to control access to the cluster. Those options are bound to contexts and users can be given access to those contexts. For that, a user must authenticate itself to the api-server and there are various ways for doing that.

The authentication options are documented here

One of those options is authentication through x509 client certificates.

A user wanting to authenticate needs to have a certificate with his/her username as the subject’s commonName. And the certificate needs to be signed by the cluster’s ca. Although the concept is quite easy, creating certificates using the openssl-cli often proves to be difficult. So here is a python script that creates/takes a ca and creates new client certificates for a kubernetes user.

Be sure apiserver runs with --client-ca-file=yourca.crt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149

#!/usr/bin/env python

from socket import gethostname

import yaml
import random
from OpenSSL import crypto
import base64

CA_CERT_FILE = "ca.crt"
CA_KEY_FILE = "ca.key"
CLIENT_CERT_FILE = "client.crt"
CLIENT_KEY_FILE = "client.key"


def generate_self_signed_ca():
    k = crypto.PKey()
    k.generate_key(crypto.TYPE_RSA, 1024)

    # create a self-signed ca
    ca = crypto.X509()
    ca.get_subject().C = "DE"
    ca.get_subject().ST = "Duesseldorf"
    ca.get_subject().L = "Duesseldorf"
    ca.get_subject().O = "Dummy GmbH"
    ca.get_subject().OU = "Dummy GmbH"
    ca.get_subject().CN = gethostname()
    ca.set_serial_number(1000)
    ca.gmtime_adj_notBefore(0)
    ca.gmtime_adj_notAfter(10 * 365 * 24 * 60 * 60)
    ca.set_issuer(ca.get_subject())
    ca.set_pubkey(k)

    ca.add_extensions([
        crypto.X509Extension(b"basicConstraints", True,
                             b"CA:TRUE, pathlen:0"),
        crypto.X509Extension(b"keyUsage", True,
                             b"keyCertSign, cRLSign"),
        crypto.X509Extension(b"subjectKeyIdentifier", False, b"hash",
                             subject=ca),
    ])
    ca.add_extensions([
        crypto.X509Extension(b"authorityKeyIdentifier", False, b"keyid:always", issuer=ca)
    ])

    ca.sign(k, 'sha1')

    open(CA_CERT_FILE, "wb").write(
        crypto.dump_certificate(crypto.FILETYPE_PEM, ca))
    open(CA_KEY_FILE, "wb").write(
        crypto.dump_privatekey(crypto.FILETYPE_PEM, k))

    return ca, k


def load_cert():
    with open(CA_CERT_FILE, "rb") as certfile:
        catext = certfile.read()

    with open(CA_KEY_FILE, "rb") as keyfile:
        keytext = keyfile.read()

    return (
        crypto.load_certificate(crypto.FILETYPE_PEM, catext),
        crypto.load_privatekey(crypto.FILETYPE_PEM, keytext, None)
    )


def generate_client_cert(ca_cert, ca_key, username):
    client_key = crypto.PKey()
    client_key.generate_key(crypto.TYPE_RSA, 2048)

    client_cert = crypto.X509()
    client_cert.set_version(2)
    client_cert.set_serial_number(random.randint(50000000, 100000000))

    client_subj = client_cert.get_subject()
    client_subj.commonName = username
    # client_subj.organizationName = "user-group"

    client_cert.add_extensions([
        crypto.X509Extension(b"basicConstraints", False, b"CA:FALSE"),
        crypto.X509Extension(b"subjectKeyIdentifier", False, b"hash", subject=client_cert),
    ])

    client_cert.add_extensions([
        crypto.X509Extension(b"authorityKeyIdentifier", False, b"keyid:always", issuer=ca_cert),
        crypto.X509Extension(b"extendedKeyUsage", False, b"clientAuth"),
        crypto.X509Extension(b"keyUsage", False, b"digitalSignature"),
    ])

    client_cert.gmtime_adj_notBefore(0)
    client_cert.gmtime_adj_notAfter(10 * 365 * 24 * 60 * 60)

    client_cert.set_subject(client_subj)

    client_cert.set_issuer(ca_cert.get_issuer())
    client_cert.set_pubkey(client_key)
    client_cert.sign(ca_key, 'sha256')

    with open(CLIENT_CERT_FILE, "wb") as f:
        f.write(crypto.dump_certificate(crypto.FILETYPE_PEM, client_cert))

    with open(CLIENT_KEY_FILE, "wb") as f:
        f.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, client_key))

    return client_cert, client_key


def new_client_cert(username):
    ca_cert, ca_key = None, None

    try:
        ca_cert, ca_key = load_cert()
    except Exception as e:
        print(e)
        ca_cert, ca_key = generate_self_signed_ca()

    client_cert, client_key = generate_client_cert(ca_cert, ca_key, username)

    return client_cert, client_key, username


def create_user_config(client_cert, client_key, username):
    with open("admin.conf") as adminconfigtext:
        config = yaml.load(adminconfigtext)

    config["users"][0]["name"] = username
    config["users"][0]["user"]["client-certificate-data"] = base64.b64encode(
        crypto.dump_certificate(crypto.FILETYPE_PEM, client_cert)).decode()
    config["users"][0]["user"]["client-key-data"] = base64.b64encode(
        crypto.dump_privatekey(crypto.FILETYPE_PEM, client_key)).decode()

    config["contexts"][0]["context"]["user"] = username
    config["contexts"][0]["name"] = username + "@kubernetes"
    config["current-context"] = username + "@kubernetes"

    print(config["users"][0]["user"]["client-certificate-data"])

    with open("user.conf", "w") as userconfigtext:
        yaml.dump(config, userconfigtext)


if __name__ == "__main__":
    client_cert, client_key, username = new_client_cert("myusername")
    try:
        create_user_config(client_cert, client_key, username)
    except FileNotFoundError:
        print("admin.conf needed to create user.conf")

If you have a admin.conf file, then the script also generates a kubectl configfile for the user.

1

kubectl --kubeconfig user.conf

Easy peasy. Hope it’ll be useful to someone.