Kubernetes Multi-Juicer

2022-07-08 (Last Modified: 2022-08-10)

Cloud

Page content

If you ever wanna run a Multiplayer OWASP Juice Shop CTF on your own, here are some Notes and Info for bloody beginners

References

Prerequisite

you’ve got

a Digital Ocean Account (or some other Cloud Provider)
a spare domain and set the NS of DigitalOcean
Digital Ocean CMD Line Tools installed and configured
helm tools (kubernetes package manager -> brew install helm)
some budget (~2 CHF/Day)
30min for Setup

btw. all this commands should run on macOS. linux may needs some adjustments …

Set Environment

set a few variables as we need them later in the scripts

test $domain || domain='your.domain'
host='ctf'
fqdn="${host}.${domain}"

echo "*** $(date)***"   >>  info.md
echo "FQDN: ${fqdn}"    >>  info.md
echo "domain=${domain}" >   .env
echo "host=${host}"     >>  .env
echo "fqdn=${fqdn}"     >>  .env

Check Connection

are we connected and authorized to DO ?

doctl account get
doctl auth list

$ doctl account get
User Email        Team       Droplet Limit    Email Verified    User UUID                                   Status
[email protected]  YourTeam   25               true              xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx    active

$ doctl auth list
default (current)

Connect to Digital Ocean if needed

if your not connected to DO, generate a Token on the API Menu and add it like this:

doctl auth init --context TOKENID

Create Cluster / 1 CPU, 2GB RAM

this needs a few minutes. just be patient :)

#time doctl kubernetes cluster create juicy-k8s --region fra1 --node-pool "auto-scale=true;min-nodes=3;max-nodes=5"

Create Cluster / 2 CPU, 2GB RAM

needed to update to a stronger box due to k8s issues!

time doctl kubernetes cluster create juicy-k8s --region fra1 --node-pool "size=s-2vcpu-2gb;auto-scale=true;min-nodes=3;max-nodes=5"

$ time doctl kubernetes cluster create juicy-k8s
Notice: Cluster is provisioning, waiting for cluster to be running
.....................................................
Notice: Cluster created, fetching credentials
Notice: Adding cluster credentials to kubeconfig file found in "/Users/stoege/.kube/config"
Notice: Setting current-context to do-nyc1-juicy-k8s
ID                                      Name         Region    Version         Auto Upgrade    Status     Node Pools
c3564b4f-501d-450b-xxxx-xxxxxxxxxxxx    juicy-k8s    nyc1      1.22.11-do.0    false           running    juicy-k8s-default-pool

real	4m57.624s
user	0m0.129s
sys	0m0.063s

List Clusters

doctl kubernetes cluster list

$ doctl kubernetes cluster list
ID                                      Name         Region    Version         Auto Upgrade    Status     Node Pools
5bd4401c-7a82-4240-xxxx-xxxxxxxxxxxx    juicy-k8s    fra1      1.22.11-do.0    false           running    juicy-k8s-pool-1

Get the Current Context

kubectl config current-context

$ kubectl config current-context
do-fra1-juicy-k8s

Install MultiJuicer via Helm

helm repo add multi-juicer https://iteratec.github.io/multi-juicer/
time helm install multi-juicer multi-juicer/multi-juicer

helm install multi-juicer multi-juicer/multi-juicer
NAME: multi-juicer
LAST DEPLOYED: Fri Jul  8 23:36:45 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
MultiJuicer deployed! 🎉🥳

To administrate the cluster you can log into the JuiceBalancer with the admin account:
Username: admin
Password: ${kubectl get secrets juice-balancer-secret -o=jsonpath='{.data.adminPassword}' | base64 --decode}

Extract Admin Password

adminpw=$(kubectl get secrets juice-balancer-secret -o=jsonpath='{.data.adminPassword}' | base64 --decode)
echo "Admin Password: ${adminpw}" |tee -a info.md
unset adminpw

Show Nodes and Pods

this may needs a few moments …

while true; do echo; kubectl get nodes; kubectl get pods; echo; sleep 5; done

kubectl get nodes
NAME                           STATUS   ROLES    AGE     VERSION
juicy-k8s-default-pool-cicp5   Ready    <none>   2m59s   v1.22.11
juicy-k8s-default-pool-cictf   Ready    <none>   15m     v1.22.11
juicy-k8s-default-pool-cictx   Ready    <none>   14m     v1.22.11
juicy-k8s-default-pool-cicty   Ready    <none>   15m     v1.22.11

kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
juice-balancer-846449bd74-g4ktk      1/1     Running   0          5m27s
progress-watchdog-6586786884-w76vh   1/1     Running   0          5m27s

wait until all nodes and pods are status “Ready/Running”

Add Loadbalancer and Expose to Inet

List Certificates

doctl compute certificate list

ID    Name    DNS Names    SHA-1 Fingerprint    Expiration Date    Created At    Type    State

Create Certificate

doctl compute certificate create --type lets_encrypt --name ctf --dns-names ${fqdn}

$ doctl compute certificate create --type lets_encrypt --name ctf --dns-names ctf.8192.ch
ID                                      Name    DNS Names      SHA-1 Fingerprint    Expiration Date         Created At              Type            State
a18460c3-554a-4bd7-xxxx-xxxxxxxxxxxx    ctf     ctf.your.domain                     0001-01-01T00:00:00Z    2022-07-08T21:47:00Z    lets_encrypt    pending

and wait a Moment until State switches to verified

doctl compute certificate list

build loadbalancer config

extract certificate id and put in config

certid=$(doctl compute certificate list |awk -v pat="$fqdn" '$0 ~ pat{print $1}')

cat << EOF > do-lb.yaml
kind: Service
apiVersion: v1
metadata:
  name: multi-juicer-loadbalancer
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-protocol: 'http2'
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: '${certid}'
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: 'true'
    service.beta.kubernetes.io/do-loadbalancer-algorithm: 'round_robin'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol: 'http'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: '/balancer/'
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/instance: multi-juicer
    app.kubernetes.io/name: multi-juicer
  ports:
    - name: http
      protocol: TCP
      port: 443
      targetPort: 3000
EOF

kubectl create -f do-lb.yaml

Wait for LBL IP

this needs 2-3min

while true; do kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}'; echo "."; sleep 5; done

Get LBL IP

lbip=$(kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}')
echo "Loadbalancer IP: ${lbip}" |tee -a info.md

-> Loadbalancer IP: 178.128.xxx.yyy

Update DNS

set A record ctf.your.domain to 178.128.xxx.yyy

doctl compute domain records create ${domain} --record-type A --record-data ${lbip} --record-ttl 3600 --record-name ${host}
doctl compute domain records list ${domain}

Check DNS

doctl compute domain records list ${domain} |grep ${host}

dig +short @9.9.9.9 ${fqdn}
dig +short @ns1.digitalocean.com ${fqdn}

host ${fqdn}

Check Page

echo "https://${fqdn}"
echo "https://${lbip}"

you should get the Welcome Page !

Now, the Teams can register themself …

… and start the Challenge !

Usefull Links

https://ctf.your.domain/balancer/score-board/ -> list of all Teams !
https://ctf.your.domain/#/score-board -> hidden Score Board Page with lot of Tips and Tricks

CleanUp

as the Machines costs Money (-> up to 60 CHF/Month), you wanna delete and remove all the stuff after the Challange)

# Delete Multi Juicer
helm delete multi-juicer

# Delete the loadbalancer
kubectl delete -f do-lb.yaml

# Delete the kubernetes cluster
doctl kubernetes cluster delete juicy-k8s -f

# Delete Certificate
certid=$(doctl compute certificate list |grep ${fqdn} |awk '{print $1}')
doctl compute certificate delete ${certid} -f

# Delete A records
arecordid=$(doctl compute domain records list ${domain} |grep ${host} |awk '{print $1}')
doctl compute domain records delete ${domain} ${arecordid} -f

# Droplets killed as well ?
doctl compute droplet list

# in Digital Ocean Page:
$-> delete the Hosts if still running (should not be ...)

Kubernets Commands

kubectl config get-contexts
kubectl cluster-info
kubectl version
kubectl get nodes
kubectl help

doctl kubernetes cluster kubeconfig
show <cluster-id|cluster-name>
doctl kubernetes

Setup Script / Part 1

copy / paste following Script. Should run under MACOS without Problems ;)

ETA: around 10min

export project="juicy-k8s"
export domain="mydomain.com"
export host="shop"

cat << 'EOS' > install_k8s.sh
#!/usr/bin/env bash

# MultiJuicer Setup Script

test -d game || mkdir game
cd game

if [[ -f .env ]]; then

  source .env

else

  test $project || domain='PROJECT-NOT-SET'
  test $domain  || domain='DOMAIN-NOT-SET'
  test $host    || host='HOST-NOT-SET'
  fqdn="${host}.${domain}"
  echo $project, $domain, $host, $fqdn

  export project="$project"
  export domain="$domain"
  export host="$host"
  export fqdn="$fqdn"

  echo "project=${project}"   >   .env
  echo "domain=${domain}"     >>  .env
  echo "host=${host}"         >>  .env
  echo "fqdn=${fqdn}"         >>  .env

fi

echo
echo "*** $(date)***"       |tee info.md
echo "Project: ${project}"  |tee -a info.md
echo "FQDN: ${fqdn}"        |tee -a info.md
echo

read -p "Press enter to continue (or CTRL-C to break)"

doctl kubernetes cluster \
  create ${project} \
  --region fra1 \
  --node-pool "size=s-2vcpu-2gb;auto-scale=true;min-nodes=3;max-nodes=5"
echo "sleep 15"; sleep 15

kubectl config current-context
helm repo add multi-juicer https://iteratec.github.io/multi-juicer/
helm install multi-juicer multi-juicer/multi-juicer
echo "sleep 15"; sleep 15

adminpw=$(kubectl get secrets juice-balancer-secret -o=jsonpath='{.data.adminPassword}' | base64 --decode)
echo "Admin Password: ${adminpw}" |tee -a info.md
unset adminpw

kubectl get nodes
kubectl get pods

doctl compute certificate list

doctl compute certificate create --type lets_encrypt --name ${host} --dns-names ${fqdn},${host}1.${domain},${host}-1.${domain},${host}2.${domain},${host}-2.${domain},${host}3.${domain},${host}-3.${domain}
echo "sleep 15"; sleep 15

certid=$(doctl compute certificate list |awk -v pat="$fqdn" '$0 ~ pat{print $1}')

cat << EOF > do-lb.yaml
kind: Service
apiVersion: v1
metadata:
  name: multi-juicer-loadbalancer
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-protocol: 'http2'
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: '${certid}'
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: 'true'
    service.beta.kubernetes.io/do-loadbalancer-algorithm: 'round_robin'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol: 'http'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: '/balancer/'
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/instance: multi-juicer
    app.kubernetes.io/name: multi-juicer
  ports:
    - name: http
      protocol: TCP
      port: 443
      targetPort: 3000
EOF

kubectl create -f do-lb.yaml
EOS

chmod 700 install_k8s.sh

Wait for Loadbalancer

let this commands running until you got an ip address … takes up to 3min

while true; do \
  lbip=$(kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}'); \
  echo $lbip; \
  echo .; \
  sleep 5; \
done

Setup Script / Part 2

get IP for Loadbalancer, set DNS Record, done

cat << 'EOS' > install_dns.sh
# Env
test -f game/.env && source game/.env || exit 1

lbip=$(kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}')
echo "Loadbalancer IP: ${lbip}" |tee -a game/info.md

# A Record
doctl compute domain records create ${domain} \
  --record-type A \
  --record-data ${lbip} \
  --record-name ${host}

# More A Records
for i in {1..3}; do
doctl compute domain records create ${domain} \
  --record-type A \
    --record-ttl 3600 \
  --record-data ${lbip} \
  --record-name ${host}${i}
done

# some CNAME
for i in {1..3}; do
  doctl compute domain records create ${domain} \
    --record-type CNAME \
    --record-ttl 3600 \
    --record-name ${host}-${i} \
    --record-data ${fqdn}.
done

doctl compute domain records list ${domain}

echo; echo "Done! Check Page:"
echo "https://${fqdn}"
echo "https://${lbip}"
echo "$(cat game/info.md |grep -i password)"; echo
EOS

chmod 700 install_dns.sh

Setup Script / Uninstall

cat << 'EOF' > uninstall.sh
# Env
test -f game/.env && source game/.env || exit 1

# Delete Multi Juicer
helm delete multi-juicer

# Delete the loadbalancer
kubectl delete -f game/do-lb.yaml

# Delete the kubernetes cluster
doctl kubernetes cluster delete juicy-k8s -f

# Delete Certificate
certid=$(doctl compute certificate list |grep ${fqdn} |awk '{print $1}')
doctl compute certificate delete ${certid} -f

# Delete A records
arecordid=$(doctl compute domain records list ${domain} |grep ${host} |awk '{print $1}')
doctl compute domain records delete ${domain} ${arecordid} -f

# Cleanup
rm -rf game

# Wait until gone
echo "press CTLR-C when all nodes are gone ..."
while true; do doctl compute droplet list; sleep 5; done
EOF

chmod 700 uninstall.sh

Project Management

Create Project

doctl projects create --name "Test123" --purpose "Learning"

List Projects

$ doctl projects list -o json
[
  {
    "id": "aa444047-7eee-4548-94db-5485b705b905",
    "owner_uuid": "8a0d816ec1a780bfb985147b10157563d1c0ff4a",
    "owner_id": 1883170,
    "name": "Test123",
    "description": "",
    "purpose": "Other: Learning",
    "environment": "",
    "is_default": false,
    "created_at": "2022-08-10T12:04:03Z",
    "updated_at": "2022-08-10T12:04:03Z"
    "updated_at": "2022-07-06T20:14:51Z"
  }

List all Records

set Variable for all following Snippets

mydomain="bliblablu.com"

List all Records

doctl compute domain records list ${mydomain}

List all ID’s for A Records

doctl compute domain records list ${mydomain} --format Type,ID |\
  awk '/^A /{ print $2 }'

Delete all “A” Records

-f will enforce without asking !

doctl compute domain records delete ${mydomain} -f $(\
  doctl compute domain records list ${mydomain} --format Type,ID |\
  awk '/^A /{ print $2 }' \
  )

List all Records except NS/SOA

doctl compute domain records list ${mydomain} --no-header --format Type,ID |\
  awk '!/^(NS|SOA) /{ print $2 }'

Delete all Records except NS/SOA

-f will enforce without asking !

doctl compute domain records delete ${mydomain} -f $(\
  doctl compute domain records list ${mydomain} --no-header --format Type,ID |\
  awk '!/^(NS|SOA) /{ print $2 }' \
  )

doctl compute domain records list ${mydomain} --no-header --format Type,ID |\
  awk '!/^(NS|SOA) /{ print $2 }' |\
  doctl compute domain records delete ${mydomain} -f $(cat -)

Any Comments ?

sha256: d8c59a70d9b6c07e289a9395b2f38cfa0b32cc27abf323fc4f32c407283d4a3c