Linux

Bigdata

How to Process Large Files … ?

Large is a variable Term, 700 GB is large for me, while it could be a small peace for others.

Assuming you need to count the lines … this simple Task can take minutes !

Size

[user@host /tmp]$ du -sh bigfile
745G bigfile

Wordcount -> 10 min

if you need to count the lines, use the wordcount command and you get the exact number … but you have to wait for minutes, depending in your disk subsystem and the file size of course

Crontab

Troubleshooting Crontab Problems

Dump the Environment

add this line to root’s crontab

* * * * * env > /root/cronenv

Load the Environment

env - $(cat /root/cronenv) /bin/sh

Run your script

./script_with_env_problems.sh

and check it’s output …


Any Comments ?

sha256: de1667fb4003135420244f48280d72e03d7559b675ab6b4a9c020b0725866525

ScionLab

Install Doku

Upgrade Fresh Debian

apt-get -y update
apt-get -y upgrade
apt-get -y distupgrade
apt-get -y install vim

Scion

apt-get install apt-transport-https
echo "deb [trusted=yes] https://packages.netsec.inf.ethz.ch/debian all main" | tee /etc/apt/sources.list.d/scionlab.list
apt-get update
apt-get install scionlab

Config

scionlab-config --host-id=1fcf8axxxxxxxxxxxxxxxxxxxxxxxxxx --host-secret=4ef93cxxxxxxxxxxxxxxxxxxxxxxxxxx

Check Services

root@scionlab:~# ip address show dev tun0
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.1.1.52/16 brd 10.1.255.255 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::aa8a:c135:f627:bcff/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

root@scionlab:~# grep Interfaces -A15 /etc/scion/gen/ISD*/AS*/endhost/topology.json
      "Interfaces": {
        "1": {
          "Bandwidth": 1000,
          "ISD_AS": "17-ffaa:0:1107",
          "LinkTo": "PARENT",
          "MTU": 1472,
          "Overlay": "UDP/IPv4",
          "PublicOverlay": {
            "Addr": "10.1.1.52",
            "OverlayPort": 50000
          },
          "RemoteOverlay": {
            "Addr": "10.1.0.1",
            "OverlayPort": 50229
          }
        }

root@scionlab:~# systemctl list-dependencies scionlab.target
scionlab.target
● ├─scion-border-router@17-ffaa_1_d85-1.service
● ├─scion-control-service@17-ffaa_1_d85-1.service
● ├─scion-daemon@17-ffaa_1_d85.service
● └─scion-dispatcher.service

root@scionlab:~# scmp echo -remote 20-ffaa:0:1404,[0.0.0.0]
Using path:
  Hops: [17-ffaa:1:d85 1>230 17-ffaa:0:1107 1>4 17-ffaa:0:1102 3>4 17-ffaa:0:1108 2>8 17-ffaa:0:1101 3>2 20-ffaa:0:1401 6>1 20-ffaa:0:1402 2>1 20-ffaa:0:1404] MTU: 1472, NextHop: 127.0.0.1:30042

176 bytes from 20-ffaa:0:1404,[0.0.0.0] scmp_seq=0 time=258.282ms
176 bytes from 20-ffaa:0:1404,[0.0.0.0] scmp_seq=1 time=259.515ms
176 bytes from 20-ffaa:0:1404,[0.0.0.0] scmp_seq=2 time=264.142ms
176 bytes from 20-ffaa:0:1404,[0.0.0.0] scmp_seq=3 time=257.675ms

root@scionlab:~# scmp tr -remote 21-ffaa:0:1501,[0.0.0.0]
Using path:
  Hops: [17-ffaa:1:d85 1>230 17-ffaa:0:1107 1>4 17-ffaa:0:1102 3>4 17-ffaa:0:1108 2>8 17-ffaa:0:1101 3>2 20-ffaa:0:1401 4>3 21-ffaa:0:1501] MTU: 1472, NextHop: 127.0.0.1:30042
0  17-ffaa:1:d85,[127.0.0.1] IfID=1  474µs 460µs 303µs
1  17-ffaa:0:1107,[192.33.93.195] IfID=230  9.533ms 10.099ms 9.75ms
2  17-ffaa:0:1107,[192.33.93.195] IfID=1  10.152ms 10.12ms 9.651ms
3  17-ffaa:0:1102,[129.132.121.164] IfID=4  11.151ms 11.874ms 9.675ms
4  17-ffaa:0:1102,[192.33.92.68] IfID=3  10.419ms 11.829ms 11.886ms
5  17-ffaa:0:1108,[195.176.0.11] IfID=4  14.57ms 11.859ms 11.739ms
6  17-ffaa:0:1108,[195.176.28.157] IfID=2  16.276ms 15.704ms 15.417ms
7  17-ffaa:0:1101,[193.247.172.154] IfID=8  15.862ms 16.496ms 16.473ms
8  17-ffaa:0:1101,[193.247.172.130] IfID=3  22.882ms 23.789ms 22.081ms
9  20-ffaa:0:1401,[134.75.250.114] IfID=2  252.212ms 252.403ms 253.067ms
10  20-ffaa:0:1401,[134.75.250.114] IfID=4  253.937ms 251.94ms 251.649ms
11  21-ffaa:0:1501,[202.255.44.48] IfID=3  314.05ms 313.705ms 313.292ms
12  21-ffaa:0:1501,[0.0.0.0]  314.419ms 313.534ms 314.605ms

Any Comments ?

sha256: 0719a8bbc818ec97a796c90b7de0027ee6905fd5d889d6eb6b500b519b968dcc

Mint Vlan

How to configure a Vlan on Linux Mint ?

Wiki

Install vlan

sudo apt-get install vlan

Configuration

root@mint:~# sudo modprobe 8021q

root@mint:~# ifconfig
ens19: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.18.83  netmask 255.255.255.0  broadcast 10.10.18.255
        inet6 2001:db8:1:2:9506:5fcd:1c61:1279  prefixlen 64  scopeid 0x0<global>
        inet6 2001:db8:1:2:a9:c50b:1348:1ec6  prefixlen 64  scopeid 0x0<global>
        inet6 2001:db8:1:2:e064:b0ec:a08f:7fd8  prefixlen 64  scopeid 0x0<global>
        inet6 2001:db8:1:2:a015:7ec:eef0:1a75  prefixlen 64  scopeid 0x0<global>
        inet6 2001:db8:1:2:a028:b890:ffe8:5231  prefixlen 64  scopeid 0x0<global>
        inet6 2001:db8:1:2:c80:addb:1273:95d5  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::9cc2:b22d:cc15:2946  prefixlen 64  scopeid 0x20<link>
        ether 4e:e5:10:00:06:83  txqueuelen 1000  (Ethernet)
        RX packets 181  bytes 58337 (58.3 KB)
        RX errors 0  dropped 7  overruns 0  frame 0
        TX packets 271  bytes 48413 (48.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 138  bytes 13723 (13.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 138  bytes 13723 (13.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@mint:~# vconfig add ens19 158
Added VLAN with VID == 158 to IF -:ens18:-
root@mint:~#

root@mint:~# ip addr add 100.60.100.83/24 dev ens19.158
root@mint:~#

root@mint:~# ip link set up ens19.158
root@mint:~#

Make Permanent

sudo su -c 'echo "8021q" >> /etc/modules'

/etc/network/interfaces
auto ens19.158
iface ens19.158 inet static
    address 100.60.100.83
    netmask 255.255.255.0
    vlan-raw-device ens19.158
    up route add -net 10.0.0.0 netmask 255.0.0.0 gw 100.60.100.1

Any Comments ?

sha256: e69d61613b45307405014943049154d70c8cf140d2a3bcf5bbf7679fbefa798f

Doas

doas, an alternative to sudo

Everybody knows sudo … right ? but the openbsd guys hacked a small and secure replacement called doas …

simple, secure and clever

here a good and quick tutorial

An introduction on Vultr, the Source Code on Github and the Man Page

Installation OpenBSD

On OpenBSD, it’s already in the Base System and no need to install anything.

Installation Linux

On Linux, for Example, you have to add the Package

Keepalive

Little Keep Alive

… mit freundlicher genehmigung von Kumpel Marc :)

keepalive.sh

#!/usr/bin/env bash

FILE="$HOME/scripts/excuses"

# Linux or BSD ? nf points to the right binary
which numfmt > /dev/null 2>&1 && nf=$(which numfmt) || nf=$(which gnumfmt);

# Linux or BSD ? gs points to the right binary
which shuf > /dev/null 2>&1 && gs=$(which shuf) || gs=$(which gshuf);

if [ ! -e "$FILE" ]; then
    echo ""
    echo "$FILE does not exist"
    echo "##############################################"
    command -v curl >/dev/null 2>&1 || { echo >&2 "Holy cow! You don't even have curl, get lost!"; echo ""; exit 1; }
    mkdir -p $HOME/scripts/
    curl -o $FILE https://pages.cs.wisc.edu/~ballard/bofh/excuses
    echo "##############################################"
fi

keep () {
 clear;
 echo "If you are reading this..." &&  echo  "Congratulations, you are alive."
 echo ""
 echo "You are stuck on the following planet:" `hostname`
 echo "It has been" `date +%s | $nf --g` "seconds since January 1st 1970, this is quite a while..."
 echo ""; echo "Your current excuse is:"; $gs -n 1 $HOME/scripts/excuses
}

while :
do
 keep
 sleep 30
done

Testrun

./keepalive.sh
If you are reading this...
Congratulations, you are alive.

You are stuck on the following planet: puffy201.planet
It has been 1658870361 seconds since January 1st 1970, this is quite a while...

Your current excuse is:
Power Company having EMP problems with their reactor

Any Comments ?

sha256: 5de0afbcacd56dbcaede593a6f243a3f46e4d94754683646d57e00f3c7840eca

Proxmox

Running a Promox Cluster with CEPH.

Here a few Commands to figure out the Status of CEPH

ceph status
ceph osd status
pveceph lspools
ceph pg dump

Manual Backup

vzdump 777 --dumpdir /mnt/backup --mode snapshot

Manual Restore

qmrestore /mnt/backup/vzdump-qemu-777.vma 777

Proxmox Performance Overview

Intel Nuc I5, 32GB RAM, 500GB SSD

root@nuc:~# pveperf
CPU BOGOMIPS:      36799.44
REGEX/SECOND:      3927398
HD SIZE:           93.99 GB (/dev/mapper/pve-root)
BUFFERED READS:    522.34 MB/sec
AVERAGE SEEK TIME: 0.11 ms
FSYNCS/SECOND:     1588.49
DNS EXT:           49.40 ms
DNS INT:           0.65 ms (planet)

Remove Cluster Config

Source: Proxmox Forum

how to move on the cli


Any Comments ?

sha256: 92b20e0a803b2e3c9a987fe89c259ac9bd069b22732d93f80d0626fae15e733b