Json

Python TinyDB

Storing Data in JSON - TinyDB

Small Example how to Store Data in JSON, and Query them afterwards like a NOSQL DB. Have a look at TinyDB if you wanna see more.

Code

from tinydb import TinyDB, Query
from pprint import pprint

# Create or load a database file
db = TinyDB('db.json')

# insert some sample data
def insert():
    # Insert data
    db.insert({'name': 'John', 'age': 30})
    db.insert({'name': 'Alice', 'age': 25, 'hobbies': 'sleep'})
    db.insert({'name': 'Max', 'age': 20, 'hobbies': ['sleep', 'play', 'eat']})

# show all entries
def show_all():
    all_records = db.all()
    pprint(all_records)

# entries with hobbies
def show_entries_with_hobbies():
    User = Query()
    result = db.search(User.hobbies.exists())
    pprint(result)

# entries without hobbies
def show_entries_without_hobbies():
    User = Query()
    result = db.search(~User.hobbies.exists())
    pprint(result)

# show entries with hobbies and older than 22 years
def show_entries_with_hobbies_and_older_than_22():
    User = Query()
    result =  db.search((User.hobbies.exists()) & (User.age > 22))
    pprint(result)

if __name__ == "__main__":
  # Add
  insert()

  # show
  print("\n-- ALL --")
  show_all()

  print("\n-- with Hobbies --")
  show_entries_with_hobbies()

  print("\n-- without Hobbies --")
  show_entries_without_hobbies()


  print("\n-- with Hobbies and older than 22 --")
  show_entries_with_hobbies_and_older_than_22()

Run

you need to install tinydb. use a virtual env like .venv, poetry or whatever you like

JC - JSON from CLI

how to build json from cli

we all like json, do we ? https://kellyjonbrazil.github.io/jc/docs/parsers/ping

add package

doas pkg_add jc

try ping

openbsd-box # ping -c 3 1.1.1.1 |jc --ping -p 2>/dev/null
{
  "destination_ip": "1.1.1.1",
  "data_bytes": 56,
  "pattern": null,
  "destination": "1.1.1.1",
  "packets_transmitted": 3,
  "packets_received": 3,
  "packet_loss_percent": 0.0,
  "duplicates": 0,
  "round_trip_ms_min": 9.219,
  "round_trip_ms_avg": 9.826,
  "round_trip_ms_max": 10.158,
  "round_trip_ms_stddev": 0.43,
  "responses": [
    {
      "type": "reply",
      "bytes": 64,
      "response_ip": "1.1.1.1",
      "icmp_seq": 0,
      "ttl": 59,
      "time_ms": 10.158,
      "duplicate": false
    },
    {
      "type": "reply",
      "bytes": 64,
      "response_ip": "1.1.1.1",
      "icmp_seq": 1,
      "ttl": 59,
      "time_ms": 9.219,
      "duplicate": false
    },
    {
      "type": "reply",
      "bytes": 64,
      "response_ip": "1.1.1.1",
      "icmp_seq": 2,
      "ttl": 59,
      "time_ms": 10.101,
      "duplicate": false
    }
  ]
}

Compatible platforms: linux, darwin, freebsd -> had to redirect the stderr to /dev/null because OpenBSD is not (yet) supported officially…

Jq

Json Query

some basics about JQ

RAW Data

cat history.shelly.0.SHEM-3#40F52000B661#1.Total.Current.json | jq '.[0:3]'
[
  {
    "val": 2.64,
    "ack": 1,
    "ts": 1607900404883,
    "q": 0,
    "user": "system.user.admin"
  },
  {
    "val": 2.61,
    "ack": 1,
    "ts": 1607900410483,
    "q": 0,
    "user": "system.user.admin"
  },
  {
    "val": 2.58,
    "ack": 1,
    "ts": 1607900416083,
    "q": 0,
    "user": "system.user.admin"
  }
]

Query First Record

cat history.shelly.0.SHEM-3#40F52000B661#1.Total.Current.json | jq '.[0]'
{
  "val": 2.64,
  "ack": 1,
  "ts": 1607900404883,
  "q": 0,
  "user": "system.user.admin"
}

Filter val and ts

cat history.shelly.0.SHEM-3#40F52000B661#1.Total.Current.json | jq '.[0] |.ts,.val'
1607900404883
2.64

Filter val and ts on one line

cat history.shelly.0.SHEM-3#40F52000B661#1.Total.Current.json | jq '.[0] | (.ts |tostring) + ";" + (.val |tostring)'
"1607900404883;2.64"

Filter first 10 val and ts on one line

cat history.shelly.0.SHEM-3#40F52000B661#1.Total.Current.json | jq '.[] | (.ts |tostring) + ";" + (.val |tostring)' |head -10
"1607900404883;2.64"
"1607900410483;2.61"
"1607900416083;2.58"
"1607900421739;2.62"
"1607900427335;2.62"
"1607900433003;2.57"
"1607900438543;2.72"
"1607900444131;2.67"
"1607900449791;2.6"
"1607900455383;2.55"

Filter first 10 val and ts on one line

cat history.shelly.0.SHEM-3#40F52000B661#1.Total.Current.json | jq '.[] |.ts,.val' |paste - - |head -10
1607900404883	2.64
1607900410483	2.61
1607900416083	2.58
1607900421739	2.62
1607900427335	2.62
1607900433003	2.57
1607900438543	2.72
1607900444131	2.67
1607900449791	2.6
1607900455383	2.55

Current over 50A

cat history.shelly.0.SHEM-3#40F52000B661#1.Total.Current.json | jq -c '.[] | select (.val >= '50')'
{"val":52.55,"ack":1,"ts":1607907152399,"q":0,"user":"system.user.admin"}
{"val":52.54,"ack":1,"ts":1607907157975,"q":0,"user":"system.user.admin"}
{"val":52.53,"ack":1,"ts":1607907163639,"q":0,"user":"system.user.admin"}
{"val":52.5,"ack":1,"ts":1607907169220,"q":0,"user":"system.user.admin"}
{"val":52.49,"ack":1,"ts":1607907174863,"q":0,"user":"system.user.admin"}
{"val":52.53,"ack":1,"ts":1607907180639,"q":0,"user":"system.user.admin"}
...

BGP Stuff

dump networks from AS 3303