Bash
‘Kein Backup, kein Mitleid.’
A common German saying meaning “no backup, no pity”. Five weeks ago when another ransomware-wave became public and I saw the disaster with the Western Digital network-drives, I realized that despite using a NAS and regularly backing up data from different devices to this RAID level 1-device, I have no real “hard” backup. In my definition: a 1:1 mirror of the NAS-content which is NOT connected to any network at all and also stored physically in a different room or even different flat. Disconnected because of: are you sure everything is secured? Also no hidden bugs? Physically distant because of: what if the NAS catches fire and the backup is also affected?
So I did a quick review of the current state (NAS is a DS213 with latest updates; offers also USB 3.0-interface; has 2 4 TB drives insides, which are filled to 3.5 TB).
Decided to buy a 6 TB external 3.5″-harddrive (not SSD, because of possible loss of data while powered down over years) for 122 € (compared to the price of losing just a fraction of the photos – Forget it!).
Formatted it to EXT4 with some Linux. Attached then to the Synology DS213. And it was not detected, despite saying it supports EXT4. So we let the DS213 format it again (don’t ask me ..).
Let’s say it outright: I don’t want to use their proprietary backup-solutions. I also don’t want any kind of encryption.
SSH’ed into the DS213 (activated before, because off by default). Then I checked which partitions have to be copied. Puzzled together a chained rsync-command (sorted by priorities) and let it run. Detaching the session via ‘nohup’ Wasn’t working.
|
1 2 3 4 5 6 7 |
rsync -avrh /volume1/Photoshare_privat/ /volumeUSB1/usbshare/Photoshare_privat/ && \ rsync -avrh /volume1/homes/Marcel/ /volumeUSB1/usbshare/homes/Marcel/ && \ rsync -avrh /volume1/homes/ruzica/ /volumeUSB1/usbshare/homes/ruzica/ && \ rsync -avrh /volume1/homes/admin/ /volumeUSB1/usbshare/homes/admin/ && \ rsync -avrh /volume1/Camera/ /volumeUSB1/usbshare/Camera/ && \ rsync -avrh /volume1/photo/ /volumeUSB1/usbshare/photo/ && \ rsync -avrh /volume1/Musik/ /volumeUSB1/usbshare/Musik/ |
Turns out quite quickly that the one core-cpu of the DS213 is the bottleneck, because it runs at 100% and therefore mere writing speeds of 8-9 MB/s are achieved despite the HDD being capable of writing up to 120 MB/s. My back of the napkin-estimation is 4-5 days for all data. Next backups should run faster, because incremental. And most of the data is written once, changed almost never.
Quick summary: I was baffled that despite having some experience in IT did not have a _real_ backup before. And out of discussions with peers I know almost none of those working in IT have either.
The proposed solution will save my family and me from any 100% losses.
edit 20210801: ideas need some time to mature. Now one of the Raspberry Pi 3B takes care of the rsync-calls to relieve the real PC (host RocketChat anyway, therefore is online 24/7) and rsync runs now without compression; the resulting writing speed is now in the range of 11-15 MiByte/s.
SSH/SCP quick help
Challenge was to reconfigure a device, which has a Linux running on a TQ-board. Some files had to be adjusted.
Connect to the device:
|
1 |
ssh <usernam>:<ip> |
(Then enter password.)
List amount of free space:
|
1 |
df -K . |
Show target of symlink:
|
1 |
readlink <file> |
how to resume broken downloads
aka: fix the weird firewalling settings ..
aka: How to download the latest stable Qt SDK source code?
curl -L -O -C - http://download.qt.io/official_releases/qt/5.12/5.12.3/single/qt-everywhere-src-5.12.3.zip
cmd replacement: cmder
Sometimes you have to call some batch-files with spoecific windows-tools, so a bash is not helpful on Win.
A colleague recommended cmdr, which is quite nice and fine.
Suits my needs 🙂
advanced whitespace-correction
(for CMake/C++-projects)
Find all fitting files and run the fixer-script in parallel over it.
|
1 |
( find . -name "CMakeLists.txt" ; find . -name "*.cmake" ; find . -name "*.h" ; find . -name "*.cpp" ) | parallel --bar ../removeTrailing.sh |
After playing for a while with sed and awk and not being able to get a fitting solution, I decided to create my own as python-script. It squashes all consecutive double-whitespace-lines (and adds one to the end if missing).
Or “run it on the files changed for the last commit”:
|
1 |
$ git diff --name-only HEAD~1 HEAD | parallel --bar ../removeTrailing.sh |
Sources:
removeTrainling.sh
squashMultipleWhitespace.py
How to get GNU parallel (developed by Ole Tange):
|
1 |
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash |
Pick a random reviewer
Most teams have a certain code-review-process. But most of the time there is no “round robin”-implementation who will be the reviewer, but the coder itself should select the reviewer. Because of domain-knowledge, etc. And then you her lots of times “I am currently busy. Not me!”
Solution: random and fair selection via bash.
|
1 2 3 4 5 6 7 8 |
#!/bin/bash # initialize the array array=(MSA MBH HGA GSC MPE NLE RNI MLA) size=${#array[@]} index=$(($RANDOM % $size)) echo "Mister ${array[$index]} is your candidate as reviewer. Congratulations!" echo "In case you picked yourself - just rerun ;))" |
Current version of the script can be found at: https://github.com/marcelpetrick/bashCollection/blob/master/randomReviewer.sh
Inspired by a stack-overflow answer.
find the fattest files of certain type
|
1 |
$ find . -name "*.LSimg" -exec ls -al {} \; | sort -nr -k5 >> listOfFattestFiles.txt |
creates something like
|
1 2 3 4 |
-rw-r--r-- 1 mpe 1049089 323707001 Mai 29 2018 ./a.LSimg -rw-r--r-- 1 mpe 1049089 21342433 Mai 29 2018 ./b/c.LSimg -rw-r--r-- 1 mpe 1049089 21342432 Mai 29 2018 ./e.LSimg .. |
Fix whitespace
Sometimes there is too much whitespace and tabs in my last commit.
In former times I used a VisualStudio-plugin, but this is not helpful there. So I wrote a small script (see on my github) which:
• replaces all tabs with four spaces
• removes all trailing whitespace
• converts line-endings to CRLF
I invoke it on the list of all changes files in my last commit in the git-repo with:
|
1 |
$ git diff --name-only HEAD~1 HEAD | xargs -L1 ../removeTrailing.sh |
(first part gives you a list of changed files and then feeds it to the script for execution)
Overview of author-activities for the git-repo
If some ‘sophisiticated’ tool like git-stats is too cumbersome to configure, just do some git-log and shell-magic:
|
1 2 3 4 5 6 7 |
$ git log --format='%aN' | sort | uniq -c | sort 4 Hans Wurst 6 Klaus Blume 6 Bernd Meier 26 Marcel Petrick 32 Ali Baba 37 Linus Torvalds |
Ok, just noticed that git has something built-in:
|
1 |
$ git shortlog -sn |
from: https://coderwall.com/p/pek-yg/git-statistics-for-repo-per-author – Thanks Marcin Olichwirowicz!
keep the catcam up to date
Logging in to the raspberry(ies) to update them on a regular basis takes time and effort. Both are currently dear.
Also: while the catcam is taking pictures, you can’t update the rpi-binaries.
So I made this nice script, which first suspends the catcam-operation while renaming the script from the cronjob, then does all updates, reverts the renaming and then reboots the MCU.
|
1 2 3 4 5 6 7 8 9 10 11 |
#!/usr/bin/env bash # author: Marcel Petrick (mail@marcelpetrick.it) mv /home/pi/Desktop/webcam/regulargif.sh /home/pi/Desktop/webcam/regulargif.sh1 sudo rpi-update sudo apt-fast update && sudo apt-fast dist-upgrade -y sudo apt-get autoclean sudo apt-get autoremove mv /home/pi/Desktop/webcam/regulargif.sh1 /home/pi/Desktop/webcam/regulargif.sh sudo reboot |
Make it executeable via ‘sudo chmod +X updateCatcam.sh’ and also add it as cronjob (once a day).