Education 2019

Written by marcelpetrick on June 27, 2019

Time to reveal the plans of the supervised education for 2019: today my course from the “Volkshochschule” (adult evening school?) for “machine learning with Python” starts (you remember the course for advanced Python I did in May 2018 and which I backep up by doing exercises for Project Euler and “365 days coding challenge”? link ). I am really, really looking forward and am eager to learn 🙂

After this is finished in June my course for basic introduction to Mandarin will start (not a ‘computer’ language, but an interesting one) in addition to my daily Duolingo-exercises for this upcoming lingua franca (from my POV). Will be a challenge, especially the spoken version with its intonations.

Last but not least: on week ago I participated in a course as first responder for emergencies (at work).

And I still plan to do a professional certified course for software architecture (the big picture) in autumn.

edit: repo with a apart of the notes from the machine learning with Python-course are at Github.

advanced whitespace-correction

Written by marcelpetrick on February 26, 2019

(for CMake/C++-projects)

Find all fitting files and run the fixer-script in parallel over it.

 ( find . -name "CMakeLists.txt" ;   find . -name "*.cmake" ;   find . -name "*.h" ;   find . -name "*.cpp" ) | parallel --bar ../removeTrailing.sh

1	( find . -name "CMakeLists.txt" ; find . -name ".cmake" ; find . -name ".h" ; find . -name "*.cpp" ) \| parallel --bar ../removeTrailing.sh

After playing for a while with sed and awk and not being able to get a fitting solution, I decided to create my own as python-script. It squashes all consecutive double-whitespace-lines (and adds one to the end if missing).

Or “run it on the files changed for the last commit”:

$  git diff --name-only HEAD~1 HEAD | parallel --bar ../removeTrailing.sh

1	$ git diff --name-only HEAD~1 HEAD \| parallel --bar ../removeTrailing.sh

Sources:
removeTrainling.sh
squashMultipleWhitespace.py

How to get GNU parallel (developed by Ole Tange):

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

1	(wget -O - pi.dk/3 \|\| curl pi.dk/3/ \|\| fetch -o - http://pi.dk/3) \| bash

Retrospective view at 2018

Written by marcelpetrick on February 1, 2019

The first month of 2019 already passed. And we passed it with flying colors!
But let’s have a look at 2018 – a year full of challenges and success: I’ve worked full-time, organized and participated in advanced courses for Python and in Requirements Engineering (officially: IREB Requirements Engineering Foundation Level-approved) and pursued a new employment as software engineer.

And I wrote some software in my spare-time, as you can see in the graph for the public github-repositories. The gaps in the commits can be explained with the birth of my daughter and the time where I acquired the new job and moved nearly 900 km across the country. Yay! Nice personal projects were and are Cullendula and the Daily Coding Challenges, which I solve mostly with fully Unit-tested Python (3).

More new, hands-on knowledge was gained in the area of CMake and Qt-charts.
Well – 2018 was great. Let me make 2019 greater! 💪

“Tell me and I forget, teach me and I may remember, involve me and I learn.”

Written by marcelpetrick on June 22, 2018

I can use the quote from Benjamin Franklin or the straight-forward one from my professor “You learn programming by programming.”. So, for gaining further knowledge about Python and how to use it, I subscribed to the mailing-list @ dailycodingproblem.com and try to solve every day a new problem. Most of them seem easily doable, because – at least – I know algorithms and have developed software for some ~years~ decades. But then hypocrisy hits reality and I realize that my Python-skills are virtually non-existant 🙂 So I do a lot of look-ups at stackoverflow, the internet and the manuals for python-standard-libraries. And this is fine, totally fine. Because this way I learn (it the hard way).
Results are committed at: https://github.com/marcelpetrick/pythonCollection/tree/master/dailyCodingTask

And – which I am really happy about – I tend to write proper unit-tests BEFORE I start implementing the real functionality!
dailycodingproblem.com always adds some example-data and the expected result. This is already the first test.

MPC: adding additional DEFINES

Written by marcelpetrick on May 25, 2018

Some weeks ago I noticed how the qDebug()-output could be enriched, so that in bigger solutions with a lot of different “unknown” components a reported error could be immediately pinned. And you save writing always __FILE__ and __LINE__. Referres to this post.
But the problem was that with the mpc-buildsystem it was unknown to me how to force it to put this DEFINE into the vcxproj-files.

It can be done via the “macros”-statement!

So I worked on my Python-skills and wrote a short script which iterates the given path recursively and fixes all mpc-files by checking for the position of the line with the last closingbrace “}” and then it adds before that position the line. Of course, the experts know several thousand ways to improve that script – but I am currently happy with it. It works, it is debug-able (.sh, I look at you!) and I will use the skeleton also for some other tasks.
It can be found (like most Python-snippets) at: https://github.com/marcelpetrick/pythonCollection

import fileinput
import sys
import os

#------------------------------------------
def addMacroToFile(filename, newStatement):
    #print("called addMacroToFile:", filename, determinePositionOfLastBrace(filename))

    returnValue = "fail"
    posLastBrace = determinePositionOfLastBrace(filename)
    posCurrentLine = 1
    for line in fileinput.input(filename, inplace=True):
        # add additional line in case of matching last brace
        if posCurrentLine == posLastBrace:
            sys.stdout.write(newStatement)
            sys.stdout.write("\n") # newline, else it would be put just before the brace
            returnValue = "success"

        sys.stdout.write(line) # prints without additional newline (on windows)
        posCurrentLine += 1

    return returnValue

#------------------------------------------
# return the position of the last brace.
# possible improvement: in case of some malformed mpc: skip ..
def determinePositionOfLastBrace(filename):
    lastPos = -1
    position = 1
    with open(filename, "r") as file: #has implicit close - which is nice
        for line in file:
            if line.__contains__("}"):
                lastPos = position
            position += 1

    return lastPos

# ------------------------------------------
# find all mpb/mpc files
def fixAllFilesRecursively(path, suffix, newStatement):
    for dirname, dirnames, filenames in os.walk(path):
        for filename in filenames:
            full_path = os.path.join(dirname, filename)
            if full_path.endswith(suffix):
                print(full_path)
                wasSuccessful = addMacroToFile(full_path, newStatement)
                if wasSuccessful == "success":
                    print("  modified :)")

        # prevent recursion into those subdirs with meta-data
        if '.svn' in dirnames:
            dirnames.remove('.svn')
        if '.git' in dirnames:
            dirnames.remove('.git')

#------------------------------------------
#------------------------------------------

path = "D:\Repo_INS_RADARNX"
suffix = "mpc"
newStatement = "	macros += QT_MESSAGELOGCONTEXT"
# TODO possible improvement: prevent double application of the added macro ..
fixAllFilesRecursively(path, suffix, newStatement)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

import fileinput

import sys

import os

#------------------------------------------

def addMacroToFile(filename, newStatement):

#print("called addMacroToFile:", filename, determinePositionOfLastBrace(filename))

returnValue = "fail"

posLastBrace = determinePositionOfLastBrace(filename)

posCurrentLine = 1

for line in fileinput.input(filename, inplace=True):

# add additional line in case of matching last brace

if posCurrentLine == posLastBrace:

sys.stdout.write(newStatement)

sys.stdout.write("\n") # newline, else it would be put just before the brace

returnValue = "success"

sys.stdout.write(line) # prints without additional newline (on windows)

posCurrentLine += 1

return returnValue

#------------------------------------------

# return the position of the last brace.

# possible improvement: in case of some malformed mpc: skip ..

def determinePositionOfLastBrace(filename):

lastPos = -1

position = 1

with open(filename, "r") as file: #has implicit close - which is nice

for line in file:

if line.__contains__("}"):

lastPos = position

position += 1

return lastPos

# ------------------------------------------

# find all mpb/mpc files

def fixAllFilesRecursively(path, suffix, newStatement):

for dirname, dirnames, filenames in os.walk(path):

for filename in filenames:

full_path = os.path.join(dirname, filename)

if full_path.endswith(suffix):

print(full_path)

wasSuccessful = addMacroToFile(full_path, newStatement)

if wasSuccessful == "success":

print(" modified :)")

# prevent recursion into those subdirs with meta-data

if '.svn' in dirnames:

dirnames.remove('.svn')

if '.git' in dirnames:

dirnames.remove('.git')

#------------------------------------------

path = "D:\Repo_INS_RADARNX"

suffix = "mpc"

newStatement = " macros += QT_MESSAGELOGCONTEXT"

# TODO possible improvement: prevent double application of the added macro ..

fixAllFilesRecursively(path, suffix, newStatement)

Python-advanced level-seminar: lifelong learning

Written by marcelpetrick on May 9, 2018

I am currently traveling back from my very first paid educational leave. Proper selection, arrangement and preparation lead to some awesome impressions: about the capabilities of Python and about the city of Detmold.
Daniel Warner lead us – an assembly of five inquisitive men in the age-range from 30 to 60 – along the details and
specialties of that programming language. I learned much, in detail:

basic structures; list comprehension
classes; objects; overrides; imports; representation; init-method
dictionaries for caching results (memoisation)
decorators (nice for for printing, caching and thread-safety)
descriptors, properties and slots, kwargs
(multi-)inheritance and its quirks
recursive functions; functional programming
threads, synchronisation, atomic access

I put all the exercises (full script with my own annotations) into a Git-repository right from the beginning and published it: https://github.com/marcelpetrick/Python_FortgeschrittenenSeminar/.
Which also makes a nice view of the github-history 🙂

Python was chosen by me by intent: I see and plan for ways to use it with artifical intelligence (TensorFlow-binding ..); microcontroller-programming (ESP can run MicroPython) and for the Raspberry (currently the tumblr-upload-script for the catcam is also Python); for daily data-manipulation-tasks which are currently done more or less on Bash or AutoIt or Batch – and then: write it once, run it both on Linux and Win).
This was a great choice! And I want to thank my wife for supporting these stays absent from home and my plan to achieve the wanted education 🙂 And I got a small certificate – but that’s just icing on the cake.

My plan as first real exercise is to re-implement the “find all islands in the given map”-programming challenge. This will be fun. Getting to know some specialties and what properties/slots mean in Python-context (compared to the Qt-ones) was nice. And the decorators are a really powerful way to add special functionality to methods without bloating them and without blocking the view to the busines logic.

tracking bugs in complex systems ..

Written by marcelpetrick on March 6, 2018

Some weeks ago I’ve replaced the case of the ‘catcam’. A Raspberry Pi 3 which was running for 1.5 years without interruption with the nice script & python-upload triggered by a cronjob. And now the device freezes. Still running, but neither accessible via SSH nor VNC (of course, if SSH is not working ..).
I left the case open; removed the camera; upgraded kernel and Raspbian: nothing helped.
My last guess is that the camera stopped working. But to be sure I want to let the RPi3 long-time with temperature-logging. So … we need a simple program:

import os
import time

def measure_temp():
        temp = os.popen("vcgencmd measure_temp").readline()
        return (temp.replace("temp=",""))

while True:
        print(time.ctime() + " ## " + measure_temp())
	time.sleep(1)

1

2

3

4

5

6

7

8

9

10

import os

import time

def measure_temp():

temp = os.popen("vcgencmd measure_temp").readline()

return (temp.replace("temp=",""))

while True:

print(time.ctime() + " ## " + measure_temp())

time.sleep(1)

executed like this, because else the output is buffered up to 4 or 8 kByte:

$ stdbuf -oL python temperature.py > worstLoggingEver.txt

1	$ stdbuf -oL python temperature.py > worstLoggingEver.txt

and the content of the textfile is something like:

Tue Mar  6 22:52:59 2018 ## 51.5'C
Tue Mar  6 22:53:00 2018 ## 52.1'C

1 2	Tue Mar 6 22:52:59 2018 ## 51.5'C Tue Mar 6 22:53:00 2018 ## 52.1'C

CodeCombat: learning Python the cute way

Written by marcelpetrick on June 23, 2015

I am currently trying to enhance my skills by learning Python. I made a choice between Ruby and Python after I was quite frustrated sometimes while writing the contactSheet-script. Bash is missing some general debug-abilities (or I don’t know them ;)) and the way to control the code-/data-flow was quite limited from my POV. I came from high level-strong typed-precomiled C++ and then hit bash ..

CodeCombat is a browser-only, completely OpenSource “game” where you lead your protagonist through its adventures by using code-snippets. One after another simple and more complicated statements are introduced, control structures like loops and alternatives follow. You can chose between different languages – besides Python also JavaScript is available.
It is quite amusing to guide the hero(ine). I finished already 22 levels.

For in detailed explainations I stick to: “Learn Python the hard way“