Binary patching is a crucial tool in software analysis and reverse engineering. This article provides an overview of the binary patching process using the Radare2 utility. The article begins by introducing the concept of binary patching.

Key concepts covered in the article include:

• Understanding binary analysis fundamentals

• Navigating and seeking specific memory addresses within binaries

• Writing assembly instructions to modify binary behaviour

Throughout the article, step-by-step examples and commands are provided to guide readers through the process of binary patching using Radare2. By the end of the article, readers will gain a fundamental understanding of how to navigate Radare2 tool and how to perform basic binary patching.

Introduction

Binary patching is a process of modifying compiled binary executable files to change their behaviour or fix issues without access to the source code.

This technique is commonly used in software development, reverse engineering, and cybersecurity for various purposes such as bug fixing, feature enhancement, and vulnerability exploitation or mitigation.

In our examples, we will use a simple C application, compile it, run it and later on change its binary using the Radar2 utility to modify its behaviour.

Note: Assembly language and instruction codes are specific to CPU architecture. In the examples provided, we will be focusing on the Intel x86 architecture.

Binary Analysis

Binary analysis is a process of examining and understanding compiled binary code (machine code) without access to the original source code. This analysis includes disassembling, decompiling, and analyzing the binary behaviour. In our case, we will know exactly what the program intends to do, since we're going to be the ones who write it. So the binary analysis will be straightforward.

Wallet application

We're going to implement a simple wallet sample app in C where the wallet is unlocked only if the user enters the correct numeric password (1234 in our case).

// file: wallet.c
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
  int num = atoi(argv[1]);
  if (num == 1234) {
    printf("Unlocked!\n");
    return 0;
  }
  printf("Wrong passcode! This incident will be reported!\n");
  return 1;
}

Le's compile it:

$ clang wallet.c -o ./wallet

Lets see what it does.

If we give it any number other than 1234 we get an error message and error status code:

$ ./wallet 1
Wrong! This incident will be reported!
$ echo $?
1

If we guess the secret number we get success exit code and a message printed:

$ ./wallet 1234
Unlocked!
$ echo $?
0

Patching the binary

Next, we're going to use Radar2 tool to do the binary patching.

Open it with Radar2:

$ r2 -w ./wallet # -w stands for write mode
[0x00001060]> aa
[0x00001060]> s main # seek main function
[0x00001150]> pdf
            ; DATA XREF from entry0 @ 0x1078(r)
┌ 107: int main (int argc, char **argv);
│           ; arg int argc @ rdi
│           ; arg char **argv @ rsi
│           ; var int64_t var_4h @ rbp-0x4
│           ; var int64_t var_8h @ rbp-0x8
│           ; var int64_t var_10h @ rbp-0x10
│           ; var int64_t var_14h @ rbp-0x14
│           0x00001150      55             push rbp
│           0x00001151      4889e5         mov rbp, rsp
│           0x00001154      4883ec20       sub rsp, 0x20
│           0x00001158      c745fc0000..   mov dword [var_4h], 0
│           0x0000115f      897df8         mov dword [var_8h], edi     ; argc
│           0x00001162      488975f0       mov qword [var_10h], rsi    ; argv
│           0x00001166      488b45f0       mov rax, qword [var_10h]
│           0x0000116a      488b7808       mov rdi, qword [rax + 8]
│           0x0000116e      e8cdfeffff     call sym.imp.atoi           ; int atoi(const char *str)
│           0x00001173      8945ec         mov dword [var_14h], eax
│           0x00001176      817decd204..   cmp dword [var_14h], 0x4d2
│       ┌─< 0x0000117d      0f851a000000   jne 0x119d
│       │   0x00001183      488d3d7a0e..   lea rdi, str.Unlocked__n    ; 0x2004 ; "Unlocked!\n"
│       │   0x0000118a      b000           mov al, 0
│       │   0x0000118c      e89ffeffff     call sym.imp.printf         ; int printf(const char *format)
│       │   0x00001191      c745fc0000..   mov dword [var_4h], 0
│      ┌──< 0x00001198      e915000000     jmp 0x11b2
│      │└─> 0x0000119d      488d3d6b0e..   lea rdi, str.Wrong_passcode__This_incident_will_be_reported__n ; 0x200f ; "Wrong passcode! This incident will be reported!\n"
│      │    0x000011a4      b000           mov al, 0
│      │    0x000011a6      e885feffff     call sym.imp.printf         ; int printf(const char *format)
│      │    0x000011ab      c745fc0100..   mov dword [var_4h], 1
│      │    ; CODE XREF from main @ 0x1198(x)
│      └──> 0x000011b2      8b45fc         mov eax, dword [var_4h]
│           0x000011b5      4883c420       add rsp, 0x20
│           0x000011b9      5d             pop rbp
└           0x000011ba      c3             ret

Let's break down the provided assembly code and understand its relation to C code:

There's a bunch of instructions related to args and string convertion to int value. But we actually care about the if statement here.

In our code:

 if (num == 1234) {
  ...
 }

Is mached to:

cmp dword [var_14h], 0x4d2
jne 0x119d

The instruction jne will jump to 0x119d which is 0x0000119d if the cmp returns false, which will print "Wrong passcode...". What we want is to remove this condition or change it to be always true. One way of achieving this is by overriding it with a noop (no operation) instruction, and that's exactly what we're going to do next.

0x90 is a NOP instruction on Intel x86 CPU family. https://en.wikipedia.org/wiki/NOP\_(code).

[0x00001150]> s 0x0000117d # navigate to jne
[0x0000117d]> px # print instruction as this location
- offset -  7D7E 7F80 8182 8384 8586 8788 898A 8B8C  DEF0123456789ABC
0x0000117d  0f85 1a00 0000 488d 3d7a 0e00 00b0 00e8  ......H.=z......
...
[0x000011de]> wx 909090909090 # override 6 bytes with noop instructions

Let's review the amended asm again:

[0x00001150]> pdf
            ; DATA XREF from entry0 @ 0x1078(r)
┌ 86: int main (int argc, char **argv);
│           ; arg int argc @ rdi
│           ; arg char **argv @ rsi
│           ; var int64_t var_4h @ rbp-0x4
│           ; var int64_t var_8h @ rbp-0x8
│           ; var char **var_10h @ rbp-0x10
│           ; var uint32_t var_14h @ rbp-0x14
│           0x00001150      55             push rbp
│           0x00001151      4889e5         mov rbp, rsp
│           0x00001154      4883ec20       sub rsp, 0x20
│           0x00001158      c745fc0000..   mov dword [var_4h], 0
│           0x0000115f      897df8         mov dword [var_8h], edi     ; argc
│           0x00001162      488975f0       mov qword [var_10h], rsi    ; argv
│           0x00001166      488b45f0       mov rax, qword [var_10h]
│           0x0000116a      488b7808       mov rdi, qword [rax + 8]    ; const char *str
│           0x0000116e      e8cdfeffff     call sym.imp.atoi           ; int atoi(const char *str)
│           0x00001173      8945ec         mov dword [var_14h], eax
│           0x00001176      817decd204..   cmp dword [var_14h], 0x4d2
│           0x0000117d      90             nop
│           0x0000117e      90             nop
│           0x0000117f      90             nop
│           0x00001180      90             nop
│           0x00001181      90             nop
│           0x00001182      90             nop
│           0x00001183      488d3d7a0e..   lea rdi, str.Unlocked__n    ; 0x2004 ; "Unlocked!\n" ; const char *format
│           0x0000118a      b000           mov al, 0
│           0x0000118c      e89ffeffff     call sym.imp.printf         ; int printf(const char *format)
│           0x00001191      c745fc0000..   mov dword [var_4h], 0
│       ┌─< 0x00001198      e915000000     jmp 0x11b2
..
│       │   ; CODE XREF from main @ 0x1198(x)
│       └─> 0x000011b2      8b45fc         mov eax, dword [var_4h]
│           0x000011b5      4883c420       add rsp, 0x20
│           0x000011b9      5d             pop rbp
└           0x000011ba      c3             ret

So why did we uesd wx 909090909090 you might ask.

As we already mentioned, 90 is an opcode for nop and the jump instruction 0f851a000000 is a 6 bytes long:

0F - Primary jump instruction opcode. 85 - Secondary jump instruction opcode - jump near if not zero/not equal (ZF=0) 1a000000 - The relative address to jump to.

So we had to override all these bits inorder to maintain the correct flow of the program.

Now we can exit the radar2 and run the changed binary.

$ ./wallet 0
Correct!
$ echo $?
0

That's it!

We successfully patched the binary and we'll get a bypass no matter what integer we input to our wallet binary.

If we want to revert the changes, we can follow the same steps in reverse order. It's all about setting the right bits at the right place (which sounds easier than it is in practice).

Summary

Binary patching is a powerful technique for modifying software behaviour without access to the original source code.

In this article, we demonstrated how to navigate, find, read and modify the behaviour of a simple binary using the Radare2 tool.

The main thing is to understand the binary itself, in our example we changed the program that we wrote, so obviously it was easier than modifying an external binary because we don't have access to its source.

I did this work as part of my Ph.D. research project where I intend to change interpreter behaviour at runtime by patching function instructions.

This article serves as a documentation of my own understanding and the organization of my thoughts, aiming to share knowledge with others.

I hope you find it valuable! If you have any further questions or wish to delve into more details, feel free to reach out.

We're continuing our journey of implementing our interpreter called Coconut.

We're going to build on top of the previous functionality of our interpreter. We will extend our Lexer and Parser and implement variable declaration and evaluation as well as a simple built-in function support.

If you haven’t already, I recommend checking out my previous article: Writing Interpreter in Rust — Repl, Files, and Comments (Part 4)

Introduction

In order to add variables to the Coconut interpreter we will need to add new tokens to our Lexer. The same goes for built-in functions. We will also implement multi-statement evaluation support, with a ";" termination token - it will be handy when working with variables.

Lexer

%%
[0-9]+ "INTEGER"
\+ "ADD"
\* "MUL"
\( "LPAR"
\) "RPAR"
; ";" 
= "ASSIGN"
let "LET" 
println "PRINT_LN" 
[a-zA-Z0-9_]+ "IDENTIFIER"
[\t\n ]+ ;
//[^\n]*?$ ;

; - Statment termination token (same as in any C-like language).

ASSIGN - Assignment operator. It will assign values to variables.

LET - Variable declaration token.

IDENTIFIER - Identity token, will be used for variable names.

PRINT_LN - Buildin function that will print expression evaluation results to the stdout.

Parser

Here's what our parser will look like:

%start StatementList
%%

StatementList -> Result<Vec<Node>, ()>:
    StatementList Statement { append($1.map_err(|_| ())?, $2.map_err(|_| ())?)  }
    | { Ok(vec![]) }
    ;

Statement -> Result<Node, ()>:
   ';' { Ok(Node::Empty{}) }
    | Expression ';' { $1 }
    | Builtins { $1 }
    ;
    
Expression -> Result<Node, ()>:
    AdditiveExpression { $1 }
    | PrimaryExpression 'ASSIGN' Expression {
        match $1.map_err(|_| ())? {
            Node::Id { value } => {
                Ok(Node::Assign { id: value, rhs: Box::new($3?) })
            },
            _ => Err(())
        }
    }
    | 'LET' PrimaryExpression 'ASSIGN' Expression {
        match $2.map_err(|_| ())? {
            Node::Id { value } => {
                Ok(Node::Declare { id: value, rhs: Some(Box::new($4?)) })
            },
            _ => Err(())
        }
    } 
    ;

AdditiveExpression -> Result<Node, ()>:
    MultiplicativeExpression { $1 }
    | AdditiveExpression 'ADD' MultiplicativeExpression { 
        Ok(Node::Add{ lhs: Box::new($1?), rhs: Box::new($3?) })
    }
    ;

MultiplicativeExpression -> Result<Node, ()>: 
    PrimaryExpression { $1 }
    | MultiplicativeExpression 'MUL' PrimaryExpression { 
      Ok(Node::Mul{ lhs: Box::new($1?), rhs: Box::new($3?) })
    }
    ;

PrimaryExpression -> Result<Node, ()>:
    'IDENTIFIER' { Ok(Node::Id { value: $lexer.span_str(($1.map_err(|_| ())?).span()).to_string() }) }
    |  'LPAR' Expression 'RPAR' { $2 }
    | 'INTEGER' { parse_int($lexer.span_str(($1.map_err(|_| ())?).span())) }
    ;

Builtins -> Result<Node, ()>:
    'PRINT_LN' 'LPAR' Expression 'RPAR' { Ok(Node::PrintLn{ rhs: Box::new($3?) }) };

%%
use crate::ast::Node;

fn append(mut lhs: Vec<Node>, rhs: Node ) -> Result<Vec<Node>, ()>{
    lhs.push(rhs);
    Ok(lhs)
}

fn parse_int(s: &str) -> Result<Node, ()> {
    match s.parse::<u64>() {
        Ok(n_val) => Ok(Node::Number{ value: n_val }),
        Err(_) => {
            eprintln!("{} cannot be represented as a u64", s);
            Err(())
        }
    }
}

Ast Node enum:

#[derive(Debug, Clone, PartialEq)]
pub enum Node {
    Add { lhs: Box<Node>, rhs: Box<Node> },
    Mul { lhs: Box<Node>, rhs: Box<Node> },
    Number { value: u64 },
    Id { value: String },
    PrintLn { rhs: Box<Node> },
    Assign { id: String, rhs: Box<Node> },
    Declare { id: String, rhs: Option<Box<Node>> },
    Empty,
}

It's a bit longer and more complex than what we had before.

The main changes are:

1. Added Builtins rule where built-in functions like println can be defined.

2. Added PrimaryExpression rule for variable declaration and assignment such as: let a = 6;

3. Added support for multiple statements with ; termination token. Now we can do something like this: let a = 1; let b = 3; println(a+b);

And it's structured in a way that we can extend easily later on. We can add new data types like booleans, add new built-in functions etc...

Evaluation

Generally, the evaluation is done in the same way as before. We added new Ast nodes to represent our new functionality and we added a notion of a Scope. Now when we support variables, these variables' identities (aka names) and their values need to be stored somewhere and that somewhere is this Scope.

Our Scope is a simple struct:

use std::collections::HashMap;

pub struct Scope {
    store: HashMap<String, u64>,
}

impl Scope {
    pub fn new() -> Self {
        Scope {
            store: HashMap::new(),
        }
    }

    pub fn dec_var(&mut self, id: String, val: u64) -> Option<u64> {
        self.store.insert(id, val)
    }

    pub fn set_var(&mut self, id: String, val: u64) -> Option<u64> {
        self.store.insert(id, val)
    }

    pub fn get_var(&self, id: String) -> Option<&u64> {
        self.store.get(&id.clone())
    }
}

Basically, it's a wrapper around the HashMap data structure, so we could've used HashMap directly but I think it creates a nice separation with not much overhead. It will also allow us to extend it in the future when we deal with functions, closures and different scopes.

With the scop in place, we can evaluate source code line by line and declare, assign and reassign variables.

For example:

let a = 1;
let b = 2;
a = 3;
print(a+b); // prints 5

Evaluation

#[derive(Debug, PartialEq, Clone)]
pub enum BinaryOp {
    Add,
    Mul,
    Assign { name: String },
    Declare { name: String },
}

#[derive(Debug, PartialEq, Clone)]
pub enum Op {
    Add,                 // Addition operation
    Mull,                // Multiplication operation
    Push { value: u64 }, // Load numeric value onto stack
    Assign { name: String },
    Declare { name: String },
    PrintLn,
    Load { id: String },
}

pub fn eval(ast: Vec<Node>, scope: &mut Scope) -> Result<Option<u64>, String> {
    let ops = &mut vec![];
    for a in ast {
        ast_to_bytecode(a, ops);
    }
    let mut stack: Vec<u64> = vec![];

    for instruction in ops {
        match instruction {
            Op::Push { value } => stack.push(*value),
            Op::Add => {
                let rhs = stack.pop().unwrap();
                let lhs = stack.pop().unwrap();
                stack.push(lhs + rhs);
            }
            Op::Mull {} => {
                let rhs = stack.pop().unwrap();
                let lhs = stack.pop().unwrap();
                stack.push(lhs * rhs);
            }
            Op::Assign { name } => {
                let val = stack.pop().unwrap();
                scope.set_var(name.clone(), val);
            }
            Op::Declare { name } => {
                let val = stack.pop().unwrap();
                scope.dec_var(name.clone(), val);
            }
            Op::PrintLn => {
                println!("{}", stack.pop().unwrap());
            }
            Op::Load { id } => {
                if let Some(value) = scope.get_var(id.clone()) {
                    stack.push(value.clone());
                } else {
                    return Err(format!("Variable '{}' not found", id.clone()));
                }
            }
        }
    }
    return Ok(stack.pop());
}

Except for the additional scope variable where we can declare and retrieve variables and their values, it should have the same structure as before.

We still using our Stack for evaluation.

Conversion from AST to Bytecode

Here's our ast_to_bytecode function:

pub fn ast_to_bytecode(node: Node, ops: &mut Vec<Op>) {
    match node {
        Node::Add { lhs, rhs } => {
            ast_to_bytecode(*lhs, ops);
            ast_to_bytecode(*rhs, ops);
            ops.push(Op::Add {})
        }
        Node::Mul { lhs, rhs } => {
            ast_to_bytecode(*lhs, ops);
            ast_to_bytecode(*rhs, ops);
            ops.push(Op::Mull {})
        }
        Node::Number { value } => ops.push(Op::Push { value }),
        Node::Declare { id, rhs } => {
            if let Some(val) = rhs {
                ast_to_bytecode(*val, ops);
            }
            ops.push(Op::Declare { name: id.clone() });
        }
        Node::Assign { id, rhs } => {
            ast_to_bytecode(*rhs, ops);
            ops.push(Op::Declare { name: id.clone() });
        }
        Node::Id { value } => ops.push(Op::Load { id: value }),
        Node::PrintLn { rhs } => {
            ast_to_bytecode(*rhs, ops);
            ops.push(Op::PrintLn {})
        }
        Node::Empty {} => {}
    }
}

This change also has no big surprises, this function extended to accompany the new AST nodes and appropriate bytecodes.

Tests

And of course, let's not forget about our tests.

We added some simple tests to test that our evaluation still works as expected - with and without variables (keeping previous tests in place).

#[cfg(test)]
#[test]
fn vars_declare_match() {
    assert_eq!(
        eval_str(&"let x = 1; let y = 2; y + x;".to_string()).unwrap(),
        Some(3)
    );
}
#[test]
fn vars_reassign_math() {
    assert_eq!(
        eval_str(&"let x = 1; let y = 2; x = 3; x + y;".to_string()).unwrap(),
        Some(5)
    );
}
#[test]
fn vars_undeclared_variable() {
    assert_eq!(
        eval_str(&"a + 1;".to_string()),
        Err("Variable 'a' not found".to_string())
    );
}

We didn't add a test for the println functionality, and that's intentional cause I didn't want to add more complexity here with stdout testing - we can explore this in the following posts.

Full source code can be found here: TODO Link.

Summary

In this article, we've expanded the capabilities of the Coconut interpreter by introducing variable declaration and assignment. Additionally, we've incorporated a simple println function, enabling us to output evaluated results to stdout.

To facilitate these changes, we introduced a new concept called Scope. This allows us to store variable declarations and their corresponding values, with potential extensions planned for the future.

While much of the heavy lifting was handled by the grmtools Lexer and Parser, the rest of the implementation followed a straightforward path.

This article serves as a documentation of my own understanding and the organization of my thoughts, aiming to share knowledge with others.

I hope you find it valuable! If you have any further questions or wish to delve into more details, feel free to reach out.

We're continuing our journey of implementing our interpreter called Coconut.

In this article, we're going to implement the REPL (Read-Eval-Print Loop), re-organise the project and add file and comments support. It will be a collection of small changes that will extend the usability of our interpreter.

If you haven’t already, I recommend checking out my previous article: Writing an Interpreter in Rust: Bytecode and Stack-Based VM (Part 3)

Introduction

A REPL (Read-Eval-Print Loop) allows the user to interactively execute code and see the results immediately.

Not all interpreters have REPL, but having REPL allows them to prototype fast, and test ideas and language features quickly.

I personally find REPL very handy especially when I am not 100% familiar with the language interface.

Project Structure

I have restructured the code a bit, but nothing fundamentally changed from the last implementation, I just moved the code form the main to separate modules in order to have a clear separation of concern.

Our project content:

$ ls -l ./src

ast.rs - AST logic
bytecode.rs - Bytecode logic
coconut.l - Lexer logic
coconut.y - Parser logic
lib.rs - Main library crate
main.rs - Main program
parser.rs - Parsing logic

Our main.rs file is very simple:

use std::env;

use coconut::eval_str;

fn main() {
    println!("Writing Interpreter With Rust Part 4");
    let args: Vec<String> = env::args().collect();
    if args.len() > 1 {
        let input = &args[1];
        match eval_str(input) {
            Ok(Some(result)) => {
                println!("{}", result);
            }
            _ => eprintln!("Unable to evaluate expression."),
        }
    } else {
        println!("Please provide at least one cli argument!")
    }
}

The main responsibility of main.rs is to get the program input, call the appropriate functions of the lib.rs crate, and display results or errors to the console.

High-level program evaluation flow:

We get program input as CLI argument, then we call eval_str which will parse the string to an AST, then AST will be translated to Bytecode and then this Bytecode will be evaluated.

Let's implement the REPL.

Implementing REPL

So far we were running our Coconut programs as CLI arguments, as:

$ cargo run '2+2'
4

We're about to change that!

The main component of REPL is (surprisingly) the loop, so let's start with that.

Here we go:

use std::{
    env,
    io::{stdin, stdout, Write},
};

use coconut::eval_str;

fn main() {
    println!("Writing Interpreter With Rust Part 4");
    let args: Vec<String> = env::args().collect();
    if args.len() > 1 {
        eval(&args[1])
    } else {
        repl()
    }
}

fn repl() {
    loop {
        print!("> ");
        stdout().flush().unwrap();
        match stdin().lines().next() {
            Some(Ok(input)) => {
                if input.trim() == "exit" {
                    break;
                }
                if input.trim().is_empty() {
                    continue;
                }
                eval(&input);
            }
            _ => {}
        }
    }
}

fn eval(input: &String) {
    match eval_str(input) {
        Ok(Some(result)) => {
            println!("{}", result);
        }
        _ => eprintln!("Unable to evaluate expression."),
    }
}

Try it out:

$ cargo run
> 2+2
4
> 5
5
> exit

Works as expected, each line will be evaluated seperatly and the process will repeat until we type exit.

That's it, we have a REPL!

Addding file support

So now we have two ways of executing our Coconut programs; with CLI arguments or as REPL. Let's add another way - files.

It will allow us to save programs into files and pass them to the interpreter that will evaluate the source code line by line, just as if we were typing the lines in the REPL.

Let's add file support:

use std::{
    env, fs,
    io::{stdin, stdout, Write},
};

fn main() {
    println!("Writing Interpreter With Rust Part 4");
    let args: Vec<String> = env::args().collect();
    if args.len() > 1 {
        if args[1].ends_with(".cnt".clone()) {
            eval_file(args[1].clone())
        } else {
            eval(&args[1])
        }
    } else {
        repl()
    }
}

fn eval_file(file_name: String) {
    match fs::read_to_string(file_name) {
        Ok(content) => {
            eval(&content);
        }
        Err(_) => eprintln!("Unable to evaluate expression."),
    }
}

Here, before running the interpreter we check whether the CLI argument contains the extension ".cnt" (short for Coconut). If it is, we are reading the file and then evaluate it in the same fashion - line by line.

That's about it.

Adding comments support

That might feel like a feature creep, but I thought that we could add a small feature to our interpreter anyway, as you will witness it will be a very tiny change. We're going to add comment support.

For simplicity reasons, we're going to support only single-line comments. So if we have a file named math.cnt with the following content:

// 1+2
2+2

We would expect our interpreter to evaluate only the 2+2.

Luckily, the only thing we need to do for that is to add the following line to our Lexer:

//[^\n]*?$ ;

You can check the Regex yourself. Basically, it will match anything between the // characters and the end of the line and remove it from parsing.

Our Lexer should look something like this:

%%
[0-9]+ "INTEGER"
\+ "ADD"
\* "MUL"
\( "LPAR"
\) "RPAR"
//[^\n]*?$ ;
[\t\n ]+ ;

Run it:

$ cargo run math.cnt
4

And don't forget about tests:

#[test]
fn comments() {
    assert_eq!(
        eval_str(&"// 2+2\n 1+1".to_string()).unwrap(),
        Some(2),
        "expected 1+1=2"
    );
    assert_eq!(
        eval_str(&"// 2+2".to_string()).unwrap(),
        None,
        "expected 1+1=2"
    );
}

And we're done.

Summary

This time we didn't introduce any new concepts or components into the Coconut interpreter, we just refactored the project and added REPL, file evaluation and comment support.

REPL will give us the ability to try out our interpreter's functionality without relaunching the interpreter every time, and program files will allow us to store our Coconut programs for later evaluation. And we can add comments to our source code. What more do we need :)?

This article was written for my own sake of understanding and the organisation of my thoughts as it was about knowledge sharing.

I trust that it proved valuable!

We’re continuing our journey of implementing an interpreter called Coconut in Rust!

In this article, you're diving into the concept of Bytecode and its implementation within the Coconut interpreter. We will also discuss bytecode evaluation in stack-based virtual machines and compare them to other alternatives.

If you haven’t already, I recommend checking out my previous article: Writing an Interpreter in Rust: AST (part 2)

Let's go.

What's Bytecode?

Bytecode is a crucial component in many interpreters and VMs (Virtual Machines). In our context, VM is a software-based environment that simulates the program execution. Our Coconut language is not running directly on the hardware, it is simulated by our VM (aka interpreter).

Here we're going to use the terms Virtual Machine and Interpreter interchangeably.

Bytecode is a low-level representation of a program's source code that is designed to be executed by the VM. It acts as an intermediary step between the high-level human-readable source code and the machine code executed by the VM.

How is it different from AST?

It might be confusing, but it's different from bytecode in its essence. AST is a structural representation of the program while Bytecode is just an instruction set that is executed by the VM.

I think it will make more sense once we'll see an actual practical example.

Advantages of Bytecode

You might wonder why do we need so many components in our interpreter: Lexer, Parser, AST, Bytecode. And its hard to justrify when we're hacing our toy Coconut example that all it does is adding to numbers. But I will try :)

Here are a few advantages of bytecode:

Portability

Bytecode can be generated once and executed on multiple target platforms. Think about it as a compilation of the source code.

Performance and optimisations

Interpreting and making optimisations on the Bytecode level is often faster than interpreting or compiling from the source code.

Bytecode can serve as an intermediary phase for JIT compilation. Some VMs, like the JVM (Java Virtual Machine), can compile bytecode into native machine code at runtime. JIT can lead to significant performance gains. But it's definitely out of this article's scope.

Separation of Concerns

As with AST, we separated the parsing and evaluation phases. With Bytecode we're separating the code structural model (how the program is organized) from the execution model (how the program is run). It makes our code more modular and easier to maintain.

Stack-based VM

Why are we talking about stacks suddenly?

There are different types of VMs, the two most common are:

Stack-based VM - stack machines use an operand stack to push and pop results to.

Stack-based VM - register machines use a number of registers to store values or pass arguments.

I chose to implement a stack-based VM cause it just simpler. With stack-based machines, there's no need to keep track of multiple register usage when compared to Register-based VMs.

Our stack will be very simple, just a vector, an array with two functions used, pop and push.

Defining Bytecode

Bytecode instructions encapsulate the operations and actions that the VM can perform. Let's define it based on the interpreter functionality we have so far:

#[derive(Debug, PartialEq, Clone)]
pub enum Op {
    Add, 
    Mull,
    Push { value: u64 }, 
}

Op is short for operation.

Here's a breakdown:

Add - This instruction represents the addition operation. It will pop the top two values from the stack, perform addition, and push the result back onto the stack.

Mull - This instruction represents the multiplication operation. Similar to the Add instruction, it will pop the top two values from the stack, perform multiplication, and push the result back onto the stack.

Push - This instruction is used to load a numeric value onto the stack.

Implementing stack-based VM

To implement a stack-based VM, we will need to have a Stack (surprisingly enough). But we will also need to convert our AST to Bytecode since want to move away from AST-time evaluation.

And that's what we're going to do next.

pub fn ast_to_bytecode(node: Node, ops: &mut Vec<Op>) {
    match node {
        Node::Add { lhs, rhs } => {
            ast_to_bytecode(*lhs, ops);
            ast_to_bytecode(*rhs, ops);
            ops.push(Op::Add {})
        }
        Node::Mul { lhs, rhs } => {
            ast_to_bytecode(*lhs, ops);
            ast_to_bytecode(*rhs, ops);
            ops.push(Op::Mull {})
        }
        Node::Number { value } => ops.push(Op::Push { value }),
    }
}

Note that ast_to_bytecode is recursive, since we're dealing with a tree here.

An expression such as 3+2*(2+1) would be parsed as AST:

Add { 
        lhs: Number { value: 3 }, 
        rhs: Mul { 
            lhs: Number { value: 2 }, 
            rhs: Add { 
                lhs: Number { value: 2 }, 
                rhs: Number { value: 1 } 
        } 
    } 
}

And Bytcode:

Push { value: 3 }, 
Push { value: 2 }, 
Push { value: 2 }, 
Push { value: 1 }, 
Add, 
Mull, 
Add

Let's break down the actual evaluation of this Bytcode iterms of Add, Mul and Stack operations:

                                        # stack = []
Push 3                                  # stack = [3]
Push 2                                  # stack = [3,2]
Push 2                                  # stack = [3,2,2]
Push 1                                  # stack = [3,2,2,1]
Push (Add(pop(), pop()) = 1 + 2 = 3)    # stack = [3,2,3]
Push (Mul(pop(), pop()) = 2 * 3 = 6)    # stack = [3,6]
Push (Add(pop(), pop()) = 6 + 3 = 9)    # stack = [9]

The result is 9. As expected!

Next, we're going to evaluate these operations one by one:

pub fn eval(ast: Vec<Node>) -> Option<u64> {
    let ops = &mut vec![];
    for a in ast {
        ast_to_bytecode(a, ops);
    }
    
    let mut stack: Vec<u64> = vec![]; // Our stack
    
    for instruction in ops {
        match instruction {
            Op::Push { value } => stack.push(*value),
            Op::Add => {
                let rhs = stack.pop().unwrap();
                let lhs = stack.pop().unwrap();
                stack.push(lhs + rhs);
            }
            Op::Mull {} => {
                let rhs = stack.pop().unwrap();
                let lhs = stack.pop().unwrap();
                stack.push(lhs * rhs);
            }
        }
    }
    return stack.pop();
}

That's it. We have an interpreter that parses our syntax into AST and then generates Bytecode which is evaluated as a Stack-based VM.

Now we have basic components that will allow us to extend our interpreter functionality more easily in the future.

Tests

We did change our interpreter internals but nothing really changed from the end user perspective, so our tests we've implemented before should be still good to go.

Tests are great, make more tests please!

Full source code:

instruction.rs - https://gist.github.com/Pavel-Durov/44dd21c246cb199ecb35a2cf31aa816f

ast.rs - https://gist.github.com/Pavel-Durov/44dd21c246cb199ecb35a2cf31aa816f

main.rs - https://gist.github.com/Pavel-Durov/85e206411b870107d7033d047fe59a8c

Summary

We’ve implemented Bytecodes in our Coconut interpreter, and we moved away from AST-time evaluation.

We discussed briefly different VM implementations and the advantages of Stack-based VMs accompanied by practical Rust examples.

All the changes were interpreter internal only so no changes in Parsing, Lexing or Testing were needed.

This change will allow us to extend the Coconut interpreter further in the future.

Next, we will start extending the Coconut interpreter functionality and hopefully, it will be clearer why we need all these components: Lexer, Parser, AST, and Bytecode in our VM.

This article was written for my own sake of understanding and the organisation of my thoughts as it was about knowledge sharing.

I trust that it proved valuable!

This article overviews the process of writing an interpreter with Rust programming language. We will use the grmtools Rust crate to help with the parsing. We will define tokens of our language and create math addition and multiplication expressions. We will also cover some basic terms and concepts related to writing an interpreter as we go. This writing is heavily based on grmtools Quickstart [1].

What are grmtools

Grmtools is a collection of Rust libraries for text parsing at compile and run times. We will focus on the compile-time YACC features of grmtools as they will provide us with the framework for parsing instead of writing one from scratch.

What's YACC?

YACC (Yet Another Compiler-Compiler) is an LALR (Look Ahead Left-to-Right Rightmost Derivation) parser generator [2].
Don't let these acronyms scare you. LALR parser is just a type of Parser. A Parser is a compiler component that translates the raw source code into a meaningful structured form.
YACC is written in a similar notation to BNF (Backus–Naur Form) [3], which will resemble what we will use. The only difference is that we'll have some Rust code and types involved.

Why YACC?

We don't have to use YACC or any YACC-compatible tools like grmtools, but it saves us time from defining our own Lexers and Parsers from scratch.If you do want to implement lexing and parsing from scratch, I would suggest reading one of these books: Crafting Interpreters, Writing Interpreter in Go.

Getting started

Grmtools includes a YACC-style Parser called lrpar and a Lexer called lrlex. We will also use cfgrammar which we'll use to identify the YACC variant we will use - YaccKind::Grmtools.

Some Terminology:
Lexeme - Part of the source code text
Lexer - Breaks the text into individual lexemes
Parser - Validates whether lexemes fit the formal grammar
Tokens - Parsed Lexemes with the assigned type

In the following sections, we will define our Lexer and Parser in a YACC-y way using grmtools libraries in a Rust project.

Creating new project

I will call my project Coconut for two reasons: Just because I can and also cause like coconuts.
Creating new cargo project:

$ cargo new coconut

It should have the following content:

$ tree coconut
    coconut
    ├── Cargo.toml
    └── src
        └── main.rs

Setting up Cargo.toml

[package]
name = "coconut"
version = "0.0.1"
edition = "2021"

[[bin]]
doc = false
name = "coconut"

[build-dependencies]
cfgrammar = "0.13"
lrlex = "0.13.1"
lrpar = "0.13.1"

[dependencies]
cfgrammar = "0.13"
lrlex = "0.13.1"
lrpar = "0.13.1

Now we have all the dependencies we need.

Lexer

First, we need to define the vocabulary we will use in our interpreter. For that, we'll create a coconut.l file in our src directory and define our keywords as key-value pairs, where the key is a normal Regex expression, and the value is the formal Lexeme assigned to it, separated by whitespace.

%%
[0-9]+ "INTEGER"
\+ "ADD"
\* "MUL"
\( "LPAR"
\) "RPAR"
[\t\n ]+ ;

Let's examine it line-by-line:

[0-9]+ "INTEGER"- Any string that will match the numeric sequence 0-9 will be identified as an INTEGER Lexeme.

\+ "ADD" - "+" characters will be identified as an ADD Lexeme.

\* "MUL" - "*" characters will be identified as a MUL Lexeme.

\( "LPAR" - "(" characters will be identified as an LPAR Lexeme.

\) "RPAR" - ")" characters will be identified as an RPAR Lexeme.

[\t\n ]+ ; - Any tabs, new lines and whitespaces will be replaced with empty strings. i.e. removed.

That should give you a general idea of what is possible. We can define any Regex and assign a formal lexeme to it that will be later used in the Parser if the Regex pattern matches a text sequence.

Parser

Defining the Parser will involve creating coconut.y in our src directory with the following content:

%start Expr
%%

Expr -> Result<u64, ()>:
      Expr 'ADD' Term { Ok($1? + $3?) }
    | Term { $1 }
    ;

Term -> Result<u64, ()>:
      Term 'MUL' Factor { Ok($1? * $3?) }
    | Factor { $1 }
    ;

Factor -> Result<u64, ()>:
      'LPAR' Expr 'RPAR' { $2 }
    | 'INTEGER'
      {
          let v = $1.map_err(|_| ())?;
          parse_int($lexer.span_str(v.span()))
      }
    ;
%%

fn parse_int(s: &str) -> Result<u64, ()> {
    match s.parse::<u64>() {
        Ok(val) => Ok(val),
        Err(_) => Err(())
    }
}

The coconut.y grammar file has three parts, separated by the %% lines:

1st Part - General settings

This is the part where general settings are set. At a minimum, it needs the start rule %start to know what rule should be the starting point for the Parser.

2nd part - YACC Grammar rules

Our example consists of three rules: Expr, Term, and Factor and six productions, also known as alternatives. Each rule can have multiple productions. Each production is separated by | character. Rules end with ; a symbol. Productions can reference other rules or Lexemes defined in the coconut.l file. If the production pattern matches, its action code is executed. Action code is set between { } symbols.
The $x variables refer to the respective Lexeme in the production.
For example:
Term { $1 } references the first $1 symbol.
while

Expr 'ADD' Term { Ok($1? + $3?) } rule references the first $1 and the third $3 symbols.

3rd part - Rust code

Any Rust code called by production action code can be defined here. We can also import other rust code from the crate or external crates.
In our example, that's where we defined the parse_int function.
And there we have it! We went through the grmtools Quickstart example and defined our Lexer and Parser.
Next, we'll wire up our application!

Compiling our Parser to Rust

To use the grammar we just defined, we need to compile it into Rust code. For that, we will create a build.rs file, which will provide us with such functionality.
Let's create build.rs file in the root of the project with the following content:

use cfgrammar::yacc::YaccKind;
use lrlex::CTLexerBuilder;

fn main() {
    CTLexerBuilder::new()
        .rust_edition(lrlex::RustEdition::Rust2021)
        .lrpar_config(|ctp| {
            ctp.yacckind(YaccKind::Grmtools)
                .rust_edition(lrpar::RustEdition::Rust2021)
                .grammar_in_src_dir("coconut.y")
                .unwrap()
        })
        .lexer_in_src_dir("coconut.l")
        .unwrap()
        .build()
        .unwrap();
}

If you're unfamiliar with build.rs see more information in The Cargo Book [4]

Application entry point

We did the Lexing, Parsing and Compilation steps. Here we will hook all of it to our application entry point - main.rs.
Our main.rs content:

use std::env;

use lrlex::lrlex_mod;
use lrpar::lrpar_mod;

lrlex_mod!("coconut.l"); // brings the lexer for `coconut.l` into scope.
lrpar_mod!("coconut.y"); // brings the Parser for `coconut.y` into scope.

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() > 1 {
        let input = &args[1]; // Create a lexer
        let lexer_def = coconut_l::lexerdef(); // Lex the input.
        let lexer = lexer_def.lexer(&input);
        let (res, errs) = coconut_y::parse(&lexer); // Parse the input.
        // Check for errors
        for e in errs {
            println!("{}", e.pp(&lexer, &coconut_y::token_epp));
        }
        // Print results
        match res {
            Some(Ok(r)) => println!("{:?}", r),
            _ => eprintln!("Unable to evaluate expression."),
        }
    } else {
        println!("Please provide at least one cli argument!")
    }
}

In this code, we accept a single CLI argument as input, Lexing it, Parsing it and checking for errors.
And we're done!

Running our interpreter

Try adding two numbers:

$ cargo -q run '2+2'
4

-q flag tells cargo to be quiet and not output verbose information.

Our interpreter also supports operation precedence and more complex math expressions:

$ cargo -q run '2+2*2+(2+2)'
10

If you're not impressed, it's OK. But there is something impressive going on here that we might have taken for granted.

If we're not careful, 1+2*3 math expression can be evaluated as: 1+2*3 = (1+2)*3 = 9. Which is obviously wrong, at least by the conventional definition of + and * math operations. We know that 1+2*3 should be evaluated as 1+(2*3) = 7.
We won't dive into how and why it worked; assume some magic is happening behind the scenes. But if you're curious, it is something to do with the fact that we just wrote an LR parser. Writing our grammar this way forces the "correct" parsing with "correct" operation precedence. It is essential mainly for arithmetic operations. Read more about it here [5].

Summary

We went through the steps of implementing our own interpreter using Rust programming language and grmtools: cfgrammar, lrlex and lrpar libraries.

We went through the quickstart of grmtools [1] and added a bit more context about the terminology used.

Rust project example can be found here.

This writing was for my own sake of understanding and the organisation of my thoughts as it was about knowledge sharing.

References

[1] https://softdevteam.github.io/grmtools/master/book/quickstart.html
[2] https://www.geeksforgeeks.org/introduction-to-yacc/
[3] https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form
[4] https://doc.rust-lang.org/cargo/reference/build-scripts.html
[5] https://tratt.net/laurie/blog/2020/which_parsing_approach.html

There is a variety of Programming Languages (PL) today, each with its speciality and features. But still, traditional software is written using a single programming language. We have tools and processes that allow language compositions, but these tools are cumbersome, slow and complex. This article will overview the Programming Language Composition (PLC) problem space and the applied composition techniques.

Introduction

Realistic software is usually not bound to a single domain; storage, serialisation, network, security, concurrency, and domain-specific problems are just a few examples. Domain-Specific Languages (DSL) are built and optimised for specific problems. Some general-purpose languages are better suited for performance-intensive applications, while others have better readability or broader library support.

PLC is a computer program written in multiple programming languages. PLC can enable us to write software that is not restricted by single PL limitations, allowing us to take advantage of different language features flexibly.

Programming Language Composition (PLC)

It happened to me multiple times when I was writing software in one PL, and I needed some functionality or library available only in another PL. And I was stuck! Usually, in that case, I default to writing some wrapper (aka glue code), invoking a CLI tool or some executable.

I see PLC as a solution for fast prototyping and experimentation with PL features. If we were not restricted by PL integration constraints, we would be able to migrate legacy software in an iterative way, module by module, function by function. Or write domain logic in a more readable PL (let's say Python) and implement the rest in a more performant language (for example, Rust).

Of course, combining multiple languages in the same project creates more complexity. It requires engineers to understand and be proficient in more than one language. But if we put this cognitive load aside. And do some blue-sky thinking; why would we want to build restrictive software solutions locked in one paradigm or ecosystem? After all, we know that we will reach a dead end.

Some tools and processes allow language composition. However, it comes with runtime performance impact and operational costs, such as integration effort, and a need for better tooling for tasks such as debugging and profiling.

Let's have an overview of these methods.

Distributed Computing

Distributed Computing [1] is a highly adopted approach in the IT industry that addresses this problem. Usually referred to as Microservices Architecture these days, but it doesn't have to be "micro" per see to be distributed. Distributed systems consist of multiple components that act together, communicating over the networking protocol. Since the network layer abstracts components' internals, it allows these components to be implemented independently, using any language or framework, as long as it has a well-defined communication protocol. However, it requires the application to be distributed and comes with significant infrastructure, networking and deployment complexity. Networking communication also comes with higher latency when compared to in-memory communication.

Process-based - Fork, Exec, Spawn

Fork–exec [2] is a Unix Operating System (OS) technique where an executing process spawns a new program. With a Process-based approach, the solution is composed of a collection of processes. One process can invoke another and create this relationship with internal abstraction on the OS level.

Fork - an operation where a process creates a copy of itself

Exec - an operation that runs an executable in an existing process context

Spawn - loads and executes a new "child" process. Where the "parent" process manages this "child"

Process-based - IPC

Yet another process-based approach is Inter-Process Communication (IPC). IPC is a mechanism that allows multiple processes on the same machine to communicate over the local network or memory.

Memory - Processes communicate using the same memory. One produces data, and the other consumes it.

Messaging - Processes communicate without shared memory. Instead, they are using messages. Similarly to the Client-Server model, the connection is established over the network, and messages are sent from one process to another.

Foreign Function Interface (FFI)

Some of the most popular applications are built with a mix of languages. Firefox browser is one example; made with C/C++, Java, JavaScript and Python (maybe more). With this kind of project, it is a general practice to write an application base system in low-level programming languages and extend it with high-level languages. The main reason for adopting multiple programming languages is the code reuse of existing software; another is to take advantage of language-specific features [3].

Multi-language projects face a challenge where each language has separate runtimes and address space. Therefore, there must be an established cross-process communication protocol for these programs to interact. SWIG [4] is a popular tool that addresses such challenges. SWIG is an interface compiler that links programs written in C/C++ with other languages; it is based on messaging and type conversion via a Foreign Function Interface (FFI), which can be cumbersome and costly in runtime performance.

Another operational overhead with SWIG-like tools is that cross-interactions require the definition of interface files [5], which are foreign to the interfacing languages; to integrate with SWIG, one must understand SWIG protocol and its toolchain. It has a steep learning curve and creates another barrier to its adoption.

Interpreters-based composition

Another approach to composing multiple languages as one solution is interpreter-based composition. If we have multiple interpreters written in the same language for different PLs, we can compose them in the same address space. However, it comes with challenges, such as Language Semantics. The Unipycation [6] case study shows that it is feasible to create an efficient Virtual Machine (VM) cross-language composition.

Unipycation composes multiple RPython interpreters; PyPy [7] and Pyrolog [8]. Both components have their own meta-tracing [10] JIT, which enables a similar runtime performance when the composed outcome is compared to stand-alone interpreters.

Eco [10] editor provides the fill of an ordinary text editor while using syntax-direct editing behind the scenes. Unlike traditional editors, Eco operates and saves files as tree structures rather than as a conventional text-based source files. This again creates friction when integrating with widely adopted engineering practices and tools, such as version control systems that are usually text-based.

The Unipycation case study shows that it can compose multiple languages without compromising performance with minimal engineering effort. But after all, this case study is experimental and not Production Ready in any form.

JS-Transpiled Web

JavaScript (JS) is the primary language that runs in our browsers. Recently we had a new addition of WebAssembly (WASM), but still, JS is the de-facto language of the web. JS has its limitations. So we came up with different js-transpired languages to address them, enabling us to write with strongly-typed code (TypeScript) or Haskell-like functional programming (ELM). Still, they all transpile into JS that runs in our browsers.

What will happen if we have a project written, let's say, in CoffeeScript, and we need to import a TypeScript (TS) library? We will need to convert our script to JS and import it. What if we use three different libraries implemented in three PLs? We would need to go through the same transpile process over and over again.

This is another place for the PLC application that can be a good fit.

Summary

In this article, we overviewed the problem of Programming Language Composition (PLC). As far as my research goes, I've enumerated all the main methods we have today for writing multi-lingual software.

The state of language composition and integration today is far from ideal, it's cumbersome and convoluted in my opinion, and I hope we see some improvements in that domain soon. We have solutions that work, and we keep on using them no matter the cost, without re-evaluating them. What we need is precisely that re-evaluation.

This write-up is for my own sake of understanding and the organisation of my thoughts as it was about knowledge sharing. It's based on my recent research proposal for a part-time PhD. I hope it was helpful.

If you have questions/objections/observations/complaints, don't hesitate to reach out!

References

[1] https://en.wikipedia.org/wiki/Distributed_computing

[2] https://en.wikipedia.org/wiki/Fork%E2%80%93exec

[3] https://ieeexplore.ieee.org/document/9246706

[4] https://www.swig.org/papers/Tcl96/tcl96.html

[5] https://swig.org/Doc4.0/SWIGDocumentation.html#Introduction_nn5

[6] https://soft-dev.org/pubs/pdf/barrett_bolz_tratt__unipycation_a_study_in_cross_language_tracing.pdf

[7] https://www.pypy.org/

[8] https://www.bibsonomy.org/bibtex/2f58dd8a58d274eab26c1eaf6f501e493/gron?lang=de

[9] https://tratt.net/laurie/research/pubs/html/bolz_tratt__the_impact_of_metatracing_on_vm_design_and_implementation/

[10] https://soft-dev.org/pubs/html/diekmann_tratt__eco_a_language_composition_editor/

This article will cover the process and the involved components of building a Command-Line Interface (CLI)[1] application using the Go programming language. We will cover the required libraries, directory structure, configuration files, testing, and Cross-Platform build process.

Command-line interpreter (CLI)

CLI stands for Command-Line Interface [1]. CLI application receives input from the user, do some computational tasks and produces an output. Compared to a Graphical User Interface (GUI) [2], CLI applications require fewer system resources since interactions with it don't involve graphics.

One of the cool things (at least in my opinion) about CLI applications is that when designed with composition in mind and having a well-defined In/Out interface. These applications can be composed together (similar to function composition) as one solution 🤓.

For example, if we have two CLI application A and B, we can create a composed solution of A.B having input of A and output of B.

Cross-Platform

Cross-Platform[5] applications are designed to work on more than one computing platform. For example, we can build the same software but run it on Linux, Windows and Android devices. These applications are also referred to as multi-platform, platform-agnostic or platform-independent.

The idea is very cool. Cause who wants to maintain more software than needed? But it also needs to be considered in the software design. Cross-Platform applications need to consider any Operatic System (OS) specifics. We will see examples of it when dealing with the local filesystem in the sections below 👇.

CLI project setup

We're going to use Cobra [3] CLI framework. Cobra is a very powerful, extendable and delightful framework to work with. You won't regret it, I promise 😶!

Go version used:

$ go version
go version go1.19 darwin/arm64

Initialise our project:

$ go mod init github.com/Pavel-Durov/cli-demo
go: creating new go.mod: module github.com/Pavel-Durov/cli-demo

Install Cobra and CobraCLI. CobraCLI will create our application and add CLI commands.

$ go get -u github.com/spf13/cobra/cobra
$ go install github.com/spf13/cobra-cli@latest

Now we can use CobraCLI to initialise our CLI app:

$ cobra-cli init

That's it. We have a working app! It should have the following structure:

$ tree 
├── cmd
│   └── root.go
├── go.mod
├── go.sum
└── main.go

We have the main.go file, which is the main entry point of our application. And one single CLI command called root that is the main entry point of Cobra framework. It's also a general convention to have application entry points for GO projects in the cmd directory.

Run our brand-new CLI app:

$ go run ./main.go 
A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.
Nothing match to see yet; we get the default message. Let's add some functionality.

For this demo, we're going to build a calculator CLI application. I know, very exciting!

Adding our first command

//file: ./cmd/add.go
package cmd

import (
    "github.com/spf13/cobra"
)

var addCmd = &cobra.Command{
    Use:   "add",
    Short: "Add operator",
    Long:  `Add operator, adds two integers and prints the result.`,
    Run: func(cmd *cobra.Command, args []string) {
        num1, _ := cmd.Flags().GetInt32("n1")
        num2, _ := cmd.Flags().GetInt32("n2")
        cmd.Printf("%d + %d = %d\n", num1, num2, num1+num2)
    },
}

func init() {
    addCmd.Flags().Int32("n1", 0, "--n1 1")
    addCmd.Flags().Int32("n2", 0, "--n1 2")
    addCmd.MarkFlagRequired("n1")
    addCmd.MarkFlagRequired("n2")
}

We could use CobraCLI for that as well. But I decided to go manual here.

Note that we defined the Use property, which means that in order to use add command, we need to specify first add in our CLI parameters.

Wire CLI commands together.

//file: ./cmd/root.go
package cmd

import (
    "os"

    "github.com/spf13/cobra"
)

var rootCmd = &cobra.Command{
    Use:   "[command]",
    Short: "A CLI calculator",
    Long:  `A CLI calculator that can add and subtract two numbers.`,
}

func Execute() {
    err := rootCmd.Execute()
    if err != nil {
        os.Exit(1)
    }
}

func init() {
    rootCmd.AddCommand(addCmd) // adding add command to root
}

Since our commands are part of the same package called cmd - the import and configuration are very straightforward.

Rerun our command, this time with -h flag:

$ go run ./main.go  -h
A CLI calculator that can add and subtractwo numbers.

Usage:
  calc [command]

Available Commands:
  add         Add operator
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command

Flags:
  -h, --help   help for calc

Use "calc [command] --help" for more information about a command.

As you can see, cobra did a lot of work for us. It did configure how to parse flags, help messages etc.

Run the actual command with specified parameters:

$ go run ./main.go  add --n1=1 --n2=3
1 + 3 = 4

We have a fully-fledged CLI application that can add two numbers and print the result to the stdout.

Let's add another command! This time we will add substitution.

It will be as simple as:

//file: ./cmd/sub.go
package cmd

import (
    "github.com/spf13/cobra"
)

var subCmd = &cobra.Command{
    Use:   "sub",
    Short: "Sub operator",
    Long:  `Sub operator, subtracts two integers and prints the result.`,
    Run: func(cmd *cobra.Command, args []string) {
        num1, _ := cmd.Flags().GetInt32("n1")
        num2, _ := cmd.Flags().GetInt32("n2")
        cmd.Printf("%d - %d = %d\n", num1, num2, num1-num2)
    },
}

func init() {
    subCmd.Flags().Int32("n1", 0, "--n1 1")
    subCmd.Flags().Int32("n2", 0, "--n1 2")
    subCmd.MarkFlagRequired("n1")
    subCmd.MarkFlagRequired("n2")
}

Same as before, add the new command to the root:

...
rootCmd.AddCommand(subCmd)
...

Give it a go:

$ go run ./main.go  sub --n1=10 --n2=4
10 - 4 = 6

It works exactly as intended! You can imagine how using the same process we can extend our CLI application further.

Adding some tests

I like tests, and I think you should too 😎. Adding unit tests in GO is very straightforward; however, testing CLI commands like Cobra might be a bit tricky. That's why I wanted to demonstrate how to do it.

Adding test to root CLI command:

// file: cmd/root_test.go
func TestTypeLocal(t *testing.T) {
    buf := new(bytes.Buffer)
    rootCmd.SetOut(buf)
    rootCmd.SetArgs([]string{"sub", "--n1=10", "--n2=4"})

    err := rootCmd.Execute()
    if err != nil {
        fmt.Println(err)
    }
    if buf.String() != "10 - 4 = 6\n" {
        t.Errorf("Expected 10 - 4 = 6, got %s", buf.String())
    }
}

Here, we set a buffer as an out stream for the cobra command and pass CLI arguments (aka flags), then we assert the output result - nothing fancy.

Run the tests:

$ go test ./...
ok      github.com/Pavel-Durov/cli-demo/cmd     0.207s

Adding profile/settings

What do we do if we want to store application configuration across sessions, or maybe we want to have secrets such as API keys defined outside of our application code. Whatever the reason, cobra got your back! Actually Viper[4] got your back. Viper is a configuration management tool for Go applications. Viper and Cobra works great together.

Install Viper:

$ go get github.com/spf13/viper

Configure Viper in our init function, root cmd:

// file: cmd/root.go
func initConfig() {
    home, err := os.UserHomeDir()
    cobra.CheckErr(err)
    viper.AddConfigPath(home)
    viper.SetConfigType("yaml")
    viper.SetConfigName(".calc")
    viper.ReadInConfig()
}

func init() {
    cobra.OnInitialize(initConfig)
    ...
}

If we create a local YAML file called .calc in our $HOME directory (cause that's what we configured) with the content:

$ cat ~/.calc
username: kimchi

We can now read these values in our application:

username := viper.Get("username")
if username != nil {
    fmt.Println("Hello", username)
}

We don't have to use YAML or the $HOME directory, this setup can be configured in multiple ways.

Note on $HOME directory

Notice how we used os.UserHomeDir() to get the user's home directory. This is important if we want to build a Cross-Platform[6] application. We could've hardcoded the path to the file. But why should we? GO has great platform-agnostic library support - os.UserHomeDir() will return the path to the $HOME directory specific to the machine it's running on without us changing a single line of code!

👉 On Unix (including macOS), it returns the $HOME environment variable

👉 On Windows, it returns %USERPROFILE%

👉 On Plan 9, it returns the $HOME environment variable

Building our CLI application

Go has an incredible build system that comes with everything we need.

We can easily build our application for multiple architectures and operating systems (OS):

Linux target build

$ CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o out/linux-arm64-calc -ldflags="-extldflags=-static" # linux, arm64 arch
$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o out/linux-amd64-calc -ldflags="-extldflags=-static" # linux, amd64 arch

Mac (aka darwin) target build

$ CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o out/darwin-arm64-calc -ldflags="-extldflags=-static" # mac, arm64 arch
$ CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o out/darwin-amd64-calc -ldflags="-extldflags=-static" # mac, amd64 arch

Windows target build

$ CGO_ENABLED=0 GOOS=windows GOARCH=arm64 go build -o out/windows-arm64-calc -ldflags="-extldflags=-static" # windows, arm64 arch
$ CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o out/windows-amd64-calc -ldflags="-extldflags=-static" # windows, amd64 arch

if we run all these build commands, we'll get these binaries:

$ ls -l ./out/
-rwxr-xr-x  1 ... darwin-amd64-calc
-rwxr-xr-x  1 ... darwin-arm64-calc
-rwxr-xr-x  1 ... linux-amd64-calc
-rwxr-xr-x  1 ... linux-arm64-calc
-rwxr-xr-x  1 ... windows-amd64-calc
-rwxr-xr-x  1 ... windows-arm64-calc

Environment Variables

GOOS

You probably noticed that the only thing that changed between the build targets is the GOOS environment variable. And that's all you need to change with go-build tooling! It is seriously that easy to use!

GOARCH

This is where we specify the CPU architecture we're targeting.

See all the supported GOOS and GOARCH combos:

$ go tool dist list
aix/ppc64
android/386
android/amd64
android/arm
android/arm64
darwin/amd64
....The list goes on

CGO_ENABLED

We also used CGO_ENABLED environment variable. CGO_ENABLED=1 leads to faster and smaller builds - it allows dynamically loading of the host OS's native libraries. However, it relies on a host OS, a dependency we would like to avoid! Otherwise, our code behaviour might differ from machine to machine if our code relies on the host libraries.

Linker flags - ldflags

We used flags called ldflags - ld stands for linker [6] therefore ldflags stands for linker flags. A linker is a program that "links" the pieces of the compiled source code into the binary outcome. We are passing extldflags to our linker. According to the link tool documentation, these flags are passed to the external linker. Long story short, we're using these flags to indicate to the GO build tool to include all the dependencies into the binary and not rely on them provided by the environment it's running in. We set the flag as -static, indicating that the binary should include all its dependencies. If not specified, our binary would be dynamically linked. For the same reasons as with CGO_ENABLED, we would like to avoid it here.

Summary

We've seen how to setup Cobra-based CLI applications from scratch. We touched on Cross-Platform application properties, such as platform-agnostic filesystem paths. We added tests and briefly overviewed Linker and the GO build tooling with different target configurations.

Building GO applications for multiple targets is straightforward and fun once you get the basics.

This write-up was for my own sake of understanding and organising my thoughts as it was about knowledge sharing. I hope it was helpful. If you have questions/objections/observations/complaints, don't hesitate to reach out!

Full source code can be found 👉 here.

References

[1] https://en.wikipedia.org/wiki/Command-line_interface

[2] https://en.wikipedia.org/wiki/Graphical_user_interface

[3] https://cobra.dev/

[4] https://github.com/spf13/viper

[5] https://en.wikipedia.org/wiki/Cross-platform_software

[6] https://pkg.go.dev/cmd/link

This article will cover Progressive Delivery techniques such as Feature Flags, A/B Testing, Ring Deployment, Dark Launching, Blue/Green and Canary deployments with high-level illustrations and a general concept overview.

Introduction

Continuous Delivery (CD) practice ensures that our code is constantly in a deployable state. Using CD, we can release faster with higher frequency. That's all great fun, but we can do better! When a new version is deployed in the traditional delivery manner - the old version is replaced with a new one, meaning that all of our users are immediately affected by the new release. Progressive Delivery offers techniques for deploying applications gradually over time while minimising deployment downtime.

Before jumping into Progressive Delivery implementation, we should ensure that we have a firm grasp of Agile, CI/CD, and DevOps practices, as Progressive Delivery includes more advanced software development practices and infrastructure management.

Feature Flags

The idea behind Feature Flags (FF) is quite simple: We add logical branches to our code - if/else statements and control them via configuration. This can be achieved using one of the SaaS (Software as a Service) platforms, such as Unleash[1] or LaunchDarkly[2] for configuration management. These services require additional integration and might be costly, but they offer support and a range of features. But a similar idea can be achieved with any remotely managed configuration, a versioned JSON file, for example.

Application using *Feature Flags* can be illustrated as follows:

In this example, depending on whether the FLAG is ON or OFF, we will add additional data to the response. Feature Flags are Progressive Delivery basics - it's the first step towards minimising deployment risk. It allows us to test in production, a powerful ability because test environments never represent production 100%!

Feature Flags don't involve any infrastructural changes, only application code and configuration management integration.

Feature Flags allow us to get immediate feedback for our changes functionality. And if something goes wrong, we can turn OFF the FLAG instead of rolling back the whole deployment. And guess what? We can turn it ON/OFF as many times as we like! Pretty cool concept 🤓!

Some call it "zero-risk" deployments. But I would avoid claiming that. If not managed carefully, it might also have a significant negative impact. Generally, Feature Flags should be short-lived to avoid confusion. If you don't need the Feature Flag anymore - delete it.

As always, all this goodness comes with additional complexity. Once our application has feature flags integration, we need to be confident in multiple flows and maintain more tests 😶.

A/B Testing

A/B Testing[3], aka bucket testing, aka split-run testing, is a user-experience research methodology [4]. A/B Testing can be thought of as an extension of Feature Flags. It allows us to experiment with features on different user audiences, monitor the behaviour, and conclude what works and does not.

For example, if we have two designs of a button, one is Purple, and the other is Gray, and we want to experiment to understand which one is better suited for our application - we can redirect 50% of our users to the Purple button and the other half to the Gray one and analyse the engagement stats to make a decision. The same thing can be done with demographics-based or geo-location-based logic.

If you're familiar with Blue/Green Deployments, A/B Testing might look the same at first glance🤔. But it's quite different. A/B Testing is a behaviour-monitoring paradigm. Blue/Green Deployments is an engineering practice. They can be used both together and in segregation.

Ring Deployment

With Ring Deployment[8], we can gradually deploy, validate and monitor deployment impact while limiting the effect on our users.

Deployment rings:

With Ring Deployment, we can choose to release first to our alpha users. Afterwards, we can move on to the next ring, which will include our external beta users and so on and so forth. The impact, also referenced as Blast Radius, will increase as the software release propagate through the defined rings.

Dark Launching

With Dark Launching[7], we deploy back-end changes without affecting the end users with the new behaviour. This deployment method allows for assessing system performance and load impacts on the system before publicly releasing the new version.

Using Dark Launching, we can run different feature implementations in parallel and analyse the impact while returning only a single response to the user.

Dark Launching in its all glory:

In this example, we're fetching data from both data sources - SQL and MongoDB. However, we're returning only the MongoDb data to the end user.

This method is robust for testing system load and the system's resiliency. Especially beneficial when testing user interaction enhancement changes. As always, it can be applied with Feature Flags to control the darkness 🧛.

Blue/Green Deployments

Blue/Green Deployments[5] is a deployment method where the new version is launched side-by-side with the old version - both versions co-exist during the deployment. It's one of the simplest ways to minimise deployment downtime on the infrastructure level. This method can be used in various applications such as Virtual Machines (VMs), Containers and Kubernetes clusters.

Blue/Green deployment phases:

1. All users' traffic is routed to V1 (Blue deployment)

2. V2 is deployed (Green deployment). This version has no traffic routed from the load balancer.

3. V2 internal tests are performed. If tests pass 1. traffic is routed to V2.

4. If all is good, the old version is discarded.

The actual strategy and steps might vary - some prefer to keep the last deployment or even further history of deployments.

The main benefit is that we can test release candidates in a production environment before affecting our users. If we decide to release the new version and find issues only after a while, we can quickly switch back. The load balancer switch is way faster than redeploying a new version! Therefore minimising the impact.

Canary Deployments

A Canary Deployment [6] is a deployment when only a subset of users get access to the new version of the application.

Why are they called Canary Deployments? Glad you asked.

Dating back to 1900, coal miners used canaries to detect carbon monoxide and other toxic gases before they hurt humans. Sad but true story.

Blue/Green Deployments minimise deployment downtime, but it's not perfect. If the deployment has an issue that is evident only after some time or with increasing load - then all users end up being affected since the traffic switch is all or nothing.

Canary improve on Blue/Green Deployments. Instead of switching 100% of traffic all at once to the new version, we redirect only a subset of users over time.

Canary illustration:

1. V1 is the current version (Blue deployment).

2. V2 has been deployed. V2 gets only a fraction of the traffic, for example, 20%.

3. Deployment is monitored. If all is good, we will increase the traffic to V2.

4. Eventually, 100% traffic is redirected to the new version.

The exact configuration of traffic percentage and time frames are configurable, and the whole process can be manual or automated.

Canary deployments are more complicated than Blue/Green Deployments on the infrastructure level. The load balancer here has to be "smarter", as it needs to understand how to route portions of the traffic towards configurable destinations.

There are also cost implications to this approach - side-by-side deployments cost is higher than traditional deployments as we need extra infrastructure at the deployment time.

Handling databases can be tricky when it comes to schema changes. The problem is that databases must simultaneously work with multiple application versions during deployment. So, for example, if we have breaking schema changes, we’re in trouble. We need to maintain backward and forward compatibility as we make changes, which adds another layer of complexity to software development and maintenance.

Summary

We covered quite a few deployment strategies 😅. They all have their place and use cases, some more complex than others. Some include the need for "smart" networking such as load balancer, and even service mesh, while others are strictly application-level.

These techniques can be combined together Feature Flags with Canary Deployment, Feature Flags and Dark Launching, Canary and Ring Deployments, and the list go on.

The more "stuff" we include in our software, the more complexity we'll have and that's ok if it's needed, but if it isn't, then why bother?

If you are lost and don't know where to start, I would suggest starting from the basics - CI/CD and DevOps practices.

For further reading, I suggest 👉 Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation

This write-up was for my own sake of understanding and the organisation of my thoughts as it was about knowledge sharing.

I hope it was useful!

References

[1] https://www.getunleash.io/

[2] https://launchdarkly.com/

[3] https://www.optimizely.com/optimization-glossary/ab-testing/)

[4] https://quod.lib.umich.edu/w/weave/12535642.0001.101?view=text;rgn=main

[5] Blue/Green Deployments

[6] https://martinfowler.com/bliki/CanaryRelease.html

[7] https://martinfowler.com/bliki/DarkLaunching.html

[8] https://learn.microsoft.com/en-us/azure/devops/migrate/phase-rollout-with-rings?view=azure-devops

In this article, I will overview the concept of Hexagonal Architecture (HA), the problem it addresses, its benefits, implications and some practical examples illustrated via code and diagrams.

There're many sources on this topic. My source is not the most detailed one, but my intention is to simplify the idea as much as I can and break it down to its core concepts and explain its jargon.

Introduction

Hexagonal architecture (HA) aka Ports and Adapters (P&A), is a software architectural pattern. HA is an alternative to traditional N-Tier architecture [1] aka Layered Architecture. HA divides a system into several loosely-coupled components, placing the application domain, aka application core, at its centre. Usually, this architecture is implemented or referred to within a single project structure, although the same concepts can be applied elsewhere.

The problem

In 2005, Alistair Cockburn presented Hexagonal Architecture as a solution to problems with traditional layering, coupling and entanglement 👉 more here [2].

The problem with layered architecture is that we (engineers) tend to take the separation of layers lightly. In layered applications, the domain logic usually "leaks" between layer boundaries. Projects without well-defined domains are hard to reason about and maintain. Additionally, in practice, the architecture is usually quite complex for large projects, and it's hard to project it into rigid, one-dimensional layer drawings.

In HA we don't have this dimensional restriction. We have the application on the inside communicating using abstraction over interfaces with the external world. Strict boundaries, abstraction and separation of concerns are the key here.

HA vs N-Tier Architecture

Conceptually, our software has little difference between UI, Database, Network or other layers. All these software components can be looked at as "black boxes" that have inputs and outputs. With Layered architectural diagrams, we naturally separate layers by type and not by functionality; UI at the top, Database at the bottom, and we throw something in the middle.

When we run the same software in different modes, such as Production, Testing etc, we want to have the ability to replace one type of interaction with another; UI with a test suite, Actual Database with an in-memory instance, etc... In other words, we want the same functionality but different implementations. Well, that's what HA is all about ☝️.

HA components

Domain

This is the Core, the centre, and the meaning of the application. Without it, we will have a bunch of technologies wired together that don't do anything business-related. And although it might sound cool. What's the point?

I like the way my colleague explained it to me once; domain code should be kept isolated, simple and easy to explain to someone non-technical. For example, your product manager or your cat 😻. Some separate the core into domain and application, but we'll keep it simple and treat it as one thing.

Ports

HA components are connected through Ports. Port can be anything that has a functionality contract, but for simplicity and probably practicality - imagine interfaces.

The term "Port" is inspired by the operating system (OS) concepts, where any device that respects the protocol of a port can be connected. For example, Universal Serial Bus (USB) 👉 more on that[2].

Adapters

Adapters deal with the outside world. Adapters will implement the Port contract and provide its fulfillment. There might be multiple adapters for any one port.

The Adapter OOP design pattern is one of the well-known design patterns from the classic Design Patterns[3] book. This pattern "adapts" two objects by converting one class's interface into another. The HA adapters are one implementation of this pattern.

Primary and Secondary

On the architectural level, all ports are equal in HA. However, in reality, adapters show up in two flavours: Primary and Secondary:

Primary - Actors that drive the application. UI, HTTP Endpoints, Events etc...

Secondary - Actors that are driven by the application. Database, EmailService etc...

This concept is referred to as Left-Right Asymmetry [2]

The distinction lies in which component initiates the communication.

Why is it called Hexoganal? Where is the Hex?

A hexagon is a six-sided polygon, so one might think that there are 6 components in HA, but we counted only 3 so far 🤔! If anything, it should've been called Triangular Architecture!

According to the author, the term "Hexagonal" is used not because of the number 6 significance or the beast 🤘. It's about aesthetics and convenience [2]. The idea is to allow architectural drawings with enough room to insert both Ports and Adapters as needed without the constraints of a one-dimensional layered drawing.

HA features

Inversion of Control (IoC)

In HA we use Ports. Naturally, it inverts the dependency direction. With IoC our components are not dependent on other component implementations. Instead, they are dependent upon abstraction. We leverage decoupling, modularisation and ease of testing since the implementation of dependent contracts can be changed without affecting the dependent component. Martin Fowler has much more to say about Inversion of Control[4].

Hollywood Principle

Referencing the phrase "don't call us, we'll call you", a common dismissal used in film auditions. The Hollywood Principle[5] is where components' lifecycle is handled by its context rather than by the object itself. If component A uses component B, the instance of the B will be provided to A rather than obtained by A.

Liscuv Substitute Principle

Remember Liscuv Substitute Principle [6]? Although the Liscuv principle is mainly about types and subtypes' relation, the idea is similar to me 🙂.

Separation of Concerns

Separation Of Concerns[8] is about, surprisingly, concerns separation. The problem is that our concerns are relative, but some concerns are quite obvious. If we have applications with UI concerns and Data persistence concerns, it's kind of obvious that these concerns should be separated.

In HA, our application layers are isolated and have clear responsibilities and definitions. The boundaries are connected and communicated through strict interfaces. Therefore it forces us to think about our concerns and their separation.

Domain-Driven Design (DDD)

The application core is agnostic of technologies, its concern is only the business logic. It's easier to understand and grasp the business requirements and make changes to it when the application domain is well-defined and isolated. It can also be built and tested in isolation. DDD is a whole topic of its own.

Application of HA

Let's assume we have an online shop order API application. Without going into too much detail, we can describe the functionality in a sequence diagram:

So how would that might look in HA? Something like that:

Does it have to be a hexagon-like drawing? If you ask me I couldn't care less if it's a hex a circle or a bunch of stars shining on my whiteboard as long as it identifies the application components' boundaries and responsibilities.

Let's do the same thing without hexagons (it won't sound as cool, though):

Nothing exotic here, just making a point again about why HA is called HA 🙃.

Let's break it down into software components:

  /src
    /core
      order
    /adapters
      http-server
      email
      email-mock
      db
      db-mock
    /ports
      email-port
      db-port

http-server - primary adapter driving our application.

email - secondary adapter, implements email-port, has no reference to anything in our application.

email-mock- email mock, used for testing, implements email-port.

db - secondary adapter, implements db-port, has no reference to anything in our application.

db-mock - db mock, used for testing, implements db-port

Have a look at http-server adapter. Why it has no port? Well, because it doesn't make sense. We're building a server-side service where http-server is the main interface of our application, are we going to reuse its Port in some way or call it from the core? It looks like a one-directional communication http-server -> core to me. Please reach out if you strongly disagree and think we should have a port for it!

IoC and Loose coupling

The application Core can reference any adapters, but as we said, not directly, only through abstraction. One way to achieve that is by using Inversion of Control (IOC) and Dependency Injection (DI). The idea is simple - instead of a Core module referencing other modules directly, i.e. knowing exactly where to find the DB adapter, all it's aware of is its interface. Core has zero knowledge of the interface implementation. Therefore the instance behind the interface can be an SQL DB, FileStorage, or mocked one.

Here's an example of tightly-coupled code:

// file: order-core.ts
import { Db } from "./db-adapter"

export class Order {
  private db: Db

  constructor(){
    this.db = new Db()
  }

  placeOrder(){
    this.db.saveOrder({})
  }
}

// file: db-adapter.ts
export class Db {
  saveOrder(order: any) {
    console.log("Saving order to db");
    return true
  }
}

Our application:

// file: index.ts
import { Order } from "./order-core";

const order = new Order()

order.placeOrder()

In this example, we have a rigid code with a dependency between Order and Db modules. Order class references Db class directly 👉 tightly-coupled components. The main idea here is that this code will be hard to change. Also, notice that our Order class decides how to initialise Db class. This sounds like a concern that Order class should not have.

Suppose I want to use another Db logic such as mocked one I will need to update all its references. We can live with that, but we can do better!

Alternative (with IoC):

// file: db-adapter.ts
import { DbPort } from "./db-port";

export class Db implements DbPort {
  saveOrder(order: any): boolean {
    console.log("Saving order to db");
    return true
  }
}

// file: order-core.ts
import { DbPort } from "./db-port"

export class Order {
  constructor(private Db: DbPort) {
    this.Db = Db
  }

  placeOrder(){
    this.Db.saveOrder({})
  }
}

Our application:

// file: index.ts
import { Order } from "./order-core";
import { Db } from "./db-adapter";

const order1 = new Order(new Db())
order1.placeOrder()

Here we don't have tight coupling between the Core component and the Adapter. Core knows about the Port interface and that's it! We can now define different behaviours of our Port!

import { OrderPort } from "./db-port";

export class MockDb implements OrderPort {
  saveOrder(order: any): boolean {
    console.log("You've been mocked!");
    return true
  }
}

Our application:

import { Order } from "./order-core";
import { Db } from "./db-adapter";
import { MockDb } from "./db-adapter-mock";

const prod = new Order(new Db())
prod.placeOrder()

const test = new Order(new MockDb())
test.placeOrder()

Boom 🤯! Revolutionary. Did your head explode? You've just experienced the power of Dependency Injection (DI) and Inversion of Control (IoC) 💪.

In this simple example, we need to configure each object constructor with its dependencies. It might be too much. As with anything, there're multiple ways of achieving the same thing. Most of the material about DI and IoC will cover more complex configurations such as IoC Containers[7]. Such containers might appear overwhelming, but they all have their place and use cases. Remember that it's all based on simple concepts.

Configuration Management

This might not be 100% on the topic of HA and IoC, but I think it's important to mention it anyway. Software misconfiguration is a very common source of bugs. It always happens; using HTTP instead of HTTPS protocol, wrong module import or invalid environment variables. Therefore, it makes sense to keep our configuration isolated in a single place of truth that defines its behaviour. We can also define our Adaptors there in HA. For example, the configuration of our logger type, whether it will stream the logs to a remote aggregation service or stream it to a local file, should be defined in one place. It will make our life easier when we try to understand what happened to our logs. In contrast, if each module is responsible for its log configuration, we might end up with different parts of our software producing different types of logs - probably a bad idea.

Summary

We started with Hexagonal Architecture and then talked about N-Tier architecture and how these two differ. We covered more abstract topics, such as Separation of Concerns and Inversion of Control, and briefly touched on Domain-Driven Design and Application configuration management. With HA, we are more flexible in introducing changes, defining the application domain and enforcing the application's testability.

This write-up was for my own sake of understanding and organising my thoughts as it was about knowledge sharing. I hope it was helpful. If you have questions/objections/observations/complaints, don't hesitate to reach out!

If you want to learn more, check out this this recording of Alistair Cockburn, the quality is not great, but its content is very valuable. And explore the provided references 👇.

References

[1] https://en.wikipedia.org/wiki/Multitier_architecture

[2] https://alistair.cockburn.us/hexagonal-architecture/

[3] https://www.goodreads.com/book/show/85009.Design_Patterns

[4] https://martinfowler.com/articles/injection.html

[5] https://wiki.c2.com/?HollywoodPrinciple

[6] https://en.wikipedia.org/wiki/Liskov_substitution_principle

[7] https://wiki.c2.com/?IocContainerComparison

[8] http://wiki.c2.com/?SeparationOfConcerns

In this article, we're going to cover UNIX process signals and SIGTERM signals in particular. We will cover how to handle them with practical examples using Node[1], TypeScript [2], Docker[3] and Kind[4] local cluster.

UNIX Signals and SIGTERM

Unix-based operating systems (OS) have multiple processes. OS uses software interrupts[5] (aka signals) as a way to communicate with the running processes, these signals are indicating that some sort of event has occurred and they can vary in their intent and purpose.

Here's a list of signals available on my machine:

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGEMT       8) SIGFPE       9) SIGKILL     10) SIGBUS
11) SIGSEGV     12) SIGSYS      13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGURG      17) SIGSTOP     18) SIGTSTP     19) SIGCONT     20) SIGCHLD
21) SIGTTIN     22) SIGTTOU     23) SIGIO       24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGINFO     30) SIGUSR1
31) SIGUSR2

There're a few, but we're going to focus on SIGTERM.

SIGTERM signal is a way for the operating system to terminate a program gracefully. By graceful, we mean that the program is given time to perform the final cleanup before shutdown. Depending on the application, cleanup tasks can vary. Interestingly enough, Unix processes can block and ignore SIGTERM. But if we want to have quality process/service we need to handle these signals as they intended, with respect, otherwise our process will be shutting down with force.

Unix Process as HTTP Server

For demonstration, we're going to create a sample HTTP server using TypeScript[1], and hapi[2].

Let's create NPM[6] project and follow the prompt:

$ npm init
...

Install dependencies:

$ npm install @hapi/hapi typesript @types/node @types/node @types/hapi__hapi

We're not going to bother with the split of development and production dependencies here 😉.

Create a local file, call it index.ts (or whatever you feel like today, go wild):

import { Server } from "@hapi/hapi";

function sleep(ms: number, value: string) {
  return new Promise((resolve) => setTimeout(() => resolve(value), ms));
}

export async function startServer(host: string, port: number): Promise<Server> {
  const server = new Server({ port, host });

  server.route({
    method: "GET",
    path: "/work",
    handler: async () => sleep(10 * 1000, `done something for 10 seconds\n`),
  });

  await server.start();
  console.log(`Server running at: ${server.info.uri}, PID: ${process.pid}`);
  return server;
}

startServer("0.0.0.0", 3000);

Here, we start the local server on 0.0.0.0 host and 3000 port. We also configured a single endpoint GET /work that simulates something that takes time to compute - 10 seconds in our case. In a real-case scenario that might be the time needed to perform some kind of a Database query.

I like node, its so cool that in 27 lines of code you can define a server, you actually need less, cause I added sleep function etc. But yeah, its great 😎.

Let's run our server:

$ ./node_modules/.bin/ts-node ./index.ts
Server running at: http://0.0.0.0:3000, PID: 16544

All good, we have server running!

Now let's send GET /work HTTP request to our endpoint. Pick any networking tool of your choice, I'm going to use CURL [7]:

$ curl http://0.0.0.0:3000/work
done something for 10 seconds

So far so good. But what will happen to that HTTP request if the server is suddenly killed, terminated or in other words, is no more 🤔. What response would we get on the client side? Will we get anything at all? What will be serving the request after the server is dead? Many questions! Let's give it a go:

Send another request to the server:

$ curl http://0.0.0.0:3000/work
done something for 10 seconds

But this time, while the server is doing the simulated work for 10 seconds - let's KILL it 😈!

$ kill -15 19346

Note: I'm using Process Id (PID) I got from index.ts output! Your local PID would we something else for sure! If it not, then its destiny and definitely send me an email 🤓.

One might wonder, why not just terminate the shell process of the server by pressing the CTRL+C keys? That would send SIGINT signal and we want SIGTERM!

Ok, so what happened to our client connection when the server is shut down? That's what happens:

$ curl http://0.0.0.0:3000/work
curl: (52) Empty reply from server

It got an empty reply, meaning that it got no information, nada. Let's add -v (stands for verbose) flag to our CURL command to see more information.

$ curl -v http://0.0.0.0:3000/work
*   Trying 0.0.0.0:3000...
* Connected to 0.0.0.0 (127.0.0.1) port 3000 (#0)
> GET /work HTTP/1.1
> Host: 0.0.0.0:3000
> User-Agent: curl/7.84.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

Looks like the server just closed the connection abruptly 👀. That's not good. What if this happens to the server clients in production? If you're running your application in one of the cloud orchestration tools, containers being shut down might be not out of ordinary.

I am using Kubernetes (K8S)[8] so I'm going to talk about that. K8S is like OS for the cloud. K8S might terminate the running Pods[9] (aka containers) at will, after all, the whole purpose of K8S is to orchestrate the distributed system. If one of the Pods is requesting too many resources or if the application is being scaled down, the container might get a signal from K8S that it's time to go 😢. That's why we have the notion of signals, to gracefully terminate our containers ☝️.

Service Graceful Termination

In the previous section, we talk about how our processes or containers might receive different signals. When K8S needs to terminate a Pod, it will send SIGTERM. That way our service is not just being cut out of resources, instead it will have some time to perform finalisation tasks.

Let's implement this logic in our node server:

import { Server } from "@hapi/hapi";

function sleep(ms: number, value: string) {
  return new Promise((resolve) => setTimeout(() => resolve(value), ms));
}

export async function startServer(host: string, port: number): Promise<Server> {
  const server = new Server({ port, host });

  server.route({
    method: "GET",
    path: "/work",
    handler: async () => sleep(10 * 1000, `done something for 10 seconds\n`),
  });

  process.on("SIGTERM", async function () {
    console.log(`Received SIGTERM`);
    await server.stop({ timeout: 10 * 1000 });
    console.log(`Server stopped.`);
    process.exit(0);
  });
  await server.start();
  console.log(`Server running at: ${server.info.uri}, PID: ${process.pid}`);
  return server;
}

startServer("0.0.0.0", 3000);

We added a listener on SIGTERM event. When such events occur, we're stopping the server but we're not just terminating it. Between the time that SIGTERM arrives and the specified timeout parameter, our server will refuse to accept any new requests and will finalise ongoing requests.

We can test that statement! Start the server again:

$ ts-node ./index.ts
Server running at: http://0.0.0.0:3000, PID: 67116

Run CURL request and terminate the server:

$ curl -v http://0.0.0.0:3000/work

We should see that the request is finished and a response sent back to the client - no more (52) Empty reply from server errors. If we try to access the server in the time frame between SIGTERM and the actual shutdown we will get:

$ curl -v http://0.0.0.0:3000/work
*   Trying 0.0.0.0:3000...
* connect to 0.0.0.0 port 3000 failed: Connection refused
* Failed to connect to 0.0.0.0 port 3000 after 3 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 0.0.0.0 port 3000 after 3 ms: Connection refused

We'll get the same error if there's no server at all!

But that's all we need to do for the graceful termination of node processes. Pretty easy 😃.

Sending SIGTERM to K8S Pods

As we mentioned, whenever the K8S Pod is terminated, it is sent a SIGTERM signal. Similarly to the way we used kill command for local processes, we can terminate pods using delete command:

$ kubectl delete pod my-pod-qgldf

When K8S decides for whatever reason to terminate the Pod, the SIGTERM signal will be sent to it, then to the Docker container, and eventually to the running process. You don't have to believe me, give it a go.

K8S sample application

We're going to reuse the server code, but containarise it and deploy it on the local K8S cluster.

Our Dockerfile:

FROM node:19-bullseye
WORKDIR /app
COPY index.ts package.json package-lock.json /app/
RUN npm install
EXPOSE 3000
ENTRYPOINT ["/app/node_modules/.bin/ts-node", "index.ts"]

Build docker container:

$ docker build -t poc -f ./Dockerfile .

First let's run it and see if we can get SIGTERM related logs:

$ docker run -t poc:latest

Get the docker id:

$ docker ps
CONTAINER ID   IMAGE                  
86b0a46730ba   poc:latest

And stop it:

$ docker stop 86b0a46730ba

We should see the docker run command terminated as well as our docker container with the same output as we had with the process.

$  docker run -t poc:latest
Server running at: http://0.0.0.0:3000, PID: 1
Received SIGTERM
Server stopped.

Now let's do some K8S!

First, we need to load our container image to the cluster:

$ kind load docker-image poc:latest --name my-cluster

K8S deployment manifest:

apiVersion: v1
kind: Namespace
metadata:
  name: poc-namespace
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: poc-deployment
  namespace: poc-namespace
spec:
  selector:
      matchLabels:
        app: poc-app
  template:
    metadata:
      labels:
        app: poc-app
    spec:
      containers:
      - name: poc
        image: poc:latest
        ports:
        - containerPort: 3000

Deploy to our cluster:

$ kubectl apply -f ./deployment.yaml 
namespace/poc-namespace created
deployment.apps/poc-deployment created

If you stream pod logs:

$ k logs -f poc-deployment-bf749f576-wfmv9

And then delete the pod:

$ kubectl delete pod poc-deployment-bf749f576-wfmv9

Should get the same results:

$ k logs -f poc-deployment-bf749f576-wfmv9
Server running at: http://0.0.0.0:3000, PID: 1
Received SIGTERM
Server stopped.

And that's a wrap 🌯!

Summary

In this article, we covered SIGTERM signals, how they are used in Unix-based and cloud-based systems as well as showed practical applications of handling to handle these signals and their potential impact of it.

SIGTERM handling is essential for horizontally scalable[10] cloud applications. It allows applications to scale up and down on demand without impacting the client's stability.

References

[1] https://nodejs.org/en/

[2] https://www.typescriptlang.org/

[3] https://www.docker.com/

[4] https://kind.sigs.k8s.io/

[5] https://en.wikipedia.org/wiki/Interrupt

[6] https://www.npmjs.com/

[7] https://curl.se/

[8] https://kubernetes.io/

[9] https://kubernetes.io/docs/concepts/workloads/pods/

[10] https://en.wikipedia.org/wiki/Scalability#Horizontal_(scale_out)_and_verticalscaling(scale_up)

Abstract

In this article, I will overview the case-sensitivity difference between macOS default filesystem (FS) and Linux filesystem with some practical examples and gotchas.

Intro

Unix and Linux might sound the same to some of us, but they are different beasts. macOS aka OS X or Max OS X is a Unix-based operating system, while Linux is Unix-like OS. There are many articles explaining the difference [1]. No need to repeat.

Experiment - maxOS FS case sensitivity

Let's do an experiment 🤓. Assuming that you're running on macOS, create a file called entrypoint.sh with content:

#!/bin/bash

echo "hello"

Try running it as:

$ bash ./entrypoint.sh
hello

Now rename it to Entrypoint.sh (with capital E) and run again:

$ bash ./entrypoint.sh
hello

Works anyway! File names are case insensitive, meaning that you can run it even as:

$ bash ./eNtryPOInt.sh
hello

I don't know how about you, but when I found it for the first time I was really surprised. When working with a mac, on the surface, it looks and feels very Linuxy, the terminal, the shell, the commands, the paths... It's all been a lie 😭.

Let's see how case-sensitivity would work on Linux using Docker

Define a simple dockerfile:

FROM alpine:3.14
WORKDIR /poc
COPY ./Entrypoint.sh /poc/Entrypoint.sh
ENTRYPOINT ["/bin/sh", "/poc/entrypoint.sh" ]

Here we're copying the entrypoint.sh file and running it as the ENTRYPOINT command. Notice that we run the lowercased file but we copied in uppercased file! Well, running it won't work since Linux filesystem is case-sensitive:

$ docker build -t macOS-FS . && docker run -t macOS-FS
/bin/sh: can't open '/poc/entrypoint.sh': No such file or directory

If we change the entrypoint to /poc/Entrypoint.sh it should work. Try it yourself!

FS case-sensitivity and Git

Another thing to note is that since FS is case-insensitive, these changes will not be tracked in Git 😶. I know! Quite shocking to stop thinking of Git as the source of the absolute truth of text-based file changes! The same experiment can be done by renaming one of the tracked files and seeing the Git-detected changes 🤯. It got me personally multiple times over the past years when programs worked locally but failed to launch in Linux-based containers or in my lovely CI/CD pipelines.

Using macOS case-sensitive file system

It's possible to have a case-sensitive filesystem on macOS. For that, we need to set the disc to use APFS (Case-sensitive) format: "APFS (Case-sensitive): Uses the APFS format, is case-sensitive to file and folder names. For example, folders named “Homework” and “HOMEWORK” are two different folders." [2].

However, it might be risky. Most macOS applications don’t recognize a case-sensitive file system. And probably won’t work as expected. So wouldn't recommend it. On the other hand, it might make sense to set a separate partition on your drive to be case-sensitive, especially if you work with Linux-based systems quite often. But not the whole drive.

Conclusion

macOS and Linux might look and feel the same, but in reality, they are quite different. Filesystem case sensitivity is only one example of such diffrence. I hope that this short overview of case-sensitivity differences was useful. And you won't be caught off-guard next time your application won't run when Git didn't pick up on file rename changes on macOS.

References

[1] https://www.softwaretestinghelp.com/unix-vs-linux/

[2] https://support.apple.com/en-gb/guide/disk-utility/dsku19ed921c/mac

This article will cover the data migration process between two data stores, the evaluation of possible alternatives, and final outcome. We will cover the difference between sequential, asynchronous, multi-threaded, and distributed applications. And finally, we will overview the practical application of Helm and K8S jobs for data migration tasks.

👉 If you care only about the K8S and Helm stuff, skip to K8S jobs and Helm section. It's ok.

Introduction

Recently I was working on data migration. We had 2M records in AWS DocumentDb that we wanted to migrate to AWS RDS PostgreSQL; the actual database tech stack is irrelevant as we will not cover the application-level implementation here.

Long story short, on top of migrating the data from one source to another and adapting record schemas accordingly, it also involved data enrichment and out-of-db context processing, such as re-hosting image links.

First, we took the simplest approach: We created a Python script that migrated one record from DocumentDb to RDS. But soon enough, we realized that:

It takes ~3 seconds to migrate one record, and we have 2M records. So the expected time for complete db migration will be (2M * 3) / 60 / 60 / 24 = ~69 days. Well, we don't want to wait 69 days, so we had to come up with something with higher throughput.

In general, I always try to keep things as simple as possible and only introduce complexity when it's needed. A wise man once said,

“Premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.” (Donald E. Knuth).

So we evaluated multiple alternatives.

Alternatives

Alternative 1 - Multiple local machines

It might seem like the simplest solution. But it comes with its complexity. If something goes wrong, it will be challenging to understand the cause. Having the application distributed to individual machines means that we don't have aggregated place to look at the logs, execution stats etc. Additionally, we might run into local setup issues, slow networking, ISP issues etc. It will also be quite expensive, as we would pay a significant amount of money for the AWS VPC in/out data.

Alternative 2 - Asynchronous code

We could take advantage of asynchronous code. Our tasks are not CPU-bound; most of the time is spent on networking requests and db queries. But unfortunately, we used Django for our PostgreSQL db and wanted to reuse the modes. It turns out Django is not that friendly when it comes to async code 😢. Django ORM calls from async code need to be wrapped with sync_to_async, which is blocking, so it beats the purpose of asynchronous code.

Alternative 3 - Multi-Threading

Another valid approach to consider is multi-threading. Threads can be created based on the number of CPUs. The logic can be described as follows:

FOR c in CPUs
  skip = THREAD_NUMBER * BATCH_SIZE
  limit = BATCH_SIZE
  SPAWN thread with batch sized portion of records
JOIN all threads

Although it worked, this approach has a couple of disadvantages. First, multi-threading is hard to reason about, debug and observe in a cloud environment. In the cloud, we deal with distributed applications. We already have system components working in parallel and asynchronously; adding concurrency within these components adds another layer of complexity. I kind of reason in big O notation:

Distributed applications are of complexity D(n), where n is the number of components, adding multi-threaded applications will have complexity of D(n * t), where t is the number of threads.

If I had to choose which system to maintain, I would select D(n) over D(n * t)! Second, threading creates overhead of synchronization as well as observing the logs and dealing with data becomes much more complicated. As well as testing, testing is way harder in a multi-threaded environment! And I love testing! So why make my life harder than it should be? The answer might be subjective...

Alternative 4 - Distributed approach

Similar to the Multi-Threaded approach. But instead of having threads, we have multiple machines working together on the data migration. In this context, we're talking about K8S jobs and pods, but it can be applied to any orchestration tool.

The thinking is similar to multi-threaded:

FOR j in JOBs
  skip = JOB_NUMBER * BATCH_SIZE
  limit = BATCH_SIZE
  SPAWN K8S job with batch sized portion of records
WAIT for all created jobs to complete

Notice that the number of CPUs no longer limits us; we're limited by the number of jobs we define. I want to emphasize the WE in this sentence! It just brings me the nice feeeling of being in control, if im the one defining stuff, I know what to expect!

This definition/configuration can be infinite(ish). Of course, there's a physical limit to K8S node instances, but that's a threshold way higher than a single-pod CPU limitation. In our micro view from the db migration job, it might feel as infinite as it gets.

Going with the distributed approach

We went through the approaches and decided to go with the distributed one, as this alternative is simple enough. We have a synchronous Python script ready and tested. We need to multiply it X times and assign the data slices to K8S jobs that will run that script. As a result, we get distributed application that doesn't need synchronization or any operational overhead. Logs are easily accessible, and job resources are visible.

Batch allocation, or data source slices

As we mentioned above, we will slice the data into chunks and assign it an appropriate K8S job. This can be illustrated as follows:

So we get the gist - now we need to actually run some math:

Calculating the number of jobs and the batch size

I'm going to use javascript for that, just because I can?

function getMigrationConfig(numberOfRecords, expectedHours){
  const singleRecordMigrationTimeSeconds = 3 
  const overallExecutionSeconds = numberOfRecords * singleRecordMigrationTimeSeconds
  const overallHours = overallExecutionSeconds / 60 / 60
  const workerCount = overallHours / expectedHours
  const batchSize = numberOfRecords / workerCount
  return { workerCount, batchSize }
}

So for 2M records and 1-hour migration, we get:

> getMigrationConfig(2000000, 1)
{ workerCount: 1666.6666666666667, batchSize: 1200 }

1666 seems to be a lot of K8S pods; maybe it's okay, maybe not; it really depends on the infrastructure. How about 5-hour migration?

> getMigrationConfig(2000000, 5)
{ workerCount: 333.33333333333337, batchSize: 5999.999999999999 }

Well, you get the point. We can choose the configuration we want. The cool thing here is that I'm in control; I can go between 1 and infinity(ish). That wouldn't be possible with threads cause there's way a lower limit on an available number of cores.

This is great 🤩! We went from an initial sequential processing time of 69 days to just under a couple of hours 🚀! It's quite cool, don't you think?

K8S jobs and Helm

Ok, we got the theory and general understanding of what we want to achieve here. Now we'll get our hands dirty with K8S and Helm.

What's Helm?

Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application [1].

In our case, we will use only a subset of its functionality. We're going to define variables and enumeration on the K8S job template.

Defining our K8S job

Let's start with a simple K8S job [5] definition as if we do not intend to use Helm yet:

Note: I am using an example docker image - nginx-helloworld. This is only for illustration purposes, don't expect it to work out of the box!

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-job
  namespace: db-migration
spec:
  template:
    spec:
      containers:
      - name: db-migration
        image: nginx-helloworld
        env:
          - name: DB_MIGRATE_JOB_NUMBER
            value: 0
          - name: DB_MIGRATE_BATCH_SIZE
            value: 1000
      restartPolicy: Never
  backoffLimit: 1

Pretty standard, we defined a job manifest with a docker image reference and two environment variables: DB_MIGRATE_JOB_NUMBER and DB_MIGRATE_BATCH_SIZE.

So what would we do if we wanted to replicate this definition more than once? We could copy-paste this file multiple times, changing the job and batch variables or some kind of bash scripting.

It's definitely possible, but as one platform engineer I'm working with asked me:

What would you choose?

I choose to avoid the crap pill.

Let's see how we'll do it using Helm:

{{- range $i, $job := until (.Values.jobsCount | int) }}
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-job-{{ $i }}
  namespace: db-migration
spec:
  template:
    spec:
      containers:
      - name: db-migration
        image: nginx-helloworld
        env:
          - name: DB_MIGRATE_JOB_NUMBER
            value: {{ $i | squote }}
          - name: DB_MIGRATE_BATCH_SIZE
            value: {{ $.Values.migrationBatchSize | squote }}
      restartPolicy: Never
  backoffLimit: 1
---
{{ end }}

Helm values file content:

jobsCount: 2
migrationBatchSize: 100

So what have we done here:

1. Range loop at the top. It will create this YAML definition as the number of jobsCount in the values file.
2. Added placeholders DB_MIGRATE_BATCH_SIZE, and DB_MIGRATE_JOB_NUMBER env variables.
3. Dynamically define job name by appending the {{ i }} variable

If you dealt with templating engines, I think this should be quite simple to grasp. If not, then welcome to the future!

Running Helm

First, install Helm if needed, docs.

I have the following directory structure in my example:

$ tree
├── Chart.yaml         <- Helm chart config
├── templates
│   ├── job.yaml       <- K8S job
│   └── namespace.yaml <- K8S namespace
└── values.yaml        <- template values

Where Chart.yaml[2] content is simple Helm chart config:

apiVersion: v2
name: k8s-job
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.16.0"

And templates/namespace.yaml file:

apiVersion: v1
kind: Namespace
metadata:
  name: db-migration

Running Helm as:

$ helm template . --values ./values.yaml

Produces:

# Source: k8s-job/templates/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-job-0
  namespace: db-migration
spec:
  template:
      containers:
      - name: db-migration
        image: MY_DOCKER_IMAGE
        env:
          - name: DB_MIGRATE_JOB_NUMBER
            value: '0'
          - name: DB_MIGRATE_BATCH_SIZE
            value: '100'
---
# Source: k8s-job/templates/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-job-1
  namespace: db-migration
spec:
  template:
      containers:
      - name: db-migration
        image: MY_DOCKER_IMAGE
        env:
          - name: DB_MIGRATE_JOB_NUMBER
            value: '1'
          - name: DB_MIGRATE_BATCH_SIZE
            value: '100'

That output can be piped with kubectl command and deployed to our K8S cluster:

$ helm template . --values ./values.yaml | kubectl apply -f -
namespace/db-migration unchanged
job.batch/db-migration-job-0 created
job.batch/db-migration-job-1 created

List the jobs:

$ kubectl get jobs -n db-migration
NAME                 COMPLETIONS   DURATION   AGE
NAME                           COMPLETIONS   DURATION   AGE
db-migration-0    1/1           7m51s        7m51s
db-migration-1    0/1           7m51s      7m51s
..
db-migration-99   0/1           7m41s      7m41s
db-migration-100  0/1           7m41s      7m41s

And Pods:

$ kubectl get pods -n db-migration

AME                                 READY   STATUS      RESTARTS   AGE
db-migration-0-5k5nb    0/1     Completed   0          8m44s
db-migration-1-9splv    1/1     Running     0          8m44s
...
db-migration-99-7klz2   1/1     Running     0          8m40s
db-migration-100-nwlg9  1/1     Running     0          8m40s

K8S makes it easy to see which job was completed, which failed, and the pod logs.

Things to consider

There are a few things to consider when spawning many K8S pods:

K8S node scaling and resource allocation

Depending on the application program, pods might require specific memory or CPU resource allocation. When we talk about many jobs requesting these resources at the same time, depending on the cluster scaling strategy it might take time to actually get them. That needs to be tailored to the application, or pre-scale the nodes. We need to calculate whether our K8S cluster will handle the expected load. In our case defined as AWS node groups [3].

Ip range

It really depends on the subnet VPC CIDR block [4] configuration. For example, IPv4 CIDR of 172.20.0.0/16 will allow a 65k+ IP range but of course, it also depends on how many IPs are already in use.

Theory meets practice

K8S Control Plane

As with anything in our practice, everything works fine in our local environment but when it needs to run somewhere else... Different story. Spawning these jobs on actual K8S clusters gave us this error from the K8S control plane:

Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job" Name: "db-migration-240", Namespace: "db-migration" from server for: "STDIN": Get "...": dial tcp: lookup

That error came when we did more than 100 jobs at a time. So then we had to batch that as well!

That's it :)

Summary

We covered the thought process behind the recent data migration I had to do on AWS cloud using K8S jobs and Helm. I think Helm's simplicity and power really shines in this application.

We haven't covered the application-level logic here as it's peculiar to our case and way beyond the the scope this article.

Hope it helped anyone that has to deal with something similar task!

References

[1] https://helm.sh/

[2] https://helm.sh/docs/topics/charts/#the-chartyaml-file

[3] https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html

[4] https://docs.aws.amazon.com/vpc/latest/userguide/configure-your-vpc.html#vpc-cidr-blocks

[5] https://kubernetes.io/docs/concepts/workloads/controllers/job

Python comes with the default, ready-to-use logging module. It's kind of nice cause we don't have to install additional libraries to start logging in to our application. However, for someone coming from other language backgrounds, such as Javascript, TypeScript, Java, C#, and Go. The Python logger module might resemble familiar tools, but it has its specifics. In this article, I would like to cover some quirks that I found along the way while I was working recently on a python project.

Read the documentation

Yeah, that would be a sensible thing to do. Make a cup of coffee, relax and read the logging module documentation [1], all the 13 sub-pages, of course 🙂. Unfortunately, not everyone has the time to do so. I usually skim through the documentation, but when I deal with multiple programming languages daily, it's hard to hold all the language and framework-specifics in my head. Anyway, enough about me and my saga with Python documentation; let's look at the logger module.

Log levels

Why do we need logging levels? Good question. Because depending on how we run our application, we might want to have different types of logs. For example, if we're in the developing phase, we would like to see verbose logs of what exactly happens in the application. But when we're ready for production, we would probably want to limit our logs to only errors - otherwise, we will have a lot of logs, where most of them are probably going to be just noise. By noise, I mean they will not have useful information while simultaneously being very verbose and distracting. So we need to understand what kind of information we need in our logs. Having everything everywhere usually will not get us far.

The logger module comes with several levels[2]:

CRITICAL = 50
ERROR = 40
WARNING = 30
INFO = 20
DEBUG = 10
NOTSET = 0

I think the names are quite self-explanatory, but you can always go back to the original docs. Each level has a number assigned to it. The higher the number, the higher its priority. For example, if we set the log level to be WARNING, we will only get logs with a priority higher than WARNING, i.e. higher than 30. Similarly, if we set it to NOTSET level we will get all the logs.

Let's run an example logging python application:

import logging

logging.debug('test')
logging.info('test')
logging.warning('test')
logging.error('test')
logging.critical('test')

Expected output:

WARNING:root:test
ERROR:root:test
CRITICAL:root:test

Where did my logs go? We have only WARNING, ERROR, and CRITICAL 🤔. As we mentioned above, it's all about the level of priority set.

Let's set the log level as DEBUG:

import logging

logging.basicConfig(level=logging.DEBUG)

logging.debug('test')
logging.info('test')
logging.warning('test')
logging.error('test')
logging.critical('test')

Run it, and observe the output:

$ python main.py 
DEBUG:root:test
INFO:root:test
WARNING:root:test
ERROR:root:test
CRITICAL:root:test

As expected, we get all the logging levels 🥳.

Logging to files

Similarly to what we did above, we can log into a local file:

import logging

logging.basicConfig(filename='test.log', filemode='w')
logging.warning('test')

expected test.log file content:

WARNING:root:test

There're more configuration options available [3]

Formatting the output

When we have multiple components in the same application logging message, it's natural to want them to log in the same format; this way, we can apply the exact configuration for message content, such as process name, level, and module names across our application.

import logging

logging.basicConfig(format='%(process)d-%(levelname)s-%(message)s')
logging.error('test')

Same here. More formatting options in the docs [4]

Capturing stack traces

What happens when we have an error? What information are we expected to see? Usually, we would want a short description of what happened and a stack trace. Stack trace provides information about the execution path, how we got to that error, function names, calls and execution context.

But first, Let's see if we can guess would the following log be. We're going to raise a ZeroDivisionError exception intentionally:

import logging

try:
  1 / 0
except Exception as e:
  logging.error(e)

Looks ok, right? Well, that would just log:

$ python ./main.py 
ERROR:root:division by zero

I don't know about you, but the first time I saw this log in production, I was pretty puzzled about it. If you made it that far I salute you 🫡. That's one of the quirks I was referring to in my intro.

We need to add the exec_info=True parameter or use logging.exception function to get the stack trace information.

import logging

try:
  1 / 0
except Exception as e:
  logging.error('OOPS', exc_info=True)
  # OR: logging.exception('OOPS')

Expected output:

ERROR:root:OOPS
Traceback (most recent call last):
  File "...py", line 4, in <module>
    1 / 0
ZeroDivisionError: division by zero

Yep. We got our stack trace. Let's move on 🚀.

Module level logs, namespaces, or however you want call them

So far, we have used the default logger, aka the root logger. That's all fine, but most Python applications consist of multiple modules logging altogether. In that case, we would probably want to differentiate which logs come from which modules. I mean, why not?

Let's see an example of such a module level logs [5]:

import logging

module_logger = logging.getLogger('module_logger')
module_logger.warning('module_logger log message')

logging.warning('module_logger log message')

Expected output:

module_logger test
root test

We created a custom logger instance named module_logger. Different from the root logger, its name is not part of the output. But have no fear! We can add to our output via formatting [4]:

import logging

logging.basicConfig(format='%(name)s %(message)s',)
module_logger = logging.getLogger('module_logger')
module_logger.warning('test')
logging.warning('test')

Observe the output:

module_logger log message
WARNING:root:module_logger log message

Kind of cool. I would say 😎.

Handlers

Let's have a look now at log handlers[6]. Handlers are helpful when we want to(surprisingly) handle logs in multiple ways, for example, send the output to various destinations. Each logger might have multiple handlers.

Example:

import logging

logger = logging.getLogger(__name__)

logger.addHandler(logging.StreamHandler())
logger.addHandler(logging.FileHandler('out.log'))

logger.warning('test')
logger.error('test')

Let's run it. We should see the message both in our terminal and in the local file named: out.log. Notice that the file will not be overridden when we re-run the program; content will be appended. One common use of logging handlers is sending logs over the network. For example, log aggregation services.

Structured Logs

We covered briefly in previous sections the log formatting. That helps with log structure. But once we start configuring our logger, we might find ourselves writing and maintaining a boilerplate code. I found jsonlogger[7] very useful for JSON-structured logs. It is easy to configure, it just works. Once installed, here's an example of how to use it:

import logging
from pythonjsonlogger import jsonlogger

root_logger = logging.getLogger()
logHandler = logging.StreamHandler()
logHandler.setFormatter(jsonlogger.JsonFormatter())

root_logger.addHandler(logHandler)

root_logger.setLevel(logging.DEBUG)

logger = logging.getLogger('test')
logger.info('test')
logger.warning('test')
logger.error('test')

Expected output:

{"message": "test"}
{"message": "test"}
{"message": "test"}

JSON-structured logs are useful when processing log files for analytics or BI (business intelligence). As well as for searching. Searching and correlating log messages is very valuable when developing and troubleshooting.

Logs and Unix streams

In Unix-based systems, we have stdin, stdout, and stderr streams. Unix streams are like everything else in Unix, treated as files 🐧. What I mean is that we can read and write to streams as if we were handling files. Well, kind of.

Ok, cool. how is it relevant to this post? Let's run one of the logging applications from above, and check which stream is it using:

import logging

logger = logging.getLogger(__name__)

logger.addHandler(logging.StreamHandler())
logger.addHandler(logging.FileHandler('out.log'))

logger.warning('test')
logger.error('test')

We can run and redirect each stream to a separate file.

$ python logs.py 1> stdout.log 2> stderr.log

Here, 1 stands for stdout, 2 for stderr. We redirect stdout stream to stdout.log file and stderr to stderr.log. More info about redirecting output [8].

If we check your stderr.log file, it has all the content. How come? Was it an actual error? No? Well, that's the default behavior of the Python logging module. I found it quite surprising 😶. It's problematic cause if our application sends logs to some kind of log aggregation service such as (logstash)[https://www.elastic.co/logstash/] or (datadog)[https://www.datadoghq.com/] our logs will be identified as errors when in reality they are not.

We can add stream parameter to point to stdout:

logger.addHandler(logging.StreamHandler(stream=sys.stdout))

We should be able to locate the log content in stdout.log now.

Streaming to both stdout and stderr segregation

When our application is monitored by other services, it can be useful to have the segregation of error and other logs.

One way to achieve to define multiple handlers:

import sys
import logging
from pythonjsonlogger import jsonlogger

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
        log_record["level"] = record.levelname
        log_record["name"] = record.name
        log_record["message"] = record.getMessage()
        if record.levelname == "ERROR" or record.levelname == "EXCEPTION":
            log_record["exception"] = record.exc_info
            log_record["stacktrace"] = record

formatter = CustomJsonFormatter("%(level)s %(name)s %(message)s")

stdoutHandler = logging.StreamHandler(stream=sys.stdout)
stdoutHandler.setLevel(logging.DEBUG)
stdoutHandler.setFormatter(formatter)

stderrHandler = logging.StreamHandler(stream=sys.stderr)
stderrHandler.setLevel(logging.ERROR)
stderrHandler.setFormatter(formatter)

root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)

root_logger.addHandler(stderrHandler)
root_logger.addHandler(stdoutHandler)

However, that results in duplication of logs; the ERROR logs will result in both stderr and stdout streams. That's far from ideal.

But! We can use builtin Filter Objects. Example:

class StdoutFilter(logging.Filter):
    def filter(self, record):
        return record.levelno >= ROOT_LEVEL and record.levelno < logging.ERROR

Here we defined a class that inherits from logging.Filter. It will decide for each log record whether it should be handled by the handler or not.

import sys
import logging
from pythonjsonlogger import jsonlogger


class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(
            log_record, record, message_dict)
        log_record["level"] = record.levelname
        log_record["name"] = record.name
        log_record["message"] = record.getMessage()
        if record.levelname == "ERROR" or record.levelname == "EXCEPTION":
            log_record["exception"] = record.exc_info
            log_record["stacktrace"] = record


ROOT_LEVEL = logging.DEBUG


class StdoutFilter(logging.Filter):
    def filter(self, record):
        return record.levelno >= ROOT_LEVEL and record.levelno < logging.ERROR


class StderrFilter(logging.Filter):
    def filter(self, record):
        return record.levelno >= logging.ERROR


formatter = CustomJsonFormatter("%(level)s %(name)s %(message)s")

stdoutHandler = logging.StreamHandler(stream=sys.stdout)
stdoutHandler.setFormatter(formatter)
stdoutHandler.addFilter(StdoutFilter())

stderrHandler = logging.StreamHandler(stream=sys.stderr)
stderrHandler.setFormatter(formatter)
stderrHandler.addFilter(StderrFilter())


root_logger = logging.getLogger()
root_logger.setLevel(ROOT_LEVEL)

root_logger.addHandler(stderrHandler)
root_logger.addHandler(stdoutHandler)


logging.debug('test')
logging.info('test')
logging.warning('test')
logging.error('test')
logging.critical('test')

Run it:

$ python logs.py 1> stdout.log 2> stderr.log

And observe file content:

stdout.log:

{"level": "DEBUG", "name": "root", "message": "test"}
{"level": "INFO", "name": "root", "message": "test"}
{"level": "WARNING", "name": "root", "message": "test"}

stderr.log:

{"level": "ERROR", "name": "root", "message": "test", "exception": null, "stacktrace": "<LogRecord: root, 40, ...main.py, 53, \"test\">"}
{"level": "CRITICAL", "name": "root", "message": "test"}

Another promising library I've encountered is loguru9]. However, I couldn't figure out how to configure stream segregation there.

Summary

Python logging module should be flexible enough to fit the general purpose. First, however, we need to understand how it works, especially if we come from another programming language background. Python logging module default behavior, and the difference between root and module loggers, were alien to me. But I survived 💪. I hope it makes sense and gives a high-level overview of how to use the logging module and its configurations.

Another thing to mention is that all these logging examples could've been implemented as configuration files[10].

References

[1] https://docs.python.org/3/library/logging.html

[2] https://docs.python.org/3/library/logging.html#logging-levels

[3] https://docs.python.org/3/library/logging.html#logging.basicConfig

[4] https://docs.python.org/3/library/logging.html#formatter-objects

[5] https://docs.python.org/3/library/logging.html#module-level-functions

[6] https://docs.python.org/3/library/logging.handlers.html

[7] https://pypi.org/project/python-json-logger/

[8] https://www.gnu.org/software/bash/manual/bash.html#Redirecting-Output

[9] https://loguru.readthedocs.io/en/stable/overview.html

[10] https://docs.python.org/3.10/howto/logging-cookbook.html#custom-handling-of-levels

I will use yarn as my package manager, just cause I like it, but similar (even almost identical) can be achieved using npm.

Ok, let's go :)

Step 1. Install create-react-app

$ yarn global add create-react-app

It will install the create-react-app package globally on our machine, which means it will be accessible from every terminal [1].

Step 2. Create React app

$ create-react-app me --template typescript

This command will create the "Hello World" project with the TypeScript template. Why TypeScript? Why not? I love it!

Step 3. Install dependencies

Go into the generated folder (in our case, me) and install all the dependencies of the project:

$ cd me && yarn

Should get a file structure similar to this:

→ ls -la
total 3144
-rw-r--r--   ... .gitignore
-rw-r--r--   ... README.md
drwxr-xr-x   ... build
drwxr-xr-x   ... node_modules
-rw-r--r--   ... package-lock.json
-rw-r--r--   ... package.json
drwxr-xr-x   ... public
drwxr-xr-x   ... src
-rw-r--r--   ... tsconfig.json
-rw-r--r--   ... yarn.lock

Step 4. Build your app

Build script creates a build directory with a production build of your app.

$ yarn build

This should result in similar to the following:

→ ls -l ./build/
total 80
-rw-r--r-- ... .gitignore
-rw-r--r-- ... README.md
drwxr-xr-x ... build
drwxr-xr-x ... node_modules
-rw-r--r-- ... package-lock.json
-rw-r--r-- ... package.json
drwxr-xr-x ... public
drwxr-xr-x ... src
-rw-r--r-- ... tsconfig.json
-rw-r--r-- ... yarn.lock

Step 5. Launch our static website in the browser and fail

Navigate to your file in your browser. You can type the path to your file in the search bar something like:

file:///<YOUR MACHINE PATH HERE>/build/index.html

Or just drag and drop it into a new tab. Anything that works for you.

Now observe. It's empty. Simplistic. Minimal. Beautiful. But not quite what you wanted, innit?

The problem is that our react app is built to be served by a server.

Open up the console and observe the error logs:

Step 6. Find solutions to our problems!

Have no fear! The documentation is here! [2]

If you read the docs, its sates:

....set up your favorite HTTP server ...

But... I don't want to set up my FAVOURITE HTTP server. I'm not even sure I have one...

If you look further down you will see a section called Building for Relative Paths [3].

That's what we need!

Let's set "homepage": "./", in your package JSON:

{
  "name": "me",
  "version": "0.1.0",
  "private": true,
 ...
  "homepage":".", <---ADD THIS
 ...

Build the app again. Openindex.html again. And you should get the spinning React logo :)

That's it!

References

[1] https://classic.yarnpkg.com/lang/en/docs/cli/global/

[2] https://create-react-app.dev/docs/deployment/

[3] https://create-react-app.dev/docs/deployment/#building-for-relative-paths

In my case, I am running different compilation processes that are CPU-specific. Since my mac comes with an M1 chip, not everything is 100% compatible, especially when you go to the more niche software or when new updates break some kind of bizarre C lib dependencies 😶.

So, here’s a guide on how I installed the local VNC server that I used daily in segregation with plain ssh when CLI access is just enough. But sometimes GUI is all you need.

I will cover Linux VNC server setup and macOS client configuration, but as you probably guessed, you can also connect with not only macOS clients to the Linux VNC server.

VNC server install & config

There are several options for VNC servers; we’re going to use x11vn [1]. VNC server allows you to view and interact remotely with a machine.

Installation:

My Linux machine distro: Ubuntu 21.04

Installing VNC server and X window system:

$ sudo apt install lightdm x11vnc

You will be prompted with a message to choose a display manager. I chose lightdm [2], don't ask me why 🙃.

Try running it after the installation to check that we’re good.

$ x11vnc

If all is good, you should see something like this; if not, then go fish 🎣.

28/09/2022 17:58:12 *** XOpenDisplay failed. No -display or DISPLAY.
28/09/2022 17:58:12 *** Trying “:0” in 4 seconds. Press Ctrl-C to abort.
28/09/2022 17:58:12 *** 1 2 3 4
28/09/2022 17:58:16 *** XOpenDisplay of “:0” successful.
....
The VNC desktop is:      kimchi-machine:0
PORT=5900

This means that x11vnc could successfully identify X display :0. I guess that meant that we could try and connect to it 🤔!

But before that, let’s configure our VNC server ☝️.

Creating systemd service

It's up to you, really, you don’t have to create a service to run on the server, but if you want the VNC server to run in the background instead of launching it from your terminal every time and make sure that nothing is closed the terminal session, you do need it. And tbh its quite easy todo.

Create our vnc service config [3] — located under/lib/systemd/system let’s call it vnc.service :

$ sudo vim /lib/systemd/system/vnc.service

I’m using vim as a text editor; you can use whatever. Here’s nano command:

$ sudo nano /lib/systemd/system/vnc.service

Then enter the following config, see comments for description:

[Unit]
# service description
Description=my local x11vnc kimchi service
# start this service after:
After=display-manager.service network.target syslog.target
[Service]
# the type of the service
Type=simple
# process config
ExecStart=/usr/bin/x11vnc -forever -display :0 -auth guess -passwd kimchi
# do this on process stop
ExecStop=/usr/bin/killall x11vnc
# restart when failed
Restart=on-failure
[Install]
# start this service before multi-user target
WantedBy=multi-user.target

You might ask — whats multi-user.target? I’m not an expert, but a quick internet search revealed that in this context, it means that our service would start when the system reaches run level 2 [4]. I guess that’s cool to know 😐.

note that I set the password to be “kimchi”, you should probably change it!

starting our systemd service

Reaload systemd [5] to load our new service:

$ systemctl daemon-reload

Enable our service:

$ systemctl enable vnc.service

Start our service:

$ systemctl start vnc.service

Check our service status:

$ systemctl status x11vnc.service

Should get something like this:

If you ever want to stop our VNC service, run:

$ systemctl disable x11vnc.service
$ systemctl stop x11vnc.service

Finally, connect from your mac

First, find your VNC server machine IP. You can do it by running:

$ ifconfig

the output differs between networks, mine looks like this:

enp5s0: flags=....
        inet ....
 broadcast 192.168.88.255

So we go the IP — 192.168.88.255 💪

Next, open up your Finder and navigate to GO > Connect to Server

Enter the IP we got with the default port for VNC

IP — 192.168.88.255

Port — 5900

Protocol: VNC

All together now:

vnc://192.168.88.255:5900

You’ll be prompted with a password. In case you forgot, its “kimchi” (or whatever you set):

And Voilà

That’s it!

References

[1] https://wiki.archlinux.org/title/X11vnc

[2] https://wiki.archlinux.org/title/LightDM

[3] https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

[4] https://unix.stackexchange.com/questions/404667/systemd-service-what-is-multi-user-target

[5] https://wiki.archlinux.org/title/Systemd

Writing an Interpreter with PyPy is about creating a BF [2] interpreter and the translation process of it with the PyPy toolchain. When I followed along, the main issues for me were related to the toolchain itself. Hence I decided to centre my attention on it here. We will not cover the BF interpreter here since it's all explained in the original blog post, but I will go through the PyPy toolchain and some basic benchmarking concepts essential for understanding the topic.

What exactly is PyPy 🤔

If you tried Python before, you were probably running CPython [3]. CPython is the most common implementation of Python VM (Virtual Machine). PyPy [4] is an alternative to CPython. With PyPy, we write our programs in RPython [5], and apply the RPython translation toolchain that generates binary executable. One major advantage of PyPy over CPython is speed which will be demonstrated later in this article using the basic benchmarking tool.

Jargon summary:

CPython — The most common Python implementation

PyPy — CPython alternative

RPython — restricted version of Python

Benchmark — the act of assessing program relative performance (time in our case)

Getting hands-on with RPython 👐

First, we need to write a simple RPython program, call it python_prime.py:

import os
import sys


def prime(n):
    primes = []
    for num in range(0, n):
        if num > 1:
            for i in range(2, num):
                if (num % i) == 0:
                    break
            else:
                primes.append(num)
    return primes


def write(s):
    os.write(1, bytes(s))


def entry_point(argv):
    num = int(argv[1])
    primes = prime(num)
    write('calculated primes: \n')
    for p in primes:
        write(str(p) + ' ')
    return 0


def target(*args):
    """
    "target" returns the entry point.
    The translation process imports your module and looks for that name,
    calls it, and the function object returned is where it starts the translation.
    """
    return entry_point, None


if __name__ == '__main__':
    entry_point(sys.argv)

It definitely resembles Python 👀. That’s cause RPython is Restricted Python. We had to choose some kind of computational problem, so we went with a prime number sequence generator. The actual compute work is not the point here 🙂.

Our RPython program also has some PyPy hooks — a function called target (the comment should be self-explanatory).

Running the program 🏃

We will use the Python 2.7.18 version since I found incompatibility issues with higher versions. If you don’t have the 2.7 Python version installed, you can use of the tools such as pyenv [6] or conda [7].

Run our program:

$ python ./python_prime.py 10

We should get similar output to this:

Benchmarking 🏋️

Now when we have our program in our hands, we can do some benchmarking to understand its runtime performance. We will use a tool called hyperfine [8] for that.

Mac installation:

$ brew install hyperfine

Linux Debian Installation:

$ wget https://github.com/sharkdp/hyperfine/releases/download/v1.15.0/hyperfine_1.15.0_amd64.deb 
$ sudo dpkg -i hyperfine_1.15.0_amd64.deb

Similarly to how we run our RPython program, we can run it through the hyperfine utility:

$ hyperfine 'python ./python_prime.py 10000'

Hyperfine runs our program ten times by default and gives us stats such as minimum and maximum execution times. We're not going to cover hyperfine in detail here; the important thing is to get a feel for it and a high-level understanding when comparing runtimes.

Generally, benchmarking results might be affected by different factors, such as the machine you run on, other processes that run simultaneously as you run benchmarking, hot vs cold starts, caching, etc. But, for our purpose, this benchmarking process we have here is good enough.

Translating our program using PyPy 🤓

The first step would be to get the PyPy source code:

$ wget https://downloads.python.org/pypy/pypy2.7-v7.3.9-src.tar.bz2 
$ tar -xvf pypy2.7-v7.3.9-src.tar.bz2 # extracting 
$ tar mv pypy2.7-v7.3.9-src ./pypy

Here we're downloading the source code, extracting it and renaming the extracted directory for later convenience. Once downloaded and extracted, run the PyPy translation toolchain:

$ export PYTHONPATH=${PWD}:${PWD}/pypy/ 
$ python ./pypy/rpython/translator/goal/translate.py python_prime.py

If you compare the tutorial code to mine, you will notice the difference in the file paths - translate.py was moved from pypy/translator/goal/translate.py to rpython/translator/goal/translate.py.

Also, I set the PYTHONPATH [9] environment variable to the current directory and the path to the PyPy source code. That would make the PyPy modules available for our program to run.

When running it, you should see toolchain logs on your terminal (it takes around 30 seconds on my M1 chip); when the process is complete, we should get the translated binary with the same name as the python file but with "-c" postfix.

$ ls -lah ./python_prime-c 
-rwxr-xr-x 1 medium medium 217K 24 Sep 11:34 ./python_plain_class-c

Let's have a look inside 👀

$ file ./python_plain_class.py ./python_plain_class-c 
python_plain_class.py: Python script text executable, ASCII text python_plain_class-c: Mach-O 64-bit executable x86_64

In case you don't trust the file command:

$ head ./python_plain_class-c
������ H__PAGEZEROx__TEXT__text__TEXT0/��0/�__stubs__TEXT��>��__stub_helper__TEXT�"��__cstring__TEXT2�q2�__const__TEXT�����__unwind_info__TEXTD��D�__eh_frame__TEXT8��...

Yep, it's indeed a binary file 🙂. Ok, so we translated Python using PyPy to a binary; now what?

Compare benchmarks 🔬

Run hyperfine on our translated program:

$ hyperfine './python_prime-c 10000'

Should get something like this output:

Spot the difference? We can also just run hyperfine to compare both programs:

$ hyperfine './python_prime-c 10000' 'python ./python_prime.py 10000'

See full result:

Say What!? 72 times faster 🤯?!

Well yeah, apparently 😶. Hyperfine won't lie to you. You can trust it☝️.

Did you actually expect that ?? Maybe I should have named this post - "Run your python programs X70 faster"? I think that's really cool… Ok, moving along 🙃.

Let's run hyperfine again, this time with worm-up and more runs:

$ hyperfine --warmup 10 './python_prime-c 10000' 'python ./python_prime.py 10000' \
&& hyperfine -m 100 -M 100 './python_prime-c 10000' 'python ./python_prime.py 10000'

We executed ten warmups before running the benchmark, then ran our programs 100 times each and recorded their execution times. The result is not far from the previous one; it's exactly the same metrics since the benchmark might have been skewed a bit as I'm not running it in a lab!

Summary ✏️

So what have we done? We built a simple RPython program, translated it using the PyPy toolchain, and witnessed substantial performance improvement with our eyes when comparing both program execution times side by side using hyperfine.

PyPy seems like a good choice for writing computational-intensive applications, especially if you want to stay in Python land. However, if you try it for a while, you will see that it might be less convenient than working with CPython; you might experience some obscure type annotation errors or unsupported function call exceptions. As with everything in life, it has its advantages and drawbacks.

I know, I know. Who cares about Python 2.7?! At the time of the article, we're already at Python 3.10.7. Why are we bothering with Python 2.7? It also reached the end of support cause you missed it!

The main reason is that I was following the article from 2011, and guess what? There was no Python version 3.10.7 in 2011. By the way, I bet you can use newer versions, as I've seen on the official download page. I just haven't tried them yet. I was excited when I heard about PyPy (almost a decade since I wrote my first Python language). Even if I'm not going to use it in production yet, just knowing that it's practically feasible to perform such optimisation is eye-opening.

What's Next ❓️

Next, we will look at PyPy jit trace logs and perform further optimisations now in our RPython applications.

References

[1] https://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html

[2] https://en.wikipedia.org/wiki/Brainfuck

[3] https://en.wikipedia.org/wiki/CPython

[4] https://www.pypy.org/

[5] https://doc.pypy.org/en/latest/coding-guide.html#restricted-python

[6] https://github.com/pyenv/pyenv

[7] https://docs.conda.io/projects/conda/en/latest

[8] https://github.com/sharkdp/hyperfine

[9] https://bic-berkeley.github.io/psych-214-fall-2016/using_pythonpath.html