Python features I didn't knew about

Today I saw some code that showed me, that I not noticed some cool features since I joined the python universe.

Enum/Flags

Python 3.4 introduces a new module called enum. Which provides Enum and Flag creation.

Enum example

from enum import Enum, auto
    
class Color(Enum):
    RED = auto()
    GREEN = auto()
    BLUE = auto()

Flag example

from enum import Flag, auto
class Color(Flag):
    RED = auto()
    BLUE = auto()
    GREEN = auto()
    BLACK = auto()
    WHITE = RED | BLUE | GREEN
    
print(Color.RED in Color.WHITE) # True
print((Color.RED | Color.BLUE) in Color.WHITE) # True
print(Color.BLACK in Color.WHITE) # False

Zipapp

Python 3.5 introduces a new module zipapp which provides a simple way to create a zipped executable from python code.

Pygopar.com provides a small example.

Because I had some problems with building a “fat” executable, I build a small script to pack my example app.

To support execution with:

  • dependencies
  • usage of your own code from __main__.py

__main__.py can just use absolute imports like from ownmodule import hello

Project structure
root
|-- ownmodule
|   |-- __init__.py
|   |-- __main__.py # some startup code
|   `-- hello.py # some code imported in __main__.py
|-- requirements.txt # list of dependencies
`-- zip_it.sh
zip_it.sh
#!/usr/bin/env bash

rm -rf ./build # clear build dir
mkdir ./build # create build dir

# install dependencies, but do not use os specific binary or compile to those
pip3 install --no-compile -U --no-binary all -r ./requirements.txt -t build

# copy own code into build
cp -R ./ownmodule ./build/

# move __main__.py to ./build/ which will be executed
mv ./build/ownmodule/__main__.py ./build/__main__.py

# create the executable archive
python3 -m zipapp build -o ownmodule.pyz -p "/usr/bin/env python3"

Now you can execute your project with ./ownmodule.pyz

If you face some issues with pip not able to install properly, use a virtualenv. That will solve the problem.

String interpolation

Python 3.6 introduces formatted string literals. Which supports string interpolation.

a = "Hello"
b= "World"

print(f"{a} {b}") # Hello World

Secrets

Python 3.6 introduces a new module secrets which provides strong random numbers generation for cryptography.

Python Basic Log Config

I already faced a lot of times where I searched for a kind of normal logging behavior for a python project/script. Here is my snippet.

Behavior:

  • Log levels <= INFO on stdout
  • Log levels > INFO on stderr

Code or Gist

import logging
import sys


def configure(logger: logging.Logger = logging.root) -> logging.Logger:
    class InfoFilter(logging.Filter):
        def filter(self, rec):
            return rec.levelno in (logging.DEBUG, logging.INFO)

    formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(name)s - %(message)s", "%d/%m/%Y %H:%M:%S")

    std_out_handler = logging.StreamHandler(sys.stdout)
    std_out_handler.setLevel(logging.DEBUG)
    std_out_handler.setFormatter(formatter)
    std_out_handler.addFilter(InfoFilter())

    std_err_handler = logging.StreamHandler()
    std_err_handler.setLevel(logging.WARNING)
    std_err_handler.setFormatter(formatter)

    logger.addHandler(std_out_handler)
    logger.addHandler(std_err_handler)

    return logger

Spring Security Overview

Spring Security

This post is focusing on Spring security, especialy in case of Spring Boots Autoconfiguration.

Please check also Spring Security Guide which explains nearly everything on its own.

Here I want to write down a small overview to keep everything in mind what I read.

Getting started

Setup a Spring Boot project with Spring Initializr. Add Web and Security to generate your project.

Should look like this:

.
├── build.gradle
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
└── src
    ├── main
    │   ├── java
    │   │   └── com
    │   │       └── example
    │   │           └── DemoApplication.java
    │   └── resources
    │       ├── application.properties
    │       ├── static
    │       └── templates
    └── test
        └── java
            └── com
                └── example
                    └── DemoApplicationTests.java

Now you can start testing all different configurations. See next, what is already configured.

I have done nothing - Default Configuration

If you just add Spring Security to your boot application, the autoconfiguration will protect your application with basic authentication.

This is achieved through an AuthenticationManager which uses an in InMemoryUserDetailsManager providing a default user called user with a random password. To get more context about the different components read Central Components

For more information check out Spring Boot Documentation

Central Components

WebSecurityConfigurerAdapter

WebSecurityConfigurerAdapter is used to configure the Spring Security Setup.

@Configuration
public static class ExampleConfig extends WebSecurityConfigurerAdapter {
    @Override
    public void configure(WebSecurity web) throws Exception {
    }

    @Override
    protected void configure(AuthenticationManagerBuilder auth) throws Exception {
        super.configure(auth);
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        super.configure(http);
    }
}
  • configure(WebSecurity web):
    Gets an shared WebSecurity object which is used to build the FilterChainProxy.
  • configure(AuthenticationManagerBuilder auth):
    Configures an AuthenticationManagerBuilder which will be used to build a local AuthenticationManager.
  • configure(HttpSecurity http):
    Configure web based security at resource level. This includes on which URL the security matches and which roles are allowed to access a resource.

FilterChainProxy

The FilterChainProxy is a delegating filter. It delegates to the most precision match of HttpSecurity matcher. For more information check out the mentioned Spring Security Guide

AuthenticationProcessingFilter

AuthenticationProcessingFilter is responsible to extract principal (username) and credential (password) from request and create an unauthoriced authentication (also called as authentication token).

The unauthenticated/ checking principal and credential will be done by the AuthenticationManager.

Authentication

Stores information:

  • authenticated (true/false)
  • authorities (List of GrantedAuthority)
  • principal/credentials (username/password)

Spring provides some general implementations like UsernamePasswordAuthenticationToken.

Authentication Manager

package org.springframework.security.authentication;

import org.springframework.security.core.Authentication;
import org.springframework.security.core.AuthenticationException;

public interface AuthenticationManager {
	Authentication authenticate(Authentication authentication) throws AuthenticationException;
}

AuthenticationManager gets unauthenticated authentications and checks principal and credentials. Without any configuration Spring will use a ProviderManager, which is an implementation of an AuthenticationManager, but delegates incoming authenticate() calls to AuthenticationProviders.

Global vs local

There exists a global parent AuthenticationManager that all local configured ones are connected to. Local configuration is handled through WebSecurityConfigurerAdapter that are most of the time responsible for just a subcontext of the url context path.

The ProviderManager will call its parent to authenticate, if no AuthenticationProvider authenticates the request.

Authentication Provider

package org.springframework.security.authentication;

import org.springframework.security.core.Authentication;
import org.springframework.security.core.AuthenticationException;

public interface AuthenticationProvider {	
	Authentication authenticate(Authentication authentication) throws AuthenticationException;

	boolean supports(Class<?> authentication);
}

AuthenticationProviders are checking if the given unauthoriced Authentication is valid and returns an authoriced Authentication. If the Developer defines one AuthenticationProvider, it will be picked up and injected into the ProviderManager automatically. The Developer can also provide multiple AuthenticationProvider but those will not be picked up automaticcally, thay has to get configured through a config class that is annotated with @EnableWebSecurity, @EnableGlobalMethodSecurity, or `@EnableGlobalAuthentication. (For the global configuration.)

Referring to the Spring Docs

If you don’t want to use global configuration, you are able to configure that in your WebSecurityConfigurerAdapter.

@EnableWebSecurity
public class SecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(AuthenticationManagerBuilder auth) throws Exception {
        auth
                .authenticationProvider(new TestingAuthenticationProvider())
                .authenticationProvider(new TestingAuthenticationProvider());
    }
}

UserDetailsService

package org.springframework.security.core.userdetails;

public interface UserDetailsService {
	UserDetails loadUserByUsername(String username) throws UsernameNotFoundException;
}

The UserDetailsService is used to provide the simplest customication. If just a UserDetailsService is provided by the developer spring will use it in a DaoAuthenticationProvider, which is loading the “UserDetails” by username from the UserDetailsService and compares the credentials with these from authentication.

There are also some UserDetailsServices which are provided by Spring for common szenarios like for JDBC

GrantedAuthority

package org.springframework.security.core;

import java.io.Serializable;

import org.springframework.security.access.AccessDecisionManager;

public interface GrantedAuthority extends Serializable {
    String getAuthority();
}

Represents an authority granted to an Authentication object.

Updates I

Introduction

After a while I think I had found enough interesting tools and projects to write about. Each following topic introduce one tool/project. Most of the time I quote from the linked web page a short description.

So lets start with the first one.


Flask

Github

Flask is a microframework for Python based on Werkzeug and Jinja2. It’s intended for getting started very quickly and was developed with best intentions in mind.

I already wrote that I started to learn Python on my own. Now I found this nice web framework, which makes it easy to start writing your own web app (in python, obvious)

Flask is surrounded by a mass of really good documentation and tutorials. Follow the links and get in touch.

Docs and Tutorials


Virtual env

I thought I do not need it, but after a while it is much nicer to split up python project setups. In Flask Docs I found a short explenation, how to use it. That was enough for me to directly start with it. In combination with Autoenv, you never have to thing about your setup again.

Flask Virtual Env Doc


Autoenv

Github

If a directory contains a .env file, it will automatically be executed when you cd into it.

This tool does a lot of magic I already searched for. It supports you with setting environment variables or do anything you need, if you cd into a project folder. You are able to create a “.env” file which will become executed each time you cd into the containing folder.


Jetbrains Toolbox App

Blog post

When I program a language, than normally I use one of the Jetbrains IDEs to have as much comfort as I could get. Even this block post is written using PyCharm. Not because I program any Python code. It just contains the best editor with multi line selection I ever saw.

The Toolbox App give you a overview over all of your Jetbrains IDEs installed and the containing projects.


Supervisor

Ubuntu/Debian process manager

Supervisor is a process manager which makes managing a number of long-running programs a trivial task by providing a consistent interface through which they can be monitored and controlled.

Just saw it, looks like easy to use for auto starting apps on a device.

Python Crossmodule Variables

Hello, I started a while ago a private python project. After some time of finding my way through the new language and the gui framework tkinter, I found an eventbus library.

I started to refactor my project to separate some stuff into modules and to think about a valuable use of the found library. But where to store the central eventbus? It should be available from every module, so let us try some stuff.

Example code

Module A
class Wrapper:
    def __init__(self, value):
        self.value = value

test_string = "A"
test_integer = 1
int_wrapper = Wrapper(1)

Module B
from A import test_string, test_integer, int_wrapper

test_string = "B"
test_integer = 2
int_wrapper.value = 2


def printContent():
    print("B-String:" + test_string)
    print("B-Integer:" + str(test_integer))
    print("B-Object:" + str(int_wrapper.value))

Module C

from A import test_string, test_integer, int_wrapper
from B import printContent

printContent()
print("C-String:" + test_string)
print("C-Integer:" + str(test_integer))
print("C-Object:" + str(int_wrapper.value))

Execute C
python C.py
# B-String:B
# B-Integer:2
# B-Object:2
# C-String:A
# C-Integer:1
# C-Object:2

What happened?

We execute C and C imports variables from A and a function from B. B imports variables from A and changes them.

What I would expect first is, that A is loaded, than B last C. A defines the variables, B changes them and C shows the same outcome like B.

BUT the variables test_string and test_integer from A are imported by value, because strings and integer are immutable.

This is not the same for objects, objects are used with references. So if you want a global eventbus for all your modules it is no problem to store it in its own package like the “int_wrapper”.