Informatics on BlocNotes

UBports part 1 - Compiling and booting for Samsung S7

Fri, 29 Jan 2021 00:00:00 +0000

I went down the hole of installing Linux on my phone (Samsung Galaxy S7). The easiest path is compiling and installing UBports, some people have heavily started the process but it is still a work in progress.

This is the first part: step by step description to build and boot the system, with helpers scripts to setup the system once flashed.

UBports screenshot

The first part is my notes gathered during the initial build. You can install TWRP then jump directly to the step by step at the end of this page.

State

This is the status of working features with my current build, some others are more advanced.

[ X ] Wi-Fi
[ X ] Web browsing
[ X ] MP4 playback
Bluetooth
Camera
Sim card
Phone calls
SMS
MMS

Documentation

TWRP installation

Just follow official instructions https://forum.xda-developers.com/t/recovery-exynos-official-twrp-for-galaxy-s7-herolte.3333770/

Issue “Custom binary blocked by FRP lock”

Go to Android developer options, activate enable OEM unlock slider.

Halium compilation

Under Debian Bullseye it is not in the docs and might be obvious, but ADB must be installed apt install adb.

First steps

Set up your build device

Package python-markdown does not exists, to be replaced by python3-markdown

The error E: Unable to locate package repo is because contrib repo needs to be activated in /etc/apt/sources.list.

Get Halium source

Adding device-specific source

android_device_[manufacturer]_[device]=android_device_samsung_herolte lineage.dependencies file on Github

File halium/devices/manifests/[manufacturer]_[device].xmlalready exists: halium/devices/manifests/samsung_herolte.xml

Vendor blobs

Nothing to do? At least I did not for now.

Build Halium

Modify the kernel configuration

Clone mer-kernel-check outside of BUILDDIR . Then the command for Samsung S7 kernel is:

 ./mer_verify_kernel_config ../halium/kernel/samsung/universal8890/arch/arm64/configs/exynos8890-herolte_defconfig

Options I had to set:


# https://cateee.net/lkddb/web-lkddb/QUOTA.html
# If you say Y here, you will be able to set per-user limits for disk usage
# It does not work for me, the kernel is not able to mount partitions without.
CONFIG_QUOTA_NETLINK_INTERFACE=y

CONFIG_QFMT_V2=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_IP_NF_MATCH_RPFILTER=y

Patch against 1046b86e869d6a77bc1f905b3a61bbd5e5478ee1: TODO

Include your device in fixup-mountpoints

Nothing to do, already included.

Building the system.img and hybris-boot.img

mka hybris-boot

Error:

/usr/bin/ld: scripts/dtc/dtc-parser.tab.o:(.bss+0x50): multiple definition of `yylloc'; scripts/dtc/dtc-lexer.lex.o:(.bss+0x0): first defined here

Due to a redundant declaration. Let’s make it extern in

./kernel/samsung/universal8890/scripts/dtc/dtc-lexer.l
./kernel/samsung/universal8890/scripts/dtc/dtc-parser.tab.c_shipped

Error:

mkbootimg: command not found

Add it to the system path after compiling it.

export PATH=./system/core/mkbootimg/:$PATH

Halium reference rootfs

Flash Halium

Restart it download mode, flash hybris boot:

heimdall flash --BOOT out/target/product/herolte/hybris-boot.img

Install Halium rootfs and system

./halium-install -p halium ../halium-rootfs-20170630-151006.tar.gz ../halium/out/target/product/herolte/system.img

I gave up Halium rootfs

But I did not succeed to make Halium rootfs boot: I had a constant reboot loop. When I asked on Halium’s Telegram channel someone told me that it is a 4 years old rootfs and I should move on UBports rootfs, so I did :-)

UBports

UBports is a touch-friendly Ubuntu flavor maintained by a community, since Canonical dropped the Edge project.

Building the images

Fix mount points

No edit necessary to the fstab file, found at:

device/samsung/hero-common/ramdisk/fstab.samsungexynos8890

Fix defconfig

Command:

./halium/halium-boot/check-kernel-config kernel/samsung/universal8890/arch/arm64/configs/exynos8890-herolte_defconfig -w

First time I ran the command:

Config file checked, found no errors.

 Made 110 fixes.

Run it again:

Config file checked, found no errors.

 Made 1 fixes.

But the lineCONFIG_DEFAULT_SECURITY is set, but to "selinux" y not "apparmor".is never fixed. Fix it manually with CONFIG_DEFAULT_SECURITY="apparmor" then run it again without the -w argument:

Config file checked, found no errors.

Ubuntu Touch requires setting console=tty0

console=tty0 is already defined in the kernel config file, so nothing required

Fix Apparmor

I did not try to understand why, but it is recommended in Github issue by and hoster in UBports Github, so let’s do it:

cd kernel/samsung/universal8890/security/
rm -r apparmor/
wget  https://raw.githubusercontent.com/ubports/apparmor-backports-ut/master/apparmor-3.18.tar.bz2
tar -xf apparmor-3.18.tar.bz2

Build halium-boot.img and system.img

Build as instructed

Installing UBports

As instructed in tge UBPort FAQ, use xenial-hybris-edge-rootfs-arm64 from https://ci.ubports.com/job/xenial-hybris-edge-rootfs-arm64

wget https://ci.ubports.com/job/xenial-hybris-edge-rootfs-arm64/lastStableBuild/artifact/out/ubuntu-touch-hybris-xenial-edge-arm64-rootfs.tar.gz

Flash boot image il download mode (phone off, long press on power+home+volume down)

./Heimdall/build/bin/heimdall flash --BOOT files/halium-boot.img

Reboot in recovery, flash system.img+rootfs

/halium-install -p ut ../files/ubuntu-touch-hybris-xenial-edge-arm64-rootfs.tar.gz ../files/system.img

Halium logging in

Wi-fi

As described nmtui then select Activate a connection.

UBpots - The graphical UI

udev rules

sudo -i
cat /var/lib/lxc/android/rootfs/ueventd*.rc|grep ^/dev|sed -e 's/^\/dev\///'|awk '{printf "ACTION==\"add\", KERNEL==\"%s\", OWNER=\"%s\", GROUP=\"%s\", MODE=\"%s\"\n",$1,$3,$4,$2}' | sed -e 's/\r//' >/usr/lib/lxc-android-config/70-herolte.rules

Edit layout (From Github)

Edit /etc/ubuntu-touch-session.d/android.conf to add

GRID_UNIT_PX=29
QTWEBKIT_DPR=3.0
NATIVE_ORENTATION=portrait
FORM_FACTOR=handset

Add user to update

Step described in Github issue, why?

sudo adduser --force-badname --system --home /nonexistent --no-create-home --quiet _apt || true

Install xenial fix dataspace

Again, from Github. Probably linked to the new user creation to update packages

sudo ubports-qa install xenial_-_fix-dataspace
sudo ubports-qa remove xenial_-_fix-dataspace
sudo reboot

Step by step with to flash system.img and halium-boot.img and boot

This step by step guide is working for me, if you have an issue please read the previous chapter which might help to understand it. In all cases please leave a comment so I can fix the instructions.

Backup

Backup your user data if you want. This process will also create a TWRP backup for Android itself.

Install TWRP

Follow official instructions https://forum.xda-developers.com/t/recovery-exynos-official-twrp-for-galaxy-s7-herolte.3333770/

Setup UBports

Reboot in recovery

Reboot in recovery, it can be done with adb reboot recovery or switch off the phone then power + volume down + home keys.

TWRP backup (optional)

If you want to be able to easily switch back to Android backup:

boot
system
data

Format data partition

In TWRP:

Touch Wipe
Touch Format Data
Enter yes
Go back to home menu thanks to the home button

Flash the boot and file system to the device

Still in TWRP:

Clone Halium install.
Download UBports rootfs.
With sudo, or as root: ./halium-install -v -p ut ./path/to/ubports-touch.rootfs-xenial-armhf.tar.gz ./path/to/system.img Note add -i if your want to have your id_rsa.pub key copied
Enter the password for the phone’s new user.
When finished, then reboot in download mode. In TWRP:
1. Touch reboot
2. Touch Download
Download and compile Heimdall.
Once in Download mode, flash the bootloader with heimdall flash --BOOT ./path/to/files/halium-boot.img
The phone will reboot, Unity will appear, the screen goes back, the phone restarts then Unity hangs. It is expected, then
Enter ip address show in a terminal, it should make display a new interface. Mine returns:

9: enp0s20u2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

So the new interface name is enp0s20u2

Download helpers files from Github.
Run the script to setup the connection, send required files to the phone and connect to it: bash helper-install-host.sh interface_name. Ex: bash helper-install-host.sh enp0s20u2
You should obtain a terminal on: phablet@ubuntu-phablet:~$
Setup Wi-Fi thanks to:
1. Enter nmtui
2. Select Activate a connection
3. Select your SSID
4. Enter your Wi-Fi password
5. Push tab to select <Quit> then enter to exit
Start the configuration script thanks to bash helper-install-phone.sh
Answer y when prompted (twice)

The phone should reboot, then you are in! :-)

Wireguard and Pi-hole in Docker containers

Wed, 18 Nov 2020 21:32:24 +0000

I had to move my previous Wireguard VPN + Pi-hole ad blocker to another server, but this time I was not able to expose Pi-hole DNS port (53) to the host machine. It was also better to have Wireguard VPN inside a Docker container… so I did!

As a reminder, Wireguard is a stateless and easy to configure VPN: share a pair of public keys between the client(s) and server then you are good to go! Moreover, stateless is great when used from a phone as there is no power-hungry keep-alive like OpenVPN, nor reconnection time when switching from Wi-FI to 4G.

DNS address issue

The main issue I had is the way to provide Pi-hole address to Wireguard container: docker-compose does not yet accept do translate a container name in dns section. The easiest workaround is to set a fixed IP for the Pi-hole container, not pretty, but hey… it works :-)Then both services must be in the same docker-compose.yml file to avoid any issue related to a network race condition when the docker daemon restarts.

Pi-hole

Pi-Hole configuration is straightforward and well documented on their Docker Hub page.

Wireguard

Instead of creating my own Dockerfile I used the image from linuxserver.io, which has been beautifully implemented. Then follow the documentation. Thanks to Wireguard’s QR Code feature, the phone setup is dumb-proof: just scan it from the container’s logs.

docker-compose file

The resulting docker-compose file is:

version: "3.5"

services:
  wireguard:
    image: linuxserver/wireguard
    depends_on:
      - pihole
    dns:
      - 172.29.0.2
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped
    volumes:
      - ../../data/wireguard:/config
      - /lib/modules:/lib/modules
    ports:
      - 51820:51820/udp
    environment:
      - TZ=Europe/Paris
      - SERVERURL=host_server.yourdomain.com
      - SERVERPORT=1194
      - PEERS=Android_phone
    networks:
      - network-pihole

  pihole:
    image: pihole/pihole:latest
    volumes:
      - ../../data/pi-hole/etc/:/etc/pihole/
      - ../../data/pi-hole/dnsmasq.d:/etc/dnsmasq.d
    environment:
      TZ: "Europe/Paris"
      PROXY_LOCATION: pihole
      VIRTUAL_HOST: pihole.yourdomain.com
      VIRTUAL_PORT: 80
      LETSENCRYPT_EMAIL: email@yourdomain.com
      LETSENCRYPT_HOST: pihole.yourdomain.com
    restart: unless-stopped
    networks:
      network-pihole:
        ipv4_address: 172.29.0.2

networks:
  network-pihole:
    name: "dns-pihole"

How to compile KiCad 5.1.x for Debian 10

Tue, 13 Aug 2019 19:39:24 +0000

I was looking to use InteractiveHtmlBom plugin for KiCad but unfortunately, Debian maintainers did not set KICAD_SCRIPTING option in the official package. After a few failing attempts I even tested the official flatpak package but it is an old version (currently 4.x) and scripting is not enabled too.

So I dug further until it actually succeeds to compile :-)

Pellet stove

Here is the thing: KiCad official documentation still recommend packages for python2, and v5.1.4 is not compatible with new versions of libglm-dev if compiled with GCC; this is now fixed in master. So the commands I used are:

# From http://kicad-pcb.org/download/debian/ without Python2 packages
apt install cmake doxygen libboost-context-dev \
  libboost-dev libboost-system-dev libboost-test-dev \
  libcairo2-dev libcurl4-openssl-dev libgl1-mesa-dev \
  libglew-dev libglm-dev libngspice-dev \
  liboce-foundation-dev liboce-ocaf-dev libssl-dev \
  libwxbase3.0-dev libwxgtk3.0-dev swig wx-common

# Add Debian Buster/Python3 required packages
apt install python3-wxgtk4.0 libwxgtk3.0-gtk3-dev \
  libboost-filesystem-dev python3-dev

# Add KiCad libraries
apt install kicad-libraries kicad-packages3d --no-install-recommends

# Downgrade libdlm-dev - For 5.1.4 or older only
apt remove libglm-dev
wget http://ftp.us.debian.org/debian/pool/main/g/glm/libglm-dev_0.9.8.3-3_all.deb
dpkg -i libglm-dev_0.9.8.3-3_all.deb

# Clone and build KiCad
git clone https://git.launchpad.net/kicad
mkdir -p kicad/build/release
cd kicad/build/release
cmake -DCMAKE_BUILD_TYPE=Release \
  -DKICAD_SCRIPTING_ACTION_MENU=ON \
  -DwxWidgets_CONFIG_OPTIONS="--toolkit=gtk3" \
  -DKICAD_SCRIPTING_WXPYTHON_PHOENIX=ON \
  -DKICAD_SCRIPTING_PYTHON3=ON \
  -DwxWidgets_CONFIG_EXECUTABLE='/usr/bin/wx-config' \
  ../..
make -j
make install

Now you can use InteractiveHtmlBom which is really pleasant for big boards!

Thinkpad T440p & fingerprint sensor under Linux

Fri, 10 May 2019 18:38:19 +0000

There are a lot of pages stating that the Thinkpad T440p fingerprint sensor is not working well under Linux, but those are old and outdated. For me it worked well after figured out which packets to install:

apt-get install libfprint fprintd libfprintd

The package fprint-demo can also be installed to play with the sensor.

Then to enable login with the sensor in Gnome3, navigate to

Users
Select Enable
Enroll you finger.

Tested under Debian Buster Buster, fingerprint sensor model is vfs5011.

Setup Pi-Hole & WireGuard VPN on smartphone

Tue, 25 Dec 2018 12:46:52 +0000

WireGuard is a new VPN software, which is described as It aims to be faster, simpler, leaner, and more useful than IPSec, while avoiding the massive headache and using state-of-the-art cryptography. Pi-hole is a black hole for Internet advertisements, ie a server blocking advertisements at DNS level.

I have Pi-hole on my home network but wanted to have it also as DNS server on my phone when I am not at home. As I have a bad DSL connection, I can not rely on my home server. So I put Pi-hole on a VPS server I rent: it works, flawlessly.
Nevertheless, it was an open DNS resolver, which can be used to amplify DDoS attacks; there is already 2212 unsecured Pi-hole on the Internet, so I do not want to add another one.

Pi-hole provides documentation to use OpenVPN, even to redirect only DNS requests but it is not efficient on a phone as it is not a stateless connection, so it consumes battery; moreover, GSM/Wifi switch is not really handled, you have to reconnect. The solution comes from WireGuard which is stateless, and provides Android & iOS apps. I struggle to configure everything, so here is what I did:

Configuration

Pi-Hole in Docker container

Pi-holes provides a [convenient Docker]https://hub.docker.com/r/pihole/pihole/) container so it is easy to build a Docker-compose file on top.

Here is an example, as there are already other Docker services serving on ports 80 and 443 through jwilder container providing reverse proxy, so I add this container to the same network. Pi-hole also recommends to have it as default on the machine in order to replace adds with white pages, it helps to keep pages layout, so jwilder’s Docker environment variable DEFAULT_HOST should be set to Pi-hole’s VIRTUAL_HOST value. If you do not use jwilder container just delete VIRTUAL_HOST, VIRTUAL_PORT, and networks related options.

version: "3.5"

services:
  pihole:
    image: pihole/pihole:latest
    dns:
      - 127.0.0.1
      - 1.1.1.1
    ports:
      - "127.0.0.1:53:53/tcp"
      - "127.0.0.1:53:53/udp"
      # Adapt to WireGuard interface
      - "192.168.2.1:53:53/tcp"
      - "192.168.2.1:53:53/udp"
    volumes:
      # Adapt to folders
      - ../../data/pi-hole/etc:/etc/pihole/
      - ../../data/pi-hole/dnsmasq.d:/etc/dnsmasq.d
    networks:
      - network-proxy
    environment:
      # Adapt to server IP address
      ServerIP: X.X.X.X
      PROXY_LOCATION: pihole
      # Adapt to domain
      VIRTUAL_HOST: pihole.YOURDOMAIN.ext
      VIRTUAL_PORT: 80
    extra_hosts:
      # Resolve to nothing domains (terminate connection)
      #  'nw2master.bioware.com nwn2.master.gamespy.com:0.0.0.0'
      # LAN hostnames for other docker containers using jwilder
      - "pihole pihole.yourDomain.lan:5.135.166.67"
    restart: unless-stopped

networks:
  network-proxy:
    name: "https-proxy"

WireGuard

Server

WireGuard configuration was more difficult. My server is running Debian 9 where Debian systemd documentation can not be used as the version is too old, and I never found out why regular documentation was failing with wg-quick.

So I had to use the old /etc/network/interfaces file. Replace eth0 with your “main” interface.

auto wg0
iface wg0 inet static
        address 192.168.200.1
        netmask 255.255.255.0
        pre-up ip link add $IFACE type wireguard
        pre-up wg setconf $IFACE /etc/wireguard/$IFACE.conf
        pre-up iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE;
        post-down iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE;
        post-down ip link del $IFACE

Now we have to configure WireGuard itself in /etc/wireguard/wg0.conf, where wg0 is WireGuard interface name:

[Interface]
ListenPort = 8124
PrivateKey = REPLACE_WITH_YOUR_PRIVATE_KEY

[Peer]
PublicKey = REPLACE_WITH_CLIENT_PUBLIC_KEY_GENERATED_AFTER
AllowedIPs = 0.0.0.0/0

Server keys are generated as per documentation: wg genkey | tee privatekey | wg pubkey > publickey Then route should be added: ip route add 192.168.2.2/32 dev wg0

Android client

Generate public/private key thanks to GENERATE button
Interface->Addresses is client IP address. ex: 192.168.200.2
Interface->Listen port is the same than wg0.conf. ex: 8124
Interface->DNS servers is your server WireGuard IP. ex: 192.168.200.1
Peer->Public key is the server’s public key
Allowed IP should be serverIp/24 but I do not know why it forwards only DNS queries to the server. Putting 0.0.0.0/0 makes it work, I have to figure out why.
Endpoint is server public IP address:port. ex: X.X.X.X:8124

Linux/Gnome - Associate an application with a filetype

Fri, 05 Jan 2018 20:36:57 +0000

I lost some time figuring out how to associate “.stl” extensions with Slic3r. Gnome wrote a good documentation but it is not obvious to find. I did not use the package manager for that because I wanted the up to date AppImage.

Create /usr/share/mime/packages/slic3r.xml

<?xml version="1.0"?>
<mime-info xmlns='http://www.freedesktop.org/standards/shared-mime-info'>
  <mime-type type="application/octet-stream">
    <glob pattern="*.stl"/>
  </mime-type>
</mime-info>

Create /usr/share/applications/slic3r.desktop

[Desktop Entry]
Version=1.0
Type=Application
Name=Slic3r
Icon=/opt/AppImages/icons/Slic3r.ico
Exec=Slic3r
Path=/opt/AppImages/
NoDisplay=false
Categories=3DGraphics;Utility;
StartupNotify=false
Terminal=false
MimeType=application/octet-stream

How to setup an OpenWrt repository

Thu, 13 Jul 2017 18:29:37 +0000

I recently wanted to setup a repository for some custom OpenWrt packages for Respeaker. It seems that it is not so well documented, or at least, I did not found it and I have been helped on OpenWrt forum. Also, it is better to sign packages so…here we go!

Prepare the repository

Compile and install usign from here.
Generate key thanks to: usign -G -p openWrtUsign.pub -s openWrtUsign.key.
Put the following script in the IPK directory.
Make it executable: chmod +x repoUpdate.sh
Execute it: ./repoUpdate.sh
Copy the whole directory and openWrtUsign.pub to a webserver.
Done!

Add key and install packages on the target

# Add the public key
wget http://youserver.ext/path/to/openWrtUsign.pub
opkg-key add openWrtUsign.pub
opkg update

# Install package
opkg install newPackage

Manifest generation script

#!/bin/bash

SCRIPT="../../../../scripts/ipkg-make-index.sh"
KEY="../openWrtUsign.key"

# Generates package manifest
$SCRIPT . 2>/dev/null > Packages.manifest
grep -vE '^(Maintainer|LicenseFiles|Source|Require)' Packages.manifest > Packages
gzip -9nc Packages > Packages.gz
usign -S -m Packages -s $KEY

ioPush-App

Wed, 27 Jan 2016 00:00:00 +0000

This web service is an IoT logbook and Android push notification server, it can be considered like Twitter but for the Internet of Things. It is a part of my home automation system: I previously used Twitter as a logbook of warning/errors but it lacks some functionalities, and for few months notifications did not work well on my phone.

Website screenshot

ioPush messages

Android notification

ioPush Android notification

So it should be able to

Keep record of events - Done
Send event to push message if asked - Done
Manage users - Done
Send push notifications to Android, maybe other OS - Done, Android app to be published
Android application -> receive notifications - Tested in the 1st version
Android application -> display user’s events

What can be logged?

Sensors out of battery warning
Pollution data over a threshold
Your 3D printer has finished its job
Server down
Motion sensor triggered
A new mail is inside your mailbox
Hoarfrost alert
Whatever you want to be written :)

Useful links

HAProxy - How to use Let's Encrypt certificates

Sat, 19 Dec 2015 14:32:00 +0000

Thanks to Let’s Encrypt program, we can now generate free SSL certificates. I used the beta program with success, but now I want to use it on another server running through HAProxy. This is a trick to combine certificate, chain, and private key to be loaded by HAProxy :-)

cat cert.pem chain.pem privkey.pem > combined.pem

Following is the command I used to generate the certificates, which can be improved to set email address and accept conditions.

./letsencrypt-auto certonly -a webroot --webroot-path /var/www/ -d domain.net -d www.domain.net

Flask-Security - How to extend a form and its validator

Thu, 17 Dec 2015 13:53:00 +0000

I am currently building a Python application, which let me use:

Flask - a simple and very well documented web Framework
Flask-Security - an extension dedicated to user management.
While it is a really great extension, there is some documentation missing or quite hard to find on search engines or StackOverflow.

How to add fields to a Flask’s security form, with a custom validator.

First, we need to declare a new class, which extends RegisterForm, then add the new fields we want on the form. If we want a custom validator for that field, we need to define it, but, to avoid losing Flask-Security validator, just call it with Form.validate(self) The last thing needed is to tell Flask-Security to use our form when instantiating it.

Instantiation

security = Security(app, user_datastore, register_form=ExtendedRegisterForm)

Form.py

from flask.ext.wtf import Form
from wtforms import StringField, BooleanField
from wtforms.validators import DataRequired
from flask_security.forms import RegisterForm
from .models import User


class ExtendedRegisterForm(RegisterForm):
    """ Add nickname field to the register's class
    """
    nickname = StringField('Nickname', [DataRequired()])

    def validate(self):
        """ Add nickname validation

            :return: True is the form is valid
        """
        # Use standard validator
        validation = Form.validate(self)
        if not validation:
            return False

        # Check if nickname already exists
        user = User.query.filter_by(
            nickname=self.nickname.data).first()
        if user is not None:
            # Text displayed to the user
            self.nickname.errors.append('Nickname already exists')
            return False

        return True

And you are done !