BlocNotes

Fix Home-Assistant statistics and energy erroneous values

Tue, 30 Sep 2025 00:00:00 +0000

Introduction

I have been experimenting with solar panels, Home Assistant and my electricity meter lately. This French meter from the utility is called Linky. It provides a small power supply (130 mW) and a serial link hidden via amplitude modulation.
To avoid using batteries, I built a prototype that harvests this power, demodulates the serial link, and feeds the data into an nRF52 development kit. It works well enough that I never bothered turning this messy setup into a proper PCB.

However, every now and then—despite CRC checks—I get an erroneous reading that messes up the Home Assistant Energy panel. Here’s how to fix incorrect statistics in the Home Assistant database.

Preparation

Before editing the database, it is strongly recommended to stop Home Assistant to avoid concurrent access that could corrupt the database.

Connect to the system running Home Assistant
Install SQLite3 if needed. For example, on Debian-based systems: apt install sqlite3
Locate the home-assistant_v2.db database file in Home-Assistant data folder and create a backup

Open the database for editing

Open the database using SQLite and set a more readable output mode:

$ sqlite3 home-assistant_v2.db`
# Then enter the commands
sqlite> .header on
sqlite> .mode column

Find the data id

Identify Home Assitant’s internal identifier, it is called metadata_id. Replace easf02 with Home-Assistant’s Entity ID you are looking for.
Using LIKE '%PARTIAL_ENTITY_STRING% allows for easier search by matching any string containing PARTIAL_ENTITY_STRING

sqlite> SELECT id, statistic_id FROM statistics_meta WHERE statistic_id LIKE '%easf01%';
id  statistic_id            
--  ------------------------
6   sensor.linky_easf01     
15  sensor.linky_easf01_cost

In this case id = 6 is the one we want to fix, and id = 15 is the associated cost.

Search for erroneous data

We let’s ask the database for any sudden jump in the data, e.g change greater than 10,000 Wh. Adjust metadata_id = 6 and the threshold (> 10000) as needed.

WITH ordered_stats AS (
  SELECT
    id,
    created_ts,
    sum,
    metadata_id,
    LAG(sum) OVER (PARTITION BY metadata_id ORDER BY created_ts) AS previous_sum,
    LAG(created_ts) OVER (PARTITION BY metadata_id ORDER BY created_ts) AS previous_created_ts
  FROM statistics
  WHERE metadata_id = 6
)
SELECT
  id,
  datetime(created_ts, 'unixepoch', 'localtime') as date,
  ABS(sum - previous_sum) AS difference
FROM ordered_stats
WHERE ABS(sum - previous_sum) > 10000 ORDER BY id DESC LIMIT 5;

In a nutshell, this query:

Selects statistics data for the entity and associates each value with the previous one.
Displays the five most recent entries where the difference between consecutive values exceeds the threshold.

Example result:

id      date                 difference
------  -------------------  ----------
229351  2025-08-18 14:00:10  20676793.0
38711   2025-01-28 12:00:10  20020401.0

Looks like we found the 20.6 MWh spike reported in the Energy panel :-)

Update the value

Since Home Assistant records cumulative sums, we need to subtract the erroneous difference from all entries starting from the anomaly. Use the following query:

UPDATE statistics SET sum = sum - 20676793.0 WHERE metadata_id = 6 and id >= 229351;

Replace 20676793.0 and 229351 with your actual values.

Once done, exit SQLite by pressing Ctrl + D. The database is now fixed, and the energy panel displays accurate values!

Fixed Home-Assistant energy panel

Reverse engineer Cubii elliptic bike BLE protocol

Wed, 13 Sep 2023 00:00:00 +0000

Introduction

I recently bought a Cubii under desk elliptical bike, I have two use cases for it:

Physiotherapy after a foot surgery
Help me to warm-up in my cold home office in the winter

Added bonus, doing light exercise. I definitely not find it as enjoyable as using a full size machine, or doing “real” sports, but this is just my preference and I am stuck at home with crutches anyway 😅

So my second hand model is a Cubii pro: no screen to display statistics, but with Bluetooth connectivity, which means that I had to install yet another proprietary app. I doubt usefulness of collecting those data but as it is BLE communications I wondered if it would be easy to reverse engineer the protocol by spying on the frames…
It appeared to be even easier to look into the Android application to find everything I need to dump the data into InfluxDB, and a fun weekend project.

Decompiling the Android application

I never decompiled any Android application but I knew that it was not hard, especially as Java is interpreted and not a compiled language. A quick search pointed me to Jadx, which did all the magic to extract the code from the Android APK to human readable Java files. Then it took only few minutes to get to the interesting code:

Calories conversion

From the class CubiiBleManager:

calories = this.this$0.getCalories(this.$revolutions, (latestCubiiNoLive == null || (cubiiResistance = latestCubiiNoLive.getCubiiResistance()) == null) ? 1 : cubiiResistance.intValue());
distance = this.this$0.getDistance(this.$revolutions);

public final double getCalories(int i, int i2) {
    // Oliv': i=revolutions, i2=resistance selector
    double d = 0.717d;
    switch (i2) {
        case 2:
            d = 0.733d;
            break;
        case 3:
            d = 0.767d;
            break;
        case 4:
            d = 0.833d;
            break;
        case 5:
            d = 0.9d;
            break;
        case 6:
            d = 1.033d;
            break;
        case 7:
            d = 1.267d;
            break;
        case 8:
            d = 1.483d;
            break;
    }
    return i * 0.03d * d;
}

public final double getDistance(int i) {
    // Oliv': i=revolutions
    return i * 1.93E-4d;
}

BLE UUID

From BLEConstants:

Name	UUID	BLE defined
RPM	00001731-2927-efef-cdab-eba0fe583711	No
Battery	00001732-2927-efef-cdab-eba0fe583711	No
Break position	00001733-2927-efef-cdab-eba0fe583711	No
Revolutions	00001734-2927-efef-cdab-eba0fe583711	No
Device info	0000180a-0000-1000-8000-00805f9b34fb	Yes
Software version	00002a28-0000-1000-8000-00805f9b34fb	Yes
Hardware version	00002a27-0000-1000-8000-00805f9b34fb	Yes
Cubii BLE service	00001730-2927-efef-cdab-eba0fe583711	No
Cubii motorized BLE service	6e400001-b5a3-f393-e0a9-e50e9ecadc24	No
Cubii + BLE service	6e400001-b5a3-f393-e0a9-e50e24dcca9e	No
Cubii+ OTA BLe service	0000fe59-0000-1000-8000-00805f9b34fb	Yes

Read BLE value from broadcast

From broadcastUpdate() in file BLEService:

GetIntValue() documentation, constant list here:

Type	Value
Float	52
Sfloat	50
Sint16	34
Sint32	36
Sint8	33
Uint16	18
Uint32	20
Uint8	17

Integer revolutions = bluetoothGattCharacteristic.getIntValue(18, 0);
Integer rpm = bluetoothGattCharacteristic.getIntValue((bluetoothGattCharacteristic.getProperties() & 1) == 0 ? 17 : 18, 0);
Integer battery = bluetoothGattCharacteristic.getIntValue((bluetoothGattCharacteristic.getProperties() & 1) == 0 ? 17 : 18, 0);
// Hardware revision
byte[] value = bluetoothGattCharacteristic.getValue();
// Software revision
byte[] value2 = bluetoothGattCharacteristic.getValue();

Read value from BLE service

From file BLEService:

    private final void readRPM() {
        BluetoothGatt bluetoothGatt = this.mBluetoothGatt;
        BluetoothGattService service = bluetoothGatt == null ? null : bluetoothGatt.getService(UUID.fromString(BLEConstants.CUBII_BLE_SERVICE));
        if (service == null) {
            Logger.INSTANCE.d(this.TAG, "Cubii service not found!");
            return;
        }
        BluetoothGattCharacteristic characteristic = service.getCharacteristic(BLEConstants.Companion.getRPM());
        if (characteristic == null) {
            Logger.INSTANCE.d(this.TAG, "Battery level not found!");
        } else {
            readCharacteristic(characteristic);
        }
    }

Easter egg

There is a small copy/paste typo in BLEService. If the RPM BLE service cannot be found the warning is about the battery :-)

    BluetoothGattCharacteristic characteristic = service.getCharacteristic(BLEConstants.Companion.getRPM());
    if (characteristic == null) {
        Logger.INSTANCE.d(this.TAG, "Battery level not found!");
    } else {
        readCharacteristic(characteristic);
    }

BLE connection mode

Cubii is using connected BLE mode, then either notify you with new data or accumulate them between two consecutive read operations.

Reading data and sending them to InfluxDB

Thanks to Python, the libraries Bleak (BLE) and InfluxDB it was fairly easy:

Connect to the device
Subscribe to notifications
Accumulate/average data for 15 seconds
Write them to InfluxDB
In case of disconnection, restart in 1.

Cubii is sending the data every 3 seconds but I am sending them only every 15 seconds to avoid extra disk space. It could easily be down sampled inside InfluxDB thanks to tasks but yeah, it was not really required in the beginning for the small amount of data, and all the actions are done in the Python code, easier to manager for such small project

Dashboard

As expected I have no real use of the data but it was fun, so I draw the data in Grafana:

Grafana dashboard with the collected data

Code

The Python script is hosted on Gitlab.

Tools

jadx: Decompile and transform Dex to Java
Apktool: Decompile/recompile APK
BLE assigned numbers

Reverse engineer a wireless temperature sensor - Rika pellet stove

Sun, 04 Dec 2022 00:00:00 +0000

I own a Rika Filo pellet stove, which I like, but it is on the expensive side of the market and the Wi-Fi module to remote control it was a no-go. This article describes the way I reverse-engineered the radio protocol.
As it is my sole heating system and winter is coming, I wanted a non invasive approach to avoid breaking anything, so I hooked up a logic analyzer to record the commands sent by the micro-controller to the radio chip, which helped me to learn the RF parameters + actual payload.

1. Hardware

The wireless temperature sensor is really simple:

Battery holder - with polarity inversion protection :-)
ATmega48 micro-controller
TI CC1101 radio transceiver
Temperature sensor in a SOT-23 package?

Fortunately, the ATmega is using a programming port called “ISP”, which shares the SPI port, and there is a nice 0.1" 2x3 header footprint on the PCB. I soldered the pin headers and easily used them to connect a logic analyzer, in order to spy the commands sent by the ATmega. The only missing signal was the chip select, which I found on a pull-up resistor between the batteries.

2. Understanding the SPI commands

I initially used LogicAnalyzer by El Dr. Gusman: a nice project based on Raspberry Pi Pico RP2040, which I recommend. Then I lent it to a friend so I finished with my work’s Saleae device.
As Dr. Gusman’s logic analyzer can export to Sigrok I wanted to use the integrated CC1101 decoder to figure out the commands sent by the device. It did not work, I tried to debug it, figured out that the bug was already fixed, compiled the latest Sigrok then… the decoder was not able to decode my frames for a reason I did not understood. I gave up and wrote a quick and dirty parser which, in the end, would have been a faster solution :).

Fill TX buffer and send data

3. Radio parameters

Decoding the SPI commands gave the following parameters:

Modulation: FSK
Manchester encoding: No
Forward Error Correction (FEC): No
Carrier frequency: 868.317 MHz
Frequency deviation: 19042 Hz
IF frequency: 304.687 kHz
Bandwidth: 203.125 kHz
Datarate: 115051 bauds
Sync mode: 30/32 bits
Preamble length: 32 bits
Sync word: 0x4904
Whitening: Yes (Seed: 0x01FF)
FSK header: implicit/fixed length, ie address and size are not added by the chip
CRC: Yes (16 bits. polynomial: XX, init: XXX)
Transmitted packet length: 5 bytes
Bit shift between preamble and sync word (not in SPI recording, I discovered it later…)

Despite the Sigrok CC1101 protocol analyzer issue, it was not that hard. I had everything I needed to simulate the Rika temperature sensor, so I took a Semtech LR1110 eval board I had laying on my desk, set the right parameters and the payload, flashed the eval board and… it did not work :(

4. Radio analysis

I doubled check each parameter, they where correct. What could be the issue then? Time to look at the FSK signal itself: I used an SDR stick with the wonderful Universal Radio Hacker software to record the transmitted payload and try to understand the issue.

4.1 Preamble

Small detail, the preamble was shifted by one because the TI chip is transmitting 0xAA, but the Semtech chip is transmitting 0x55. Fortunately, the LR1110 allows to select the number of bits of sync word to transmit, so I can added the missing ‘0’ bit before my sync word and set its length to 33 instead of 32.

4.2 What the…Whitening?

Now the difficult part: the preamble and the sync word are the same between both records, but not the payload and the CRC. It has to be a whitening issue! Both chips are using a PN9 algorithm, and I set in the Semtech chip the same init than TI is using: 0x01FF. What next? It took me a while to understand that the CC1101 starts the whitening on the most significant bit of each byte, while LR1110 starts the whitening on the least significant bit!

Whitening: Also called scrambling, is to avoid long series of the same bit. Alternate bits helps the receiver to avoid desynchronization as TX and RX clock do not perfectly match.

4.3 Wrapping-up

Knowing this I was able to implement the whitening in software, and this time, it worked! The pellet stove was receiving my packets :-)

Note: I also had to implement the CRC16 as the whitening is applied to it:

FSK payload, from Semtech SX126x datasheet

5 Payload analysis

5.1 Header

The payload is surprising, in FSK we explicit header can often see preamble | syncword | payload length | address | payload |. In our case, the DIP switch on the board allows to change the second byte, which corresponds to the address but the first byte is always 0x11, so it might actually be a custom Rika header…

=> header is one byte, 0x11

5.2 Address

The second byte is an address directly correlated to the first three DIP switches, 1 is the LSB, and 3 is the MSB:

| Switch #3 | Switch #2 | Switch #1 | Address value |
+ --------- + --------- + --------- + ------------- +
|    Off    |    Off    |    Off    |      0x00     |
|    ...    |    ...    |    ...    |       ...     |
|    On     |    On     |    Off    |      0x06     |
|    On     |    On     |    On     |      0x07     |

Note on switch #4: as written in the documentation it is selecting power output of 1 dBm (Off) or 10 dBm (On).

5.3 Battery

The last byte was more difficult to figure out and I must admit that I decided to not bother with it until I found Hublol’s github project when writing this article, who already did some reverse engineering. Even if the radio parameters are not fully described on Github he found that this byte was the battery. I do not plan to use it, but for the completeness of the reverse engineering I hooked my lab power supply and recorded the value transmitted when I swiped the voltage from 1.8 V to 3 V:

Graph - Sent value = f(supply voltage)

| Supply voltage (V) | Value sent | Trendline |
+ ------------------ + ---------- + --------- +
|         3.0        |     215    |    213    |
|         2.8        |     193    |    193    |
|         2.6        |     172    |    172    |
|         2.5        |     159    |    162    |
|         2.3        |     143    |    142    |
|         2.1        |     118    |    122    |
|         1.8        |      97    |     62    |

=> The function encoded_value = voltage_V * 100.02 - 87.556

5.4 Temperature

I eventually figured out that the temperature is encoded in the first two bytes, it is weird as I was not able to find a “usual” way to encode the temperature… I then swiped the values with steps of 256 and read the temperature displayed by the stove: the conversion is not linear!
As we usually heat around 19°C, I did a linear trendline between 15 and 22°C where I want the highest precision: the room never reached more than 22°C thanks to the stove, and the temperature accuracy below 15°C is not important as the stove need to heat at full power

Graph - temperature = f(sent value)

| Value | Displayed        |           Hublol         |         Trendline        |
|  Hex  | temperature (°C) | temperature (°C) | delta | temperature (°C) | delta |
+ ----- + ---------------- + ---------------- + ----- + ---------------- + ----- +
| 4080  |       2.0        |        -0.80     |  2.8  |         2.4      | -0.4  |
| 4500  |      10.2        |         8.20     |  2.0  |        10.2      |    0  |
| 4600  |      11.9        |        10.20     |  1.7  |        11.9      |    0  |
| 4700  |      13.7        |        12.20     |  1.5  |        13.7      |    0  |
| 4800  |      15.4        |        14.20     |  1.2  |        15.4      | -0.1  |
| 4900  |      17.2        |        16.20     |  1.0  |        17.2      |    0  |
| 4A00  |      18.8        |        18.20     |  0.6  |        18.9      | -0.1  |
| 4B00  |      20.5        |        20.20     |  0.3  |        20.7      | -0.1  |
| 4C00  |      22.4        |        22.20     |  0.2  |        22.4      |    0  |
| 4D00  |      24.0        |        24.20     | -0.2  |        24.1      | -0.1  |
| 4E00  |      25.6        |        26.20     | -0.6  |        25.9      | -0.3  |
| 4EAD  |      26.8        |        27.55     | -0.8  |        27.0      | -0.2  |

=> The function is encoded_value = temperature_celsius * 0.0068 - 109.91.

6. Code

The library to create the payload, as well as the whitening and CRC, is on available on Gitlab.

7. Conclusion

I am now able to replace Rika’s temperature sensor, so I will be able to create a “smart thermostat” and remotely control my stove to rise the temperature when I am returning from vacation, or prevent its start when the sun will heat the house in the next hour or so. Next step is to create a dedicated board to be able to control the remote from LoRaWAN.

I tested it against Rika pellet stove Filo and Sumo, but I am surprised that the temperature encoding differs so much from Hublol’s findings; he might have a more recent stove where they changed the electronics, I will ask him…

LoRaWAN on RAK3172 thanks to ST CubeMX, with VS Code

Thu, 06 Jan 2022 00:00:00 +0000

Kongduino kindly sent me a WisDuo RAK3272s few weeks ago, it is a a breakout board for RAK nice module (RAK3172) around the STM32WL5 system on chip from ST. Let’s see how to use STM32 CubeMX to generate the initialization files and the LoRaWAN stack, then setup Visual Studio code debugger.

1. STM32 CubeMX

Download STM32 CubeMX from ST website, install it as instructed.
Open it then create a new project thanks to ACCESS TO MCU SELECTOR. In the Part Number searchbox look for STM32WLE5CC then select STM32WLE5CCUx and Start Project. CubeMX will now create empty project with only the RF and oscillator pins initialized plus some core peripherals, we have to select the features we want:

1.1 Pinout & configuration tab

Analog -> ADC

Check Vrefint Channel, it will be used by the demo application.

Timers -> RTC

Check Activate Clock Source and Activate calendar. Add the following values in configuration section, tab User constants:

RTC_PREDIV_A: ((1<<(15-RTC_N_PREDIV_S))-1)
RTC_N_PREDIV_S: 10
RTC_PREDIV_S: ((1<<RTC_N_PREDIV_S)-1)

In Parameters Settings tab:

Bin mode: Free running binary mode
Asynchronous predivider value: RTC_PREDIV_A

Connectivity -> SUBGHZ

Check Activated to use the LoRa IP.

Connectivity -> USART2

To send traces (printf) from the STM32 to the computer.

Select Mode to Asynchronous
In Configuration section, tab DMA Settings:
1. Click on Add
2. In DMA request select USART2 TX
3. In Channel select DMA 1 channel 5
4. In Direction select Memory to peripheral
5. In Priority select Low
Tab GPIO Settings should already have:
- PA2 as USART2_TX
- PA3 as USART2_RX

Middleware -> LoRaWAN

Check enabled to add the LoRaWAN stack to the project then in the configuration section, tab LoRaWAN Middleware:

lorawan_conf: select your region
Tx Rfo Config: CONF RFO HP

In tab LoraWAN Application:

Application: End Node skeleton
LoRa App->Active region: Select your region

In tab Platform Settings:

Name	IPs or Components	Found Solutions
ADC	ADC: Vrefint channel	ADC
USART	USART: Asynchronous	USART2
RTC	RTC: RTC Enabled	RTC

Note: If you do not know which EUI/Key to use follow the chapter TTN OTAA Device Registration in RAK’s quickstart.

In tab LoRaWAN commissioning:

Check Static Device Eui
Fill LoRaWAN device eui, App/JoinEUI and Application key

System core -> NVIC

Check SUBGHZ Radio Interrupt.

Pin configuration

On the STM32 picture click on:

Pin PB0: select VDDTCXO
Pin PC14: select RCC_OSC32_IN
Pin PC15: select RCC_OSC32_OUT

1.2 Clock Configuration tab

In RTC Clock Mux select LSE input
MSI RC can be pushed to 48000

1.3 Project Manager tab

Set Project name, then in Toolchain/IDE select Makefile.
The project is ready, click on GENERATE CODE in top right corner.

When generated you can open the folder in your favorite editor, I will use VS Code.

2. Configure the C code

Now the program skeleton is generated but it can not be compiled yet, otherwise the compiler with output warnings like:

#warning user to provide its board definitions pins

We have to provide some specifics about the radio pins. Download radio_board_if_c.patch and radio_board_if_h.patch, place them in the root directory of your project then:

patch -u LoRaWAN/Target/radio_board_if.c -i radio_board_if_c.patch
patch -u LoRaWAN/Target/radio_board_if.h -i radio_board_if_h.patch

3. Compile the binary

Download arm-none-eabi toolchain, add it to your path then it is as simple as make -j.

4. Flash the microcontroller

RAK official documentation is using a serial<>USB converter and the integrated STM32 serial bootloader to flash the firmware, but I prefer an ST-Link as I will also be able to debug the code.

4.1 ST-Link pinout and wiring

You can use an ST-Link or any Nucleo header as long as it is broke from the target board or the jumpers ST-LINK are removed. Then the connector SWD/CN4 must be wired to the RAK module, Nucleo’s serial<>USB converter can be used thanks to CN3:

Signal name	Nucleo header	RAK3272s
VDD_ref (optional)	CN4->1	3V3
SWCLK	CN4->2	SWCLK-PA14
GND	CN4->3	GND
SWDIO	CN4->4	SWDIO-PA13
nRST	CN4->5	RST
UART_TX	CN3->1	UART2_TX
UART_RX	CN3->2	UART2_RX
Power Supply (3.3V)	CN1->1	3V3

4.2 Use ST-Link USB mass storage

You can flash the microcontroler by copy/pasting the .bin file in the ST Link folder.

4.3 OpenOCD

Required only if you want to use the debugger. As of december 2021 most OpenOCD releases does not yet include STM32WL support, it has to be compiled from the sources. The steps are easy for Linux:

git clone https://git.code.sf.net/p/openocd/code openocd-code
apt install make libtool pkg-config autoconf automake texinfo libusb-1.0-0 libusb-1.0-0-dev
./bootstrap
./configure --enable-stlink
make
make install

4.5 VS Code

Install the extension Cortex-Debug if you want to use the debugger, then edit the launch.json file, or create it if required, to have

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Cortex Debug",
            "cwd": "${workspaceRoot}",
            "executable": "./build/${workspaceFolderBasename}.elf",
            "request": "launch",
            "type": "cortex-debug",
            "servertype": "openocd",
            "configFiles": [
                "interface/stlink.cfg",
                "target/stm32wlx.cfg"
            ],
        }
    ]
}

5. LoRaWAN uplinks

TTN uplinks in the console

If you have any issue the project is available on Gitlab. The project also contains a populated SendTxData() function to actually send uplinks :)

6. Next steps

Now that we are connected to the network the next post should be about optimizing power consumption and enter in stop mode to only consume µAmps when not sending data.

Update iTead Sonoff Zigbee 3.0 USB Dongle Plus

Sun, 12 Dec 2021 00:00:00 +0000

I recently bought a Sonoff Zigbee 3.0 USB Dongle Plus from iTead to use it with Home Assistant.
While it worked out of the box, the update itself was not complicated but you have to find the right tools…

Update the firmware

The nice Python script cc2538-bsl can be used, but the branch feature/ITead_Sonoff_Zigbee-delay needs to be selected until the pull request is merged.

Install cc2538-bsl

git clone https://github.com/JelmerT/cc2538-bsl.git
cd cc2538-bsl
git checkout feature/ITead_Sonoff_Zigbee-delay
python3 -m venv .venv
source .venv/bin/activate
pip install wheel
python setup.py install
pip install intelhex python-magic

Flash the dongle

Download the firmware from Koenkk’s Z-Stack:

Then use the command

python cc2538-bsl.py -e -v -w --bootloader-sonoff-usb ./CC1352P2_CC2652P_launchpad_coordinator_20211207.hex

Which should output:

sonoff
Opening port /dev/ttyUSB0, baud 500000
Reading data from ./CC1352P2_CC2652P_launchpad_coordinator_20211207.hex
Firmware file: Intel Hex
Connecting to target...
CC1350 PG2.0 (7x7mm): 352KB Flash, 20KB SRAM, CCFG.BL_CONFIG at 0x00057FD8
Primary IEEE Address: 00:12:4B:00:25:6C:59:51
    Performing mass erase
Erasing all main bank flash sectors
    Erase done
Writing 360448 bytes starting at address 0x0000000
Write 104 bytes at 0x00057F980
    Write done                                
Verifying by comparing CRC32 calculations.
    Verified (match: 0x6dca550e)

After that you can use the nice Zigbee Home Automation integration to easily setup the Zigbee dongle.

UBports part 1 - Compiling and booting for Samsung S7

Fri, 29 Jan 2021 00:00:00 +0000

I went down the hole of installing Linux on my phone (Samsung Galaxy S7). The easiest path is compiling and installing UBports, some people have heavily started the process but it is still a work in progress.

This is the first part: step by step description to build and boot the system, with helpers scripts to setup the system once flashed.

UBports screenshot

The first part is my notes gathered during the initial build. You can install TWRP then jump directly to the step by step at the end of this page.

State

This is the status of working features with my current build, some others are more advanced.

[ X ] Wi-Fi
[ X ] Web browsing
[ X ] MP4 playback
Bluetooth
Camera
Sim card
Phone calls
SMS
MMS

Documentation

TWRP installation

Just follow official instructions https://forum.xda-developers.com/t/recovery-exynos-official-twrp-for-galaxy-s7-herolte.3333770/

Issue “Custom binary blocked by FRP lock”

Go to Android developer options, activate enable OEM unlock slider.

Halium compilation

Under Debian Bullseye it is not in the docs and might be obvious, but ADB must be installed apt install adb.

First steps

Set up your build device

Package python-markdown does not exists, to be replaced by python3-markdown

The error E: Unable to locate package repo is because contrib repo needs to be activated in /etc/apt/sources.list.

Get Halium source

Adding device-specific source

android_device_[manufacturer]_[device]=android_device_samsung_herolte lineage.dependencies file on Github

File halium/devices/manifests/[manufacturer]_[device].xmlalready exists: halium/devices/manifests/samsung_herolte.xml

Vendor blobs

Nothing to do? At least I did not for now.

Build Halium

Modify the kernel configuration

Clone mer-kernel-check outside of BUILDDIR . Then the command for Samsung S7 kernel is:

 ./mer_verify_kernel_config ../halium/kernel/samsung/universal8890/arch/arm64/configs/exynos8890-herolte_defconfig

Options I had to set:


# https://cateee.net/lkddb/web-lkddb/QUOTA.html
# If you say Y here, you will be able to set per-user limits for disk usage
# It does not work for me, the kernel is not able to mount partitions without.
CONFIG_QUOTA_NETLINK_INTERFACE=y

CONFIG_QFMT_V2=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_IP_NF_MATCH_RPFILTER=y

Patch against 1046b86e869d6a77bc1f905b3a61bbd5e5478ee1: TODO

Include your device in fixup-mountpoints

Nothing to do, already included.

Building the system.img and hybris-boot.img

mka hybris-boot

Error:

/usr/bin/ld: scripts/dtc/dtc-parser.tab.o:(.bss+0x50): multiple definition of `yylloc'; scripts/dtc/dtc-lexer.lex.o:(.bss+0x0): first defined here

Due to a redundant declaration. Let’s make it extern in

./kernel/samsung/universal8890/scripts/dtc/dtc-lexer.l
./kernel/samsung/universal8890/scripts/dtc/dtc-parser.tab.c_shipped

Error:

mkbootimg: command not found

Add it to the system path after compiling it.

export PATH=./system/core/mkbootimg/:$PATH

Halium reference rootfs

Flash Halium

Restart it download mode, flash hybris boot:

heimdall flash --BOOT out/target/product/herolte/hybris-boot.img

Install Halium rootfs and system

./halium-install -p halium ../halium-rootfs-20170630-151006.tar.gz ../halium/out/target/product/herolte/system.img

I gave up Halium rootfs

But I did not succeed to make Halium rootfs boot: I had a constant reboot loop. When I asked on Halium’s Telegram channel someone told me that it is a 4 years old rootfs and I should move on UBports rootfs, so I did :-)

UBports

UBports is a touch-friendly Ubuntu flavor maintained by a community, since Canonical dropped the Edge project.

Building the images

Fix mount points

No edit necessary to the fstab file, found at:

device/samsung/hero-common/ramdisk/fstab.samsungexynos8890

Fix defconfig

Command:

./halium/halium-boot/check-kernel-config kernel/samsung/universal8890/arch/arm64/configs/exynos8890-herolte_defconfig -w

First time I ran the command:

Config file checked, found no errors.

 Made 110 fixes.

Run it again:

Config file checked, found no errors.

 Made 1 fixes.

But the lineCONFIG_DEFAULT_SECURITY is set, but to "selinux" y not "apparmor".is never fixed. Fix it manually with CONFIG_DEFAULT_SECURITY="apparmor" then run it again without the -w argument:

Config file checked, found no errors.

Ubuntu Touch requires setting console=tty0

console=tty0 is already defined in the kernel config file, so nothing required

Fix Apparmor

I did not try to understand why, but it is recommended in Github issue by and hoster in UBports Github, so let’s do it:

cd kernel/samsung/universal8890/security/
rm -r apparmor/
wget  https://raw.githubusercontent.com/ubports/apparmor-backports-ut/master/apparmor-3.18.tar.bz2
tar -xf apparmor-3.18.tar.bz2

Build halium-boot.img and system.img

Build as instructed

Installing UBports

As instructed in tge UBPort FAQ, use xenial-hybris-edge-rootfs-arm64 from https://ci.ubports.com/job/xenial-hybris-edge-rootfs-arm64

wget https://ci.ubports.com/job/xenial-hybris-edge-rootfs-arm64/lastStableBuild/artifact/out/ubuntu-touch-hybris-xenial-edge-arm64-rootfs.tar.gz

Flash boot image il download mode (phone off, long press on power+home+volume down)

./Heimdall/build/bin/heimdall flash --BOOT files/halium-boot.img

Reboot in recovery, flash system.img+rootfs

/halium-install -p ut ../files/ubuntu-touch-hybris-xenial-edge-arm64-rootfs.tar.gz ../files/system.img

Halium logging in

Wi-fi

As described nmtui then select Activate a connection.

UBpots - The graphical UI

udev rules

sudo -i
cat /var/lib/lxc/android/rootfs/ueventd*.rc|grep ^/dev|sed -e 's/^\/dev\///'|awk '{printf "ACTION==\"add\", KERNEL==\"%s\", OWNER=\"%s\", GROUP=\"%s\", MODE=\"%s\"\n",$1,$3,$4,$2}' | sed -e 's/\r//' >/usr/lib/lxc-android-config/70-herolte.rules

Edit layout (From Github)

Edit /etc/ubuntu-touch-session.d/android.conf to add

GRID_UNIT_PX=29
QTWEBKIT_DPR=3.0
NATIVE_ORENTATION=portrait
FORM_FACTOR=handset

Add user to update

Step described in Github issue, why?

sudo adduser --force-badname --system --home /nonexistent --no-create-home --quiet _apt || true

Install xenial fix dataspace

Again, from Github. Probably linked to the new user creation to update packages

sudo ubports-qa install xenial_-_fix-dataspace
sudo ubports-qa remove xenial_-_fix-dataspace
sudo reboot

Step by step with to flash system.img and halium-boot.img and boot

This step by step guide is working for me, if you have an issue please read the previous chapter which might help to understand it. In all cases please leave a comment so I can fix the instructions.

Backup

Backup your user data if you want. This process will also create a TWRP backup for Android itself.

Install TWRP

Follow official instructions https://forum.xda-developers.com/t/recovery-exynos-official-twrp-for-galaxy-s7-herolte.3333770/

Setup UBports

Reboot in recovery

Reboot in recovery, it can be done with adb reboot recovery or switch off the phone then power + volume down + home keys.

TWRP backup (optional)

If you want to be able to easily switch back to Android backup:

boot
system
data

Format data partition

In TWRP:

Touch Wipe
Touch Format Data
Enter yes
Go back to home menu thanks to the home button

Flash the boot and file system to the device

Still in TWRP:

Clone Halium install.
Download UBports rootfs.
With sudo, or as root: ./halium-install -v -p ut ./path/to/ubports-touch.rootfs-xenial-armhf.tar.gz ./path/to/system.img Note add -i if your want to have your id_rsa.pub key copied
Enter the password for the phone’s new user.
When finished, then reboot in download mode. In TWRP:
1. Touch reboot
2. Touch Download
Download and compile Heimdall.
Once in Download mode, flash the bootloader with heimdall flash --BOOT ./path/to/files/halium-boot.img
The phone will reboot, Unity will appear, the screen goes back, the phone restarts then Unity hangs. It is expected, then
Enter ip address show in a terminal, it should make display a new interface. Mine returns:

9: enp0s20u2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

So the new interface name is enp0s20u2

Download helpers files from Github.
Run the script to setup the connection, send required files to the phone and connect to it: bash helper-install-host.sh interface_name. Ex: bash helper-install-host.sh enp0s20u2
You should obtain a terminal on: phablet@ubuntu-phablet:~$
Setup Wi-Fi thanks to:
1. Enter nmtui
2. Select Activate a connection
3. Select your SSID
4. Enter your Wi-Fi password
5. Push tab to select <Quit> then enter to exit
Start the configuration script thanks to bash helper-install-phone.sh
Answer y when prompted (twice)

The phone should reboot, then you are in! :-)

Python script to change temperature setpoint into gcode

Fri, 01 Jan 2021 00:00:00 +0000

Small hacky script to set regular temperature changes in 3D printer gcode, to easily use “Temperature towers”.

Temperature tower

I personally use this one, which is a bit small but fast to print.

Usage

Use is simple:

Slice your temperature tower with filament temperature set to the highest temperature
Export your gcode to a file, ex: tower.gcode
Run the python script with correct parameters:
1. file: Input gcode file to be parsed
2. tower_steps: Tower steps in mm
3. temperature: First step temperature in degrees Celcius
4. temperature_step: Temperature steps in degree Celcius
The generated filename is the input name with out_ prefix, ex: out_tower.gcode

Integrated help

There is an integrated help thanks to the -h option:

➜ python temperature.py -h
usage: temperature.py [-h] file tower_steps temperature temperature_step

positional arguments:
  file              Input GCode file to be parsed
  tower_steps       Tower steps in mm
  temperature       First step temperature in degrees °C
  temperature_step  Temperature steps in °C

optional arguments:
  -h, --help        show this help message and exit

Example

➜ python temperature.py tower.gcode 8 215 5
Height 8 mm: temperature set to 215°C
Height 16 mm: temperature set to 210°C
Height 24 mm: temperature set to 205°C
Height 32 mm: temperature set to 200°C
Height 40 mm: temperature set to 195°C
Height 48 mm: temperature set to 190°C
Height 56 mm: temperature set to 185°C
Height 64 mm: temperature set to 180°C
Height 72 mm: temperature set to 175°C
Height 80 mm: temperature set to 170°C

Code

nfcAudio - WiFi enabled MP3 player with NFC capability

Wed, 23 Dec 2020 00:00:00 +0000

nfcAudio is an MP3 player, with audio files selected thanks to an NFC tag placed on top of it. I build this two years ago for my kid so he can play by himself his preferred nursery rhymes and songs, it is still used at least once a week :-)

It is powered by an ESP8266 to play audio from a remote server, an I2S DAC driving the speaker and an NFC reader. Thanks to WiFi we can use it to play local files or any webradio stream from the Internet.

Finished nfcAudio project

Hardware

Electronics

I just used several evaluation boards, linked by soldered wires:

ESP8266: Wemos Mini D1
NFC reader: PN532 board from Aliexpress
I2S DAC: MAX98357A board from Aliexpress

The MAX98357A I2S DAC can easily be replaced by another I2S, or even a single transistor: please see ESP8266Audio project which is wonderfully documented.

Case

I designed a laser-cut box to hold everything together and allow NFC tag storage. All parts of the case are bolted, electronic boards are fixed thanks to zip ties, then the speaker is hot glued.

Speaker under the case

Top opened, with the PN532

Software

The most complicated part, MP3 management, is made by the great ESP8266Audio project.

Then I had to modify Adafruit’s PN532 library as it was only polling the NFC chip, so I implemented interrupts to be able to detect new tags while playing audio; my fork can be found here. Adafruit never merged my pull request but two months ago another (cleaner) one based on my work have been merged. As Adafruit changed all the library API since 2018 I am still using my fork.

My code is really simple: wait for a tag to be detected, read its content then play the associated file :-)

I encode the MP3 URL directly in the tag in the standardized NDEF format so I wrote a small parser for it, then it was done. I program the tags directly from my phone with the application NFC Reader & Writer.

Project sources

Everything, including case files and the wiring is available on Github.

Switch to static site generator: Hugo

Sat, 19 Dec 2020 00:00:00 +0000

For the fifth year of this blog, I switched from Ghost to Hugo. I used to like Ghost as a small and simple tool when I started the blog.

Ghost is now a powerful content management system but far more complex than the simple tool I am looking for, even writing a full post in markdown became less and less easy over the years. As I do not need fancy stuff and like the simplicity to pre-generate the pages and drop the HTML files somewhere I looked at static site generators, I gave a try to Hugo and I liked it. Let’s see what it can do in production but I already enjoy to avoid Ghost, Node, and NPM maintenance :)

This might break RSS feed thought, sorry if you receive 10 updates for old pages.

Wireguard and Pi-hole in Docker containers

Wed, 18 Nov 2020 21:32:24 +0000

I had to move my previous Wireguard VPN + Pi-hole ad blocker to another server, but this time I was not able to expose Pi-hole DNS port (53) to the host machine. It was also better to have Wireguard VPN inside a Docker container… so I did!

As a reminder, Wireguard is a stateless and easy to configure VPN: share a pair of public keys between the client(s) and server then you are good to go! Moreover, stateless is great when used from a phone as there is no power-hungry keep-alive like OpenVPN, nor reconnection time when switching from Wi-FI to 4G.

DNS address issue

The main issue I had is the way to provide Pi-hole address to Wireguard container: docker-compose does not yet accept do translate a container name in dns section. The easiest workaround is to set a fixed IP for the Pi-hole container, not pretty, but hey… it works :-)Then both services must be in the same docker-compose.yml file to avoid any issue related to a network race condition when the docker daemon restarts.

Pi-hole

Pi-Hole configuration is straightforward and well documented on their Docker Hub page.

Wireguard

Instead of creating my own Dockerfile I used the image from linuxserver.io, which has been beautifully implemented. Then follow the documentation. Thanks to Wireguard’s QR Code feature, the phone setup is dumb-proof: just scan it from the container’s logs.

docker-compose file

The resulting docker-compose file is:

version: "3.5"

services:
  wireguard:
    image: linuxserver/wireguard
    depends_on:
      - pihole
    dns:
      - 172.29.0.2
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped
    volumes:
      - ../../data/wireguard:/config
      - /lib/modules:/lib/modules
    ports:
      - 51820:51820/udp
    environment:
      - TZ=Europe/Paris
      - SERVERURL=host_server.yourdomain.com
      - SERVERPORT=1194
      - PEERS=Android_phone
    networks:
      - network-pihole

  pihole:
    image: pihole/pihole:latest
    volumes:
      - ../../data/pi-hole/etc/:/etc/pihole/
      - ../../data/pi-hole/dnsmasq.d:/etc/dnsmasq.d
    environment:
      TZ: "Europe/Paris"
      PROXY_LOCATION: pihole
      VIRTUAL_HOST: pihole.yourdomain.com
      VIRTUAL_PORT: 80
      LETSENCRYPT_EMAIL: email@yourdomain.com
      LETSENCRYPT_HOST: pihole.yourdomain.com
    restart: unless-stopped
    networks:
      network-pihole:
        ipv4_address: 172.29.0.2

networks:
  network-pihole:
    name: "dns-pihole"