ESP32 Thermal Webcame Stream

During the time Putin entered Ukraine, gas prices sky-rocket. Therefore, making improvements to the house’s isolation was much to lower the monthly energy bill. Therefore, it became more important to understand where the house is leaking its heat.

I had some chip sensors lying around like the MLX90640 and an ESP32-C3 (RISC) WiFi chip and connecting to any SPI screen that I had sounded too impractical. The MLX90640 is a thermal camera that has a 24x32-bit sensor. Using the I2C protocol the frames can be read.

If you Google for solutions, many of them are focused on an SPI LCD, but none of them is stream-focused like IP cameras. I wanted to use the ESP32 WiFi chip to display the thermal images to the phone in the browser. A cheap solution and a small solution.

side side

The way it should work is using a Livestream like a general IP camera. The current solution works fairly easily, but I didn’t implement a hotspot and multiple streams handler yet.

The current implementation works with grey-scale images, but cannot handle multiple streams. There is no hotspot option added yet.

Thermal video streaming webserver

For the thermal camera, the MLX90640 is used, but any thermal camera could work. The following lines, you need to be extra aware of are the following code blocks shown but basically the float frame[32*24] and mlx definition. Also the definition in the setup part where the initialisation of the thermal camera is done. Also, the loop function is important to watch because this solves a tiny bug with the thermal camera function. If you are not quickly enough communicating with the thermal camera it crashes/fails to send heat images anymore.

#ifdef ESP32
#include <Arduino.h>
#include <WiFi.h>
#include <WiFiClient.h>
#include <WebServer.h>
#include <ESPmDNS.h>
#elif defined(ESP8266)
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include <ESP8266mDNS.h>
#endif

#include <Adafruit_MLX90640.h>
#include "settings.h"
#include "Arduino.h"
#include "fb_gfx.h"
#include "soc/soc.h"          //disable brownout problems
#include <soc/rtc_cntl_reg.h> //disable brownout problems
#include <esp_http_server.h>

Adafruit_MLX90640 mlx;
camera_fb_t       fb;

// low range of the sensor (this will be blue on the screen)
#define MINTEMP (float)20

// high range of the sensor (this will be red on the screen)
#define MAXTEMP (float)35

float frame[32 * 24]; // buffer for full frame of temperatures

...

    Serial.println("Adafruit MLX90640 Camera");
    if (!mlx.begin(MLX90640_I2CADDR_DEFAULT, &Wire)) {
        Serial.println("Not Found Adafruit MLX90640");
    } else {
        Serial.println("Found Adafruit MLX90640");
    }
    Serial.print("Serial number: ");
    Serial.print(mlx.serialNumber[0], HEX);
    Serial.print(mlx.serialNumber[1], HEX);
    Serial.println(mlx.serialNumber[2], HEX);

    mlx.setMode(MLX90640_CHESS);
    mlx.setResolution(MLX90640_ADC_18BIT);
    mlx.setRefreshRate(MLX90640_8_HZ);
    Wire.setClock(1000000); // max 1 MHz

...
void loop() {

    delay(1);
    if (mlx.getFrame(frame) != 0) {
        Serial.println("Failed");
    } else {
        Serial.println("Got a frame..");
    }
}

Currently, the chess layout is used for getting the frames out of the thermal camera. The reason is for achieving a higher frame rate. The frame rate should be close to 8FPS.

Install

Use VSCode and install platform.io as an Extension. Clone the project: git clone https://github.com/martijnvwezel/Home-Assist_sensors.git And open in vscode the folder esp-heatcamera.

Go to the src/settings.h and replace the following strings with your wifi SSID name and your password:

const char* ssid     = "ssid";
const char* password = "password";

Build the project and upload it using PlatformIO.

Weird problem? Create an issue in Github.

Video Streaming Web Server

The raw arrays of frames are floats and need to be converted to something the web application can use. This is done to first convert the images to JPG. Why JPG and not PNGs? Basically, the ESP32 has enough processing power and compressing the images means faster sending of them to another application. You lose a bit of information and in some situations, the images are bigger if you would sent them raw. Nevertheless, it works and doesn’t notice the artefacts that much.

Home Assistant users

Somehow the Home-Assistant picture card doesn’t show the stream, but the website card does! The following setting is used to show the screen in home assist. I want to enlarge the heat camera image in some way, but didn’t yet figure that out.

type: iframe
url: http://192.168.1.124/
aspect_ratio: '10'

Other integrations

General

There is in that project folder also an index.html. You can edit this by changing the URL to the one from your ESP32 (if you connect the esp to your PC with a serial monitor on baud rate 115200, you can see in the logs which IP it uses. Or use an application like Fing on your phone/PC).

<!DOCTYPE html>
<html>
<body>

<div style="margin-bottom: 10px;">
    <img src="http://192.168.1.124" width="650px">
</div>
<a href="https://sales.muino.nl/">sales.muino.nl</a>
</body>
</html>

Node-RED

For some integration like Node-RED you have to use to code below, change the IP address to the one of

<div style="margin-bottom: 10px;">
    <img src="http://192.168.1.124" width="650px">
</div>

FAQ’s

No images loaded using index.html?

This can be solved simply by refreshing the webpage or closing a stream that is already being used. Or do you have to select the correct IP

Stream stops from the sensor.

Currently, figuring out why sometimes the stream can stop, probably a simple restart of the communication would be enough.