Gas prices soared when Putin entered Ukraine, causing homeowners to seek ways to lower their monthly energy bills. I found myself with some spare chip sensors, including the MLX90640 and an ESP32-C3 (RISC) WiFi chip. While connecting to an SPI screen seemed impractical, the MLX90640’s 24x32-bit sensor allowed me to read frames using the I2C protocol.

As I searched for solutions to display the thermal images, most focused on an SPI LCD, but I wanted a stream-focused approach like IP cameras. By using the ESP32 WiFi chip to display the images on my phone’s browser, I was able to create a cheap and small solution to help homeowners understand where their house was leaking heat.

Developing a successful Livestream for an IP camera requires a solution that can handle multiple streams and hotspots while adapting to the heat for auto-scaled color generated images. While my current implementation works relatively easily with auto-scaled color generated images, I’m still working on integrating additional features to improve the user experience, like a Hotspot handling multiple users.

Thermal video streaming webserver

When working with a thermal camera like the MLX90640, it’s important to pay close attention to the following code blocks: float frame[32*24] and the mlx definition, as well as the setup initialization for the thermal camera. Additionally, it’s crucial to monitor the loop function, as it addresses a bug where the thermal camera fails to send heat images if communication is delayed. While any thermal camera could be used, understanding these key code blocks is essential for successful implementation.

Link to github page

#ifdef ESP32
#include <Arduino.h>
#include <WiFi.h>
#include <WiFiClient.h>
#include <WebServer.h>
#include <ESPmDNS.h>
#elif defined(ESP8266)
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include <ESP8266mDNS.h>
#endif

#include <Adafruit_MLX90640.h>
#include "settings.h"
#include "Arduino.h"
#include "fb_gfx.h"
#include "soc/soc.h"          //disable brownout problems
#include <soc/rtc_cntl_reg.h> //disable brownout problems
#include <esp_http_server.h>

Adafruit_MLX90640 mlx;
camera_fb_t       fb;

// low range of the sensor (this will be blue on the screen)
#define MINTEMP (float)20

// high range of the sensor (this will be red on the screen)
#define MAXTEMP (float)35

float frame[32 * 24]; // buffer for full frame of temperatures

...

    Serial.println("Adafruit MLX90640 Camera");
    if (!mlx.begin(MLX90640_I2CADDR_DEFAULT, &Wire)) {
        Serial.println("Not Found Adafruit MLX90640");
    } else {
        Serial.println("Found Adafruit MLX90640");
    }
    Serial.print("Serial number: ");
    Serial.print(mlx.serialNumber[0], HEX);
    Serial.print(mlx.serialNumber[1], HEX);
    Serial.println(mlx.serialNumber[2], HEX);

    mlx.setMode(MLX90640_CHESS);
    mlx.setResolution(MLX90640_ADC_18BIT);
    mlx.setRefreshRate(MLX90640_8_HZ);
    Wire.setClock(1000000); // max 1 MHz

...
void loop() {

    delay(1);
    if (mlx.getFrame(frame) != 0) {
        Serial.println("Failed");
    } else {
        Serial.println("Got a frame..");
    }
}

To achieve a higher frame rate of approximately 8FPS, the current implementation uses a chess layout for retrieving frames from the thermal camera. While this approach has proven effective, other methods could be explored to further optimize frame rate and overall performance.

Install

Use VSCode and install platform.io as an Extension. Clone the project: git clone https://github.com/martijnvwezel/Home-Assist_sensors.git And open in vscode the folder esp-heatcamera.

Go to the src/settings.h and replace the following strings with your wifi SSID name and your password:

const char* ssid     = "ssid";
const char* password = "password";

Build the project and upload it using PlatformIO.

Weird problem? Create an issue in Github.

Video Streaming Web Server

To ensure compatibility with web applications, the raw float arrays of frames need to be converted to a format that can be easily transmitted. The first step in this process is converting the images to JPG format. While some may question why JPG is preferred over PNG, the ESP32 has enough processing power to compress the images, allowing for faster transmission. Although this approach may result in a loss of information and some unique cases slightly larger file sizes than transmitting raw images, it has proven effective without noticeable artifacts in most situations.

Home Assistant users

Despite successful display of the thermal camera stream on the website card, the Home-Assistant picture card is currently not displaying the stream. I am currently investigating this issue and exploring options to enlarge the heat camera image within the card. Once a solution is found, I will update the settings accordingly.

type: iframe
url: http://192.168.1.124/
aspect_ratio: '10'

Other integrations

General

If you’re interested in customizing the project further, you can find an index.html file in the project folder. To use it, simply update the URL in the file to match the IP address of your ESP32. You can find the IP address by connecting the ESP32 to your PC and monitoring the logs on baud rate 115200 or by using a network scanning application like Fing on your phone or PC.

<!DOCTYPE html>
<html>
<body>

<div style="margin-bottom: 10px;">
    <img src="http://192.168.1.124" width="650px">
</div>
<a href="https://sales.muino.nl/">sales.muino.nl</a>
</body>
</html>

Node-RED

If you’re looking to integrate the project with Node-RED, you’ll need to use the code below and update the IP address to match that of your ESP32:

<div style="margin-bottom: 10px;">
    <img src="http://192.168.1.124" width="650px">
</div>

FAQ’s

No images loaded using index.html?

This can be solved simply by refreshing the webpage or closing a stream that is already being used. Or do you have to select the correct IP

Stream stops from the sensor.

Currently, figuring out why sometimes the stream can stop, probably a simple restart of the communication would be enough.