Solar Scraping

The home solar / battery system has been running for just over 5 years. While the system has been running smoothly in the background, we’ve encountered a few hiccups over the past six months. Such challenges are perhaps an inherent part of being an early adopter of new technologies. Nevertheless, during this period, we have generated about 1.5 megawatt-hours of energy per year from our roof. I had started to think about expanding the number of panels on our roof (we have 9, space for 6 more) so maybe these “issues” are an opportunity to kickstart that process. So what are the challenges we have been having.

First up, the battery looks like it might be reaching end of life - an issue with the controller and the product no longer being manufactured means that when they come out to check it they will mst likely be removing the system (which means I am in the market for a new battery). Second, when I went to view the amount of solar being generated via an app on my phone I noticed that it was no longer available on the app store (I have my phone setup to offload apps that I don’t use frequently - mistake). Luckily the app was also on an iPad so I can still see our data. But I wanted to start saving the data elsewhere - just in case.

This GitHub Gist is running on a RPi at home that I use to control various things (like the Bus Nook). It connects directly to the Solar Inverter on my home network which seems to be the conduit for getting data out to the Zanussi app.

I am using Beautiful Soup to process data from web page served on the Inverter itself - this gives me power now in Watts, todays cumulative Energy in kWh’s and the system total cumulative energy in kWh’s. I discovered that the inverter web page seems to go off line regularly once the sun goes down (is is solar powered?) so the extra bit of script logging the readings to a log file were implemented so that when i cannot connect to the inverter i still send the last known readings to the MQTT broker (since I might want to display the days total / total energy). Once I had implemented that I realised that using retained messages on the MQTT broker would probably have been a better route to take…

Python Script:

import requests
from bs4 import BeautifulSoup
import paho.mqtt.client as mqtt

url = "http://192.168.1.232/monitor.htm"
username = 'xxx'
password = 'xxx'

# MQTT broker details
mqtt_broker = 'mqtt.cetools.org'
mqtt_port = 1884
mqtt_username = 'xxx'
mqtt_password = 'xxx'

client = mqtt.Client('dwsolar')

log_file_path = "solar.log"

def write_to_log(file_path, power, today, total):
    try:
        with open(file_path, 'w') as log_file:
            log_file.write(f"Power Now: {power}, Todays Total Energy: {today}, Total Energy: {total}\n")
        print("Variables written to log file successfully.")
    except Exception as e:
        print(f"An error occurred while writing to the log file: {e}")

def read_from_log(file_path):
    try:
        with open(file_path, 'r') as log_file:
            content = log_file.read().strip()
            if content:
                # Extracting values from the log file content
                parts = content.split(',')
                power = float(parts[0].split(':')[-1].strip())
                today = float(parts[1].split(':')[-1].strip())
                total = float(parts[2].split(':')[-1].strip())

                return power, today, total
            else:
                print("Log file is empty.")
    except Exception as e:
        print(f"An error occurred while reading and converting from the log file: {e}")


try:
    # Make an HTTP GET request with authentication
    response = requests.get(url, auth=(username, password))

    # Check if the request was successful (status code 200)
    if response.status_code == 200:
        # Make an HTTP GET request with authentication
        response = requests.get(url, auth=(username, password))
        # Process the response data here

        # parse the HTML using BeautifulSoup
        solarpage = BeautifulSoup(response.content, 'html.parser')

        # Find the table with id 'displayme'
        table = solarpage.find('table', {'id': 'displayme'})

        # Find the rows in the table
        rows = table.find_all('tr')

        # Create a dictionary to store the values
        data = {}

        # Loop through each row and extract the values
        for row in rows:
            columns = row.find_all(['td', 'th'])
            if len(columns) == 2:
                key = columns[0].text.strip()
                value = columns[1].text.strip()
                data[key] = value

        # Extract the values for 'Power Now' and 'Total Energy'
        power_now = data.get('Power Now:', 'N/A').rstrip('W')
        today_energy = data.get('Today\'s Energy:', 'N/A').rstrip('kWh')
        total_energy = data.get('Total Energy:', 'N/A').rstrip('kWh')

        print(f"URL : Power Now (W): {power_now}, Todays Total Energy (kWh): {today_energy}, Total Energy (kWh): {total_energy}")

        # Write variables to the log file
        write_to_log(log_file_path, power_now, today_energy, total_energy)

    else:
        print(f"Request failed with status code: {response.status_code}")

        # Read and convert values from the log file
        result = read_from_log(log_file_path)
        if result:
            power_now, today_energy, total_energy = result
            power_now = 0 # I dont want to show the last value in the mqtt feed
            print(f"File: Power Now (W): {power_now}, Todays Total Energy (kWh): {today_energy}, Total Energy (kWh): {total_energy}")


except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")
    # Handle the error gracefully, e.g., by logging the error or taking appropriate action
    # Read and convert values from the log file
    result = read_from_log(log_file_path)
    if result:
        power_now, today_energy, total_energy = result
        power_now = 0 # I dont want to show the last value in the mqtt feed
        print(f"File: Power Now (W): {power_now}, Todays Total Energy (kWh): {today_energy}, Total Energy (kWh): {total_energy}")


# Even when inverter is offline I still want to send the last known data to MQTT
client.username_pw_set(mqtt_username, mqtt_password)
client.connect(mqtt_broker, mqtt_port, 60)

client.publish('personal/ucjtdjw/33ph/EM/solar/now/W', power_now)
client.publish('personal/ucjtdjw/33ph/EM/solar/today-total/kWh', today_energy)
client.publish('personal/ucjtdjw/33ph/EM/solar/total-since-11-2018/kWh', total_energy)

# Disconnect from MQTT broker
client.disconnect()