My Waste My Future

It's time to put boring, manual tasks in the bin

UPDATE: I have since updated the code to make use of a new data source: My Waste My Future (Part 2)

It's a chilly Friday morning and I'm sleeping soundly in our big, warm bed. Without warning, the door creaks and a thunderous stampede of little footsteps fill the room.

"Daddy, Truck! Truck!" Announces my son excitedly.

My wife rolls over and buries her head beneath some pillows.

"Yes mate, the trucks are coming today, let's go back to bed" I try to calm him.

But in that brief moment of elevating him onto our bed, a bismerched groan enters my consciousness. Indeed, the garbage trucks are arriving today, but the bins that are to be emptied have not been put at their appropriate location. The groan vocalises, extending towards my wife, inciting her to go and do the chore, but her ears are well and truly muffled at this point. It's funny how in certain situations she's a super-heavy sleeper. I begrudingly settle my son in the cozy, warm spot where I had been sleeping and go and put the bins out.

As I check the neighbours bins to determine which ones need to be emptied this week, I wonder why we always seem to forget bin night and which bins need to be emptied. Is it willful ignorance? Is it laziness? It's the same night every week, and it's on a fortnightly rotation, surely it can't be that difficult.

Entering back in the house and pulling my now icey-cold feet from my thongs, I slip back into bed. Making sure to "accidently" touch my wife's feet with mine as punishment for her faux-deafness earlier. I was determined that next week we would put the bins out the night before so we did not have to do this little dance with the deadline again.

I was wrong. The next week I put the bins again out in the morning, checking the neighbours first. I need to fix this the only way I know how, using technology to over-engineer a solution.

Welcome, Wollongong Waste

Since 1987, Wollongong City Council has contracted Remondis to perform it's waste management. In July 2014, a new collection contract was devised for both Wollongong Council and neighbour Shellharbour Council named Remondis Harbour Cities.

Wollongong Waste is split into two websites: wollongongwaste.com.au and wollongongwaste.net.au. The .com.au site appears to be designed for marketing and information purposes, running Redback Solution's Visionscape CMS, while the .net.au is designed for processing services for the end-user, running Zen Cart. I'm not sure why they designed things to use two systems, but this article will focus on the .net.au site.

For the purposes of this article, I will pretend that I live at The Old Court House, Wollongong.

I don't mind the design of the Wollongong Waste website, it looks clean and functional. There's not much cruft and it's relatively easy to navigate. We moved into our current house a couple of years ago and I recall that we only used this site once or twice just after we moved in. I had been opting to rely on my neighbours to remember the approprirate bin for so long that I had forgotten why I stopped using this website.

Then after trying it out, it became excruciatingly apparent. I had been avoiding this website because it takes a whopping 5 page loads to find out which bins I need to put out this week... Five!

Check it out:

Here is a breakdown of this "trial by HTTP request":

1. ?main_page=vrp_advanced_search
2. ?main_page=vrp_advanced_search&suburbs_id=79
3. ?main_page=vrp_advanced_search&suburbs_id=79&street=Harbour+Street
4. ?main_page=vrp_advanced_search&suburbs_id=79&street=Harbour+Street&product_id=73508
5. ?main_page=product_vrp_info&cPath=87_79&products_id=73508

Now any astute person (or rather, anyone with common sense) reading this would immediately say to themself: "Why not bookmark the page and navigate directly to it?". I too posited this question. Unfortunately, due to the way this website is programmed, opening up the page of an address manually reveals a blank page:

This struck me as extremely odd. I've got the products_id of my house, why does it not show me the information straight away? I need to do some digging...

Ohmmm, eCommerce, ohmmmm

As mentioned previously, the Wollongong Waste .net.au site is built using Zen Cart. This platform allows Remondis to process orders for an "On-Call Household Cleanup", or to request modifications to their bins (additional bin, repairs and resize). This eCommerce platform is so old it could legally drink in Australia, but it's still updated to this day (the latest update as of this writing is March 2021). There is some concern that the Apache instance that sits underneath Zen Cart (Apache/2.2.15), became end of life in 2017. I have seen the website in maintenance mode which gives some confidence that Zen Cart may be up to date. I have made sure to provide feedback to Remondis about their ancient version of Apache and recommend anyone else in Wollongong to do the same.

When you land on a page in Zen Cart, a zenid cookie is set in your browser. If cookies are not enabled in your browser, it will append a zenid variable to the request string. Curiously, navigating directly to the product_vrp_info page will set zenid, but it does not appear to be valid enough to display the product results. Navigating to any other page before that page sets a session cookie that is valid and allows the product results to display.

Looking through the Zen Cart code does not show anything at face value as to why the page does not return any results if navigating directly to the page, so to save myself any more head scratching, I'll build a script that does an initial request to set a cookie before scraping the data.

Zen Cart also enforces some simple logic to prevent spiders from creating sessions. It looks through a text file of commonly used bot names and will not create sessions if the word is found in the user-agent string somewhere.

Building a Simple Scraper

I'm going to build this script with python because I find python the easiest language to use when it comes to any type of web scraping. The requests and BeautifulSoup libraries make the task incredibly easy.

In the first part of the script, it sets a valid session cookie, then requests the page with the actual data:

import requests

setcookie_url = 'https://wollongongwaste.net.au/index.php?main_page=vrp_advanced_search'
data_url = 'https://wollongongwaste.net.au/index.php?main_page=product_vrp_info&cPath=87_79&products_id=73508'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36'
}

session = requests.Session()

print("Getting session creation URL")
r = session.get(setcookie_url, headers=headers)

print("Getting data URL")
r = session.get(data_url, headers=headers)

contents = r.content.decode('utf8')

Since the information on this page will only update once a week, I built a simple caching mechanism so that the results are saved locally most of the time.

Once I have the raw HTML, it can be passed into BeautifulSoup and I can extract only the information I need. I'm still a beginner python developer so I'm painfully aware that this code can be made a lot better. If you are a python developer and are feeling generous, please reach out to me and let me know how I can do this better.

soup = BeautifulSoup(contents, 'html.parser')

table = soup.find_all(id="vrpProduct")[1]
rows = table.find_all('tr')
rows = table.select('tr')

bins = []

for idx, row in enumerate(rows):
  if idx == 0:
      continue

  code_name = row.select(".code_name")[0].text.strip()
  if(code_name.__contains__("Residual")):
      colour = 'red'
  elif (code_name.__contains__("FOGO")):
      colour = 'green'
  elif (code_name.__contains__("Recycle")):
      colour = 'yellow'
  else:
      colour = 'blue' 

  schedule_servicedate = row.select(".schedule_servicedate")[0].text.strip()
  if not schedule_servicedate:
      schedule_timestamp = ''
  else:
      schedule_timestamp = int(time.mktime(datetime.strptime(schedule_servicedate, "%a %d %b, %Y").timetuple()))

  if schedule_timestamp == '':
      is_next_week = False
  else:
      time_between = time.time() - schedule_timestamp
      if (time_between < 604800):
          is_next_week = True
      else:
          is_next_week = False

  bin = {
      "code_name": code_name,
      "colour": colour,
      "units": row.select(".units")[0].text.strip(),
      "is_next_week": is_next_week,
      "schedule_description": row.select(".schedule_description")[0].text.strip(),
      "schedule_servicedate": schedule_servicedate,
      "schedule_timestamp": schedule_timestamp
  }

  bins.append(bin)

This provides a list of dicts that can be easily interpreted and output somewhere:

[{'code_name': '240ltr Recycle Bin',
  'colour': 'yellow',
  'is_next_week': True,
  'schedule_description': 'Fortnightly',
  'schedule_servicedate': 'Tue 30 Nov, 2021',
  'schedule_timestamp': 1638190800,
  'units': '1'},
 {'code_name': '240ltr FOGO Bin',
  'colour': 'green',
  'is_next_week': True,
  'schedule_description': 'Weekly',
  'schedule_servicedate': 'Tue 30 Nov, 2021',
  'schedule_timestamp': 1638190800,
  'units': '1'},
 {'code_name': '240ltr Residual Waste Bin',
  'colour': 'red',
  'is_next_week': True,
  'schedule_description': 'Weekly',
  'schedule_servicedate': 'Tue 30 Nov, 2021',
  'schedule_timestamp': 1638190800,
  'units': '1'},
 {'code_name': 'On-Call Household Cleanup',
  'colour': 'blue',
  'is_next_week': False,
  'schedule_description': '',
  'schedule_servicedate': '',
  'schedule_timestamp': '',
  'units': '2'}]

Making use of this information

Now that the data is available, how should it be displayed?

For a few weeks I ran this manually in a terminal window, using some simple code to iterate over the list and output anything where is_next_week was True. However, after running the command more than twice, even the simple act of typing began to feel tedious. Instead of getting the information in the kitchen where the bins reside, I had to go into the office, open a terminal window and enter a command. My laziness/compulsion to automate simply wouldn't allow it for much longer. A couple more weeks and some idle thought here and there, a solution came to mind.

A few months back, I bought a Kano Computer Kit 2017 (with Light Ring) from Facebook Marketplace for only $50. The kit contains a Raspberry Pi 3 Model B, a multicoloured-LED ring hat and a really sweet wireless keyboard/trackpad combo. It runs Kano's own KanoOS and it's a great little product for teaching children programming and computing concepts.

My general idea is to make use of the ring light to output red, green and yellow respectively on the week that those particular bins are set to be put kerbside. I would attach the ring LEDs to my Raspberry Pi running PiHole, install some libraries, write a bit of code and the LEDs would start working. I boldly assumed this would be relatively easy to achieve. I mean, the whole point of this product is to make it easy for kids to use, so surely it wouldn't be that difficult, right?

Wrong. Well, sort-of. The product is easy to use as long as you use KanoOS and make use of their GUI tools to achieve what you want with the light ring.

I did a bit of digging and found Kano's Github Repository for their peripherals. Fortunately for me, they appear to be using Python to interact with what they call the "speaker leds". Unfortunately for me, the code is poorly documented and no efforts were made to make this developer-friendly on non-Kano Raspberry Pi's.

I discovered a script called calibrate-pihat-leds that allows a user to cycle the LEDs through a list of CSS colours. This script gives me a nice base to create a test script, cycle-leds, that will set each LED a random colour every single for 10 seconds. I think I sat there for at least 5 minutes just marvelling at the rainbow colours flashing on the little device. I think that's what Kano were going for with these computers, even if I am a bit older than their target market. After that I created a script trigger-leds, that would take in a list of dicts that come from the scraped page previously, and output the respective colours:

import time
import random
from kano_peripherals.pi_hat.driver.high_level import get_pihat_interface

pihat_interface = get_pihat_interface()
if not pihat_interface:
    print("Could not grab the D-Bus interface to the PiHat board!")

max_priority_lock = pihat_interface.get_max_lock_priority()
pihat_interface.lock(max_priority_lock)

pihat_interface.set_leds_off()

pihat_num_leds = pihat_interface.get_num_leds()

bins = [
    {'colour': 'yellow','units': '1', 'is_next_week': True},
    {'colour': 'red', 'units': '1', 'is_next_week': True},
    {'colour': 'green', 'units': '1', 'is_next_week': True}
]

current_led = 1
leds = []

for bin in bins:
    if bin['colour'] == 'yellow':
        red = 1.0
        green = 0.95 # Max green makes things look less yellow
        blue = 0.0
    elif bin['colour'] == 'green':
        red = 0.0
        green = 1.0
        blue = 0.0
    elif bin['colour'] == 'red':
        red = 1.0
        green = 0.0
        blue = 0.0
    elif bin['colour'] == 'blue':
        red = 0.0
        green = 0.0
        blue = 1.0

    print(str(red) + ", " + str(green) + ", " + str(blue))

    if bin['is_next_week'] == True:
        print(bin['is_next_week'])
        print(bin['units'])

        i = 0
        while i < int(bin['units']):
            print("iter: " + str(i) + ", current_led: " + str(current_led))

            pihat_interface.set_led(current_led, (red, green, blue))
            current_led = current_led+1
            i = i+1

time.sleep(2)

Success, we have red, green and yellow LEDS lit up in a row. And if we happen to have 2 red bins, we have 2 red leds!

Now to merge the two scripts together.

After a night of hacking at it and getting thoroughly lost in SystemD, udev rules, the DBus system and python version incompatabilities, I realised I don't have the skills or patience necessary to get things working on the Pi attached to my router. Instead, I did the lazy thing and decided to just use the Kano Pi itself. I can hear cries of environmentalism already, how inefficient it is to run a whole Pi just to run a few LEDs, but I'll get back to it one day.

Things were pretty easy to get going on the KanoPi, all I had to do was install the requests and BeautifulSoup libraries. I had to make a small change the BeautifulSoup extraction code, opting to do table.select('tr') instead of table.find_all('tr').

Observations, Changes and Future Improvements

Voila! The Kano Pi has been mounted on my kitchen wall using some heavy duty mounting tape. I routed the cable up through a the hole in the cabinetry that houses a frequently broken and now unused downlight.

After having it on the wall for an hour or so, we identified that it was a bit tricky to distinguish the LEDs as they were right next to each other. I modified the code so that there was always a gap in between the LEDs.

Running it for a day I began to have concerns about the fact that only 2 or 3 LEDs will be on the same colour permenantly. To mitigate this, I modified the code so that it chose a starting position at random. Since it runs hourly, the LEDs are constantly at different positions. It does not affect the time required to distinguish the colours.

Now it is very easy to identify which bins are required for the week at a quick glance and it has helped us be one of the first ones in the street to have the correct bins out at the kerb.

There are a few things I would like to update in the future. Currently the LEDs can change before the bins are collected. So the bins would be out but the new bin schedule would be shown. I would like to set it so that it does not update until at least 2-5pm of the day of collection (just in case I forget the night before). There is also no indication that it is the day before collection day. I would like to add some sort of blinking or pulsing mechanism to remind us that it is the night before bin day.

Thankyou for taking the time to read my first "proper" articles about a project. I hope you enjoyed reading as much as I enjoyed making and writing.