All posts by Oliver

RS232 / USB ‘Cloning Module’

This device is designed to connect to a specific system via RS232 and then receive or transmit a few kilobytes of settings, to or from the MCU flash. This allows you to clone the settings and reproduce the setup on another system easily. It appears as a serial USB CDC device when connected to the mini-USB port which then emulates the behaviour of being connected to the system via RS232.

The parts, footprints, physical construction and materials were chosen to minimise costs – it was built to be cheap and as minimal as possible!

An additional piece of board was routed out to frame the USB connector and support the strain relief for the cable. It locks in through slots in the main PCB and top perspex.

The top cover has been laser cut / engraved from frosted perspex. The buttons come almost flush with the surface.

Although this isn’t good practice, the PCB is threaded directly (normally you would use PEM ReelFast spacers) with the overlap of each screw acting as mini-feet. There’s a few mm clearance around each hole to the nearest active trace.

The whole design is routed on a two layer PCB with almost the entire underside being a ground plane.

Three LED’s sit between the buttons – red, green and amber to indicate the state.

Bang & Olufsen Beomaster 2400

This beautiful piece of design from Bang & Olufsen is now 36+ years old. Some features include touch sensitive controls (pioneered on the Beomaster 1900 a year earlier), an ultrasonic remote control and 5 preset FM stations.

As with most electronic equipment, electrolytic capacitors are common points of failure. A summary of why this can happen is explained here (Aluminum electrolytic capacitor failure). Replacing these and the bridge rectifiers, also known to be a problem point, will significantly extend its useful life.

It’s quite a complicated assembly, mainly due to its compact design.

The replacements are all smaller than their 70’s counterparts and long-life versions have been chosen where possible. The rectifiers have been swapped for uprated versions.

The replaced components can be found in the table below and have been highlighted in the BeoMaster 2400 Service Manual.

B&O REFERENCETYPEVALUEREPLACEMENT PART #MFGVRATED HOURSLINK
4200098Cap100uFEEUFR1H101Panasonic50V6000 @ 105°CLink
4201065Cap10uFEEUEB1J100SPanasonic63V5000 @ 105°CLink
4200298Cap1uFEEUEB1J1R0SPanasonic63V5000 @ 105°CLink
4200296Cap2.2uFEEUEB1J2R2SPanasonic63V5000 @ 105°CLink
4200332Cap2200uFMAL202117222E3Vishay40V8000 @ 85°CLink
4200097Cap220uFEEUEB1E221SPanasonic25V5000 @ 105°CLink
4200100Cap22uFEEUEB1H220SPanasonic50V7000 @ 105°CLink
4201061Cap4.7uFEEUEB1J4R7SPanasonic63V5000 @ 105°CLink
4200092Cap47uFEEUFR1J470Panasonic63V5000 @ 105°CLink
4200305Cap5000uFECOS1VP682CAPanasonic35V3000 @ 85°CLink
4200109Cap470uFEEUFR1C471Panasonic10V6000 @ 105°CLink
8310275B.RGSIB2520-E3/45Vishay / I.RLink
8300277B.RVS-KBPC602PBFVishay / I.RLink
5370174Pot220RCB10LV221MTycoLink

3M Microtouch touchscreen emulator in Python

A colleague asked about virtualising a touch screen DOS application, designed for use with a microtouch touch screen interfacing over RS232. The familiar story of the monitor / old PC hardware dying and replacing it seemed a shorter term solution to a problem that could be solved with virtualisation or emulation. Note: This software no longer has any external hardware connected.

The proposal:

  1. Virtualise or emulate the application within the windows environment.
  2. Link the application to a serial port on the host system (virtual or real)
  3. Write a touch screen emulator to simulate presses using mouse clicks within the application window.

VirtualBox and VMWare Workstation are two virtualisation solutions that support mapping host serial ports to virtual ports. After attempting to use both of these, the timing critical nature of this specific DOS application upon startup didn’t like [my suspected]  lag of the virtualised ports. Running it again in DOSBox, a x86 emulator, resolved this issue. Virtual Serial Port Emulator is a tool to create virtual ports that can act as a connector between programs. This maps the serial streams from the touchscreen emulator to the serial port DOSBox is looking for in this case (how-to configure DOSBox serial ports).

VSPE-screenshot

Finally, the touch screen emulation is written in Python as a quick and simple way to test it out. PyHook is used to capture mouse clicks and mouse position, while PySerial is used for all serial communication. This program  will firstly respond to a handful of serial commands that are required upon loading the system in DOSBox. Once in, it will ask you to calibrate the emulated touch screen by clicking the top left and bottom right of the application window. It will only action on clicks in the specified window, in this case, DOSBox. Each click within this area will be mapped across the standard 10 bit vertical and horizontal touch screen space and sent to the serial port specified.

# Microtouch emulator
# Copyright (C) 2014 Oliver S
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

import serial
import pyHook
import pythoncom

left_click = 0
    
# check that value passed is within specified range
def set_limits(n, minn, maxn):
    if n < minn:
        return minn
    elif n > maxn:
        return maxn
    else:
        return n

# on each mouse click
def OnMouseEvent(event):

    # record current mouse position
    x,y = event.Position

    global left_click,tlp_x,tlp_y,brp_x,brp_y # tlp = top left position, brp = bottom right position

    # check to see if we're clicking in the dosbox window
    if event.WindowName and "DOSBox" in event.WindowName:
        
        left_click += 1

        if (left_click == 1):
            print "Please click top left of desired window"
        elif (left_click == 2): # if second click store top left position
            print "Please click bottom right of desired window"
            tlp_x = x
            tlp_y = y
        elif (left_click == 3): # if third click then store bottom right position
            brp_x = x
            brp_y = y
            print "Finished - Window size:", (brp_x-tlp_x), (brp_y-tlp_y)

        # if calibration is complete        
        if left_click > 3:

            # now calculate the position
            # the full range of the 3M microtouch range in hexadecimal mode is 000 - 3FF
            # use the full range (0 - 1023)
            
            screen_x = int ((1023.0 / (brp_x - tlp_x)) * (x - tlp_x))
            screen_y = int (((1023.0 / (brp_y - tlp_y)) * ((brp_y - tlp_y)-(y - tlp_y))))

            # set upper and lower limits on values if outside of defined area
            screen_x = set_limits(screen_x,0,1023)
            screen_y = set_limits(screen_y,0,1023)

            print "Sending (",screen_x,",",screen_y, ") Hex: (\x01"+format(screen_x, '03X')+","+format(screen_y, '03X')+"\x0d) "
            ser.write("\x01"+format(screen_x, '03X')+","+format(screen_y, '03X')+"\x0d")      # write a string

    else:
        print "Clicked outside DOSBox;"

    # return True to pass the event to other handlers
    # return False to stop the event from propagating
    return True

  
if __name__ == '__main__':

    # attempt to open serial port with parameters as specified
    try: 
        ser = serial.Serial(
            port='COM1',
            baudrate=9600,
            bytesize=7,
            parity='N',
            stopbits=2,
            timeout=0,
            xonxoff=True,
            rtscts=0
            )
        
        if ser.isOpen():

            # mouse stuff            
            
            hm = pyHook.HookManager()               # create the hook mananger           
            hm.MouseAllButtonsDown = OnMouseEvent   # register two callbacks            
            hm.HookMouse()                          # hook into the mouse and keyboard events

            # serial stuff
            
            ser.flushInput()                        # flush input buffer
            ser.flushOutput()                       # flush output buffer
            
            while True:
                pythoncom.PumpWaitingMessages() 

                # read any new lines
                response = ser.readline() # this is non-blocking, provided that timeout is set to 0
                
                # if there is a new line
                if response:

                    print (response)
                    
                    if "R" in response:
                        print "Reset Command"
                        ser.write('\x010\x0d')
                    elif "MS" in response:
                        print "Mode Stream"
                        ser.write('\x010\x0d')
                    elif "FH" in response:
                        print "Format Hex"
                        ser.write('\x010\x0d')
                    elif "MP" in response:
                        print "Mode Point"
                        ser.write('\x010\x0d')
                    elif "CI" in response:
                        print "Calibrate Interactive"
                        ser.write('\x010\x0d')  
                                        
            ser.close()    

    except serial.SerialException as e:
            print("Failed to open Serial Port: {}".format(e))

Please note that this has been tailored to the application that I’m running – but can be easily tweaked / amended as necessary using the microtouch reference guide. The shell output:

microtouch-emulator-output

The DOS application running  – with some numbers entered through the emulator using a mouse.

DOSBox-Output

Scraping data from the BBC with Python

There is an annoying tendency of the BBC News site to have numerous ancient stories in their ‘Popular’ sidebar. It was a good excuse to try out Python and collect a bit more information on when this occurs.

This Python script follows the following process:

  • Visit the BBC News homepage and scrape the ‘Most Popular’ sidebar.
  • Visit the URL of each story.
  • Collect the published date (from the meta data and front end)
  • Calculate the difference between present date / time and the published date.
  • Store the data as a CSV file.
  • Repeat the whole process every n minutes or seconds.

Beautiful Soup makes relatively light work of parsing what we want, along with PrettyTable, CSV, Regular Expressions and Requests.

# BBC News scraper
# Copyright (C) 2014 Oliver S
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

import time
import requests
import re
import csv
from prettytable import PrettyTable
from datetime import datetime
from bs4 import BeautifulSoup

def scrape_bbc_popular_stories():
    """ Scrape all of the shared and popular stories from BBC homepage and record published date """

    try:
        # request BBC news homepage    
        response = requests.get("http://www.bbc.co.uk/news/")

        # parse HTML using Beautiful Soup
        # returns a `soup` object
        soup = BeautifulSoup(response.content)

        # find all the posts in the page.

        # find each list element belonging to class ol[number]

        most_popular=soup.find_all('li',{'class': re.compile('ol[0-9]')})

        #article group is 0 = shared 1 = most read 2 = audio / video
        #story number is the number assigned to the article (the number next to the title)

        article_group = 0
        
        last_story_number = 0

        # set up prettytable for nice printout
        x = PrettyTable(["Days Old", "Group Name", "Story Number", "Story Title", "URL", "Present D/T", "Original Pub Date", "Orig Pub Date (front)"])
        x.align["URL"] = "l"            # Left align URL
        x.align["Story Title"] = "l"    # Left align story title
        
        for each_story in most_popular:

            # store the URL of each story
            link = each_story.find('a').attrs['href']   
            print link

            # get the story title
            story_title = each_story.find('a').text 
            story_title = re.sub(r'^\W*\w+\W*', '', story_title)

            # get the story number (thats listed next to the article)
            story_number = each_story.span.text   
            story_number = re.sub(':\s*',"",story_number)

            # work out which group it belongs to
            if (abs(last_story_number-int(story_number)) >= 4)   :
                article_group += 1
                
            last_story_number = int(story_number)
            
            if (article_group == 0) :
                article_group_name = "Most Shared"
            elif (article_group == 1) :
                article_group_name = "Most Read"
            else :
                article_group_name = "Audio / Video"

            
            # if its an audio / video story that uses javascript to generate the time relations, ignore
            
            if article_group != 2:
                # retrieve the missed connection with requests
                response = requests.get(link)

                # Parse the html of the missed connection post
                soup = BeautifulSoup(response.content)

                # get the original publication date from the meta tags
                orig_pub_date_meta =  soup.find(attrs={"name":"OriginalPublicationDate"})['content']

                # get the original publication date from the front end (new BBC site)
                try:
                    orig_pub_date_front = soup.find('span',{'class':'story-date'}).text
                    orig_pub_date_front = datetime.strptime(orig_pub_date_front, "\n%d %B %Y\nLast updated at %H:%M\n")
                except AttributeError:
                    print "Old site - set front date to meta date"
                    orig_pub_date_front = orig_pub_date_meta
                    
                    
                # calculate the difference between original publishing date and time of checking
                orig_pub_date_meta = datetime.strptime(orig_pub_date_meta, "%Y/%m/%d %H:%M:%S")
                days_difference = abs((datetime.now()-orig_pub_date_meta).days)

                #"Days Old", "Group Name", "Story Number", "Story Title", "URL", "Present D/T", "Original Pub Date", "Orig Pub Date (front)"
                x.add_row([days_difference,article_group_name,story_number,story_title,link,datetime.now().strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_meta.strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_front])
                #"Check TS", "Panel", "Title", "URL", "Pub Date Meta", "Pub Date Front"
                f.writerow([days_difference,article_group_name.encode('ascii','ignore'),story_number.encode('ascii','ignore'),story_title.encode('ascii','ignore'),link,datetime.now().strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_meta.strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_front])

        # print the nicely formatted table that PrettyTable generated, once all stories collected
        print x
        
    except requests.exceptions.RequestException as e:    # This is the correct syntax
        print e



if __name__ == '__main__':
    

    f = csv.writer(open("bbc_popular_stories.csv", "wb"))
    f.writerow(["Days Old", "Group Name", "Story Number", "Story Title", "URL", "Present D/T", "Original Pub Date", "Orig Pub Date (front)"]) # Write column headers as the first line

    while True:
        scrape_bbc_popular_stories()
        print "Waiting for 5 minutes...."
        time.sleep(300)

Running this in IDLE shell looks like this:

and some of the ‘old’ results collected over a 3 hour period can be seen in this spreadsheet . Out of 46 news stories listed under popular stories during this period, 21 were over 67 days old!

Days OldStory TitleURLPresent D/TOriginal Pub Date
281Rent ‘unaffordable’ in third of UKhttp://www.bbc.co.uk/news/business-2327344823/04/2014 00:0115/07/2013 06:35
852O’Donnell warns of UK challengeshttp://www.bbc.co.uk/news/uk-politics-1629542123/04/2014 00:0122/12/2011 18:10
67Pension system ‘is not working’http://www.bbc.co.uk/news/business-2617811323/04/2014 00:1114/02/2014 11:31
593Union joint strike action warninghttp://www.bbc.co.uk/news/business-1951419523/04/2014 00:2606/09/2012 23:40
854Government outlines pension dealhttp://www.bbc.co.uk/news/business-1625923823/04/2014 00:3620/12/2011 22:20
244Millions ‘worse off’ on new pensionhttp://www.bbc.co.uk/news/business-2377032723/04/2014 00:4621/08/2013 12:35
508Plain packs for Australia smokershttp://www.bbc.co.uk/news/world-asia-2055958523/04/2014 01:1201/12/2012 01:08
509Energy Bill for ‘cleaner economy’http://www.bbc.co.uk/news/business-2053998123/04/2014 01:1729/11/2012 15:14
615Australia court backs tobacco lawhttp://www.bbc.co.uk/news/business-1926424523/04/2014 01:3215/08/2012 01:56
887Doctors call for car smoking banhttp://www.bbc.co.uk/news/health-1574435223/04/2014 01:5217/11/2011 17:10
435Bedroom tax’s’ impact on the northhttp://www.bbc.co.uk/news/uk-england-tyne-2141282623/04/2014 02:0211/02/2013 14:01
317Labour ‘would cap welfare spending’http://www.bbc.co.uk/news/uk-politics-2278528223/04/2014 02:0709/06/2013 16:37
323Ed Balls seeks to restore Labour’s economic credibilityhttp://www.bbc.co.uk/news/uk-politics-2275304023/04/2014 02:1703/06/2013 13:24
281Benefit cap ‘leads to more in work’http://www.bbc.co.uk/news/business-2330609223/04/2014 02:1715/07/2013 17:51
818Britons ‘becoming more dishonest’http://www.bbc.co.uk/news/uk-1671487223/04/2014 02:4325/01/2012 08:54
218Benefit cheats face 10-year termshttp://www.bbc.co.uk/news/uk-2410474323/04/2014 02:4816/09/2013 12:49
811Family life on benefitshttp://www.bbc.co.uk/news/uk-1681218523/04/2014 02:5301/02/2012 12:47
135Most people in poverty are ‘in work’http://www.bbc.co.uk/news/uk-2528706823/04/2014 03:2308/12/2013 21:49
187Work ‘may be no way out of poverty’http://www.bbc.co.uk/news/uk-politics-2455361123/04/2014 03:2817/10/2013 14:59
692Professions ‘must be more open’http://www.bbc.co.uk/news/uk-politics-1825421923/04/2014 03:3930/05/2012 15:32
309Top unis ‘now less socially diverse’http://www.bbc.co.uk/news/education-2291260923/04/2014 04:0417/06/2013 07:38