Tag Archives: python

3M Microtouch touchscreen emulator in Python

A colleague asked about virtualising a touch screen DOS application, designed for use with a microtouch touch screen interfacing over RS232. The familiar story of the monitor / old PC hardware dying and replacing it seemed a shorter term solution to a problem that could be solved with virtualisation or emulation. Note: This software no longer has any external hardware connected.

The proposal:

  1. Virtualise or emulate the application within the windows environment.
  2. Link the application to a serial port on the host system (virtual or real)
  3. Write a touch screen emulator to simulate presses using mouse clicks within the application window.

VirtualBox and VMWare Workstation are two virtualisation solutions that support mapping host serial ports to virtual ports. After attempting to use both of these, the timing critical nature of this specific DOS application upon startup didn’t like [my suspected]  lag of the virtualised ports. Running it again in DOSBox, a x86 emulator, resolved this issue. Virtual Serial Port Emulator is a tool to create virtual ports that can act as a connector between programs. This maps the serial streams from the touchscreen emulator to the serial port DOSBox is looking for in this case (how-to configure DOSBox serial ports).

VSPE-screenshot

Finally, the touch screen emulation is written in Python as a quick and simple way to test it out. PyHook is used to capture mouse clicks and mouse position, while PySerial is used for all serial communication. This program  will firstly respond to a handful of serial commands that are required upon loading the system in DOSBox. Once in, it will ask you to calibrate the emulated touch screen by clicking the top left and bottom right of the application window. It will only action on clicks in the specified window, in this case, DOSBox. Each click within this area will be mapped across the standard 10 bit vertical and horizontal touch screen space and sent to the serial port specified.

# Microtouch emulator
# Copyright (C) 2014 Oliver S
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

import serial
import pyHook
import pythoncom

left_click = 0
    
# check that value passed is within specified range
def set_limits(n, minn, maxn):
    if n < minn:
        return minn
    elif n > maxn:
        return maxn
    else:
        return n

# on each mouse click
def OnMouseEvent(event):

    # record current mouse position
    x,y = event.Position

    global left_click,tlp_x,tlp_y,brp_x,brp_y # tlp = top left position, brp = bottom right position

    # check to see if we're clicking in the dosbox window
    if event.WindowName and "DOSBox" in event.WindowName:
        
        left_click += 1

        if (left_click == 1):
            print "Please click top left of desired window"
        elif (left_click == 2): # if second click store top left position
            print "Please click bottom right of desired window"
            tlp_x = x
            tlp_y = y
        elif (left_click == 3): # if third click then store bottom right position
            brp_x = x
            brp_y = y
            print "Finished - Window size:", (brp_x-tlp_x), (brp_y-tlp_y)

        # if calibration is complete        
        if left_click > 3:

            # now calculate the position
            # the full range of the 3M microtouch range in hexadecimal mode is 000 - 3FF
            # use the full range (0 - 1023)
            
            screen_x = int ((1023.0 / (brp_x - tlp_x)) * (x - tlp_x))
            screen_y = int (((1023.0 / (brp_y - tlp_y)) * ((brp_y - tlp_y)-(y - tlp_y))))

            # set upper and lower limits on values if outside of defined area
            screen_x = set_limits(screen_x,0,1023)
            screen_y = set_limits(screen_y,0,1023)

            print "Sending (",screen_x,",",screen_y, ") Hex: (\x01"+format(screen_x, '03X')+","+format(screen_y, '03X')+"\x0d) "
            ser.write("\x01"+format(screen_x, '03X')+","+format(screen_y, '03X')+"\x0d")      # write a string

    else:
        print "Clicked outside DOSBox;"

    # return True to pass the event to other handlers
    # return False to stop the event from propagating
    return True

  
if __name__ == '__main__':

    # attempt to open serial port with parameters as specified
    try: 
        ser = serial.Serial(
            port='COM1',
            baudrate=9600,
            bytesize=7,
            parity='N',
            stopbits=2,
            timeout=0,
            xonxoff=True,
            rtscts=0
            )
        
        if ser.isOpen():

            # mouse stuff            
            
            hm = pyHook.HookManager()               # create the hook mananger           
            hm.MouseAllButtonsDown = OnMouseEvent   # register two callbacks            
            hm.HookMouse()                          # hook into the mouse and keyboard events

            # serial stuff
            
            ser.flushInput()                        # flush input buffer
            ser.flushOutput()                       # flush output buffer
            
            while True:
                pythoncom.PumpWaitingMessages() 

                # read any new lines
                response = ser.readline() # this is non-blocking, provided that timeout is set to 0
                
                # if there is a new line
                if response:

                    print (response)
                    
                    if "R" in response:
                        print "Reset Command"
                        ser.write('\x010\x0d')
                    elif "MS" in response:
                        print "Mode Stream"
                        ser.write('\x010\x0d')
                    elif "FH" in response:
                        print "Format Hex"
                        ser.write('\x010\x0d')
                    elif "MP" in response:
                        print "Mode Point"
                        ser.write('\x010\x0d')
                    elif "CI" in response:
                        print "Calibrate Interactive"
                        ser.write('\x010\x0d')  
                                        
            ser.close()    

    except serial.SerialException as e:
            print("Failed to open Serial Port: {}".format(e))

Please note that this has been tailored to the application that I’m running – but can be easily tweaked / amended as necessary using the microtouch reference guide. The shell output:

microtouch-emulator-output

The DOS application running  – with some numbers entered through the emulator using a mouse.

DOSBox-Output

Scraping data from the BBC with Python

There is an annoying tendency of the BBC News site to have numerous ancient stories in their ‘Popular’ sidebar. It was a good excuse to try out Python and collect a bit more information on when this occurs.

This Python script follows the following process:

  • Visit the BBC News homepage and scrape the ‘Most Popular’ sidebar.
  • Visit the URL of each story.
  • Collect the published date (from the meta data and front end)
  • Calculate the difference between present date / time and the published date.
  • Store the data as a CSV file.
  • Repeat the whole process every n minutes or seconds.

Beautiful Soup makes relatively light work of parsing what we want, along with PrettyTable, CSV, Regular Expressions and Requests.

# BBC News scraper
# Copyright (C) 2014 Oliver S
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

import time
import requests
import re
import csv
from prettytable import PrettyTable
from datetime import datetime
from bs4 import BeautifulSoup

def scrape_bbc_popular_stories():
    """ Scrape all of the shared and popular stories from BBC homepage and record published date """

    try:
        # request BBC news homepage    
        response = requests.get("http://www.bbc.co.uk/news/")

        # parse HTML using Beautiful Soup
        # returns a `soup` object
        soup = BeautifulSoup(response.content)

        # find all the posts in the page.

        # find each list element belonging to class ol[number]

        most_popular=soup.find_all('li',{'class': re.compile('ol[0-9]')})

        #article group is 0 = shared 1 = most read 2 = audio / video
        #story number is the number assigned to the article (the number next to the title)

        article_group = 0
        
        last_story_number = 0

        # set up prettytable for nice printout
        x = PrettyTable(["Days Old", "Group Name", "Story Number", "Story Title", "URL", "Present D/T", "Original Pub Date", "Orig Pub Date (front)"])
        x.align["URL"] = "l"            # Left align URL
        x.align["Story Title"] = "l"    # Left align story title
        
        for each_story in most_popular:

            # store the URL of each story
            link = each_story.find('a').attrs['href']   
            print link

            # get the story title
            story_title = each_story.find('a').text 
            story_title = re.sub(r'^\W*\w+\W*', '', story_title)

            # get the story number (thats listed next to the article)
            story_number = each_story.span.text   
            story_number = re.sub(':\s*',"",story_number)

            # work out which group it belongs to
            if (abs(last_story_number-int(story_number)) >= 4)   :
                article_group += 1
                
            last_story_number = int(story_number)
            
            if (article_group == 0) :
                article_group_name = "Most Shared"
            elif (article_group == 1) :
                article_group_name = "Most Read"
            else :
                article_group_name = "Audio / Video"

            
            # if its an audio / video story that uses javascript to generate the time relations, ignore
            
            if article_group != 2:
                # retrieve the missed connection with requests
                response = requests.get(link)

                # Parse the html of the missed connection post
                soup = BeautifulSoup(response.content)

                # get the original publication date from the meta tags
                orig_pub_date_meta =  soup.find(attrs={"name":"OriginalPublicationDate"})['content']

                # get the original publication date from the front end (new BBC site)
                try:
                    orig_pub_date_front = soup.find('span',{'class':'story-date'}).text
                    orig_pub_date_front = datetime.strptime(orig_pub_date_front, "\n%d %B %Y\nLast updated at %H:%M\n")
                except AttributeError:
                    print "Old site - set front date to meta date"
                    orig_pub_date_front = orig_pub_date_meta
                    
                    
                # calculate the difference between original publishing date and time of checking
                orig_pub_date_meta = datetime.strptime(orig_pub_date_meta, "%Y/%m/%d %H:%M:%S")
                days_difference = abs((datetime.now()-orig_pub_date_meta).days)

                #"Days Old", "Group Name", "Story Number", "Story Title", "URL", "Present D/T", "Original Pub Date", "Orig Pub Date (front)"
                x.add_row([days_difference,article_group_name,story_number,story_title,link,datetime.now().strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_meta.strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_front])
                #"Check TS", "Panel", "Title", "URL", "Pub Date Meta", "Pub Date Front"
                f.writerow([days_difference,article_group_name.encode('ascii','ignore'),story_number.encode('ascii','ignore'),story_title.encode('ascii','ignore'),link,datetime.now().strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_meta.strftime('%Y/%m/%d %H:%M:%S'),orig_pub_date_front])

        # print the nicely formatted table that PrettyTable generated, once all stories collected
        print x
        
    except requests.exceptions.RequestException as e:    # This is the correct syntax
        print e



if __name__ == '__main__':
    

    f = csv.writer(open("bbc_popular_stories.csv", "wb"))
    f.writerow(["Days Old", "Group Name", "Story Number", "Story Title", "URL", "Present D/T", "Original Pub Date", "Orig Pub Date (front)"]) # Write column headers as the first line

    while True:
        scrape_bbc_popular_stories()
        print "Waiting for 5 minutes...."
        time.sleep(300)

Running this in IDLE shell looks like this:

and some of the ‘old’ results collected over a 3 hour period can be seen in this spreadsheet . Out of 46 news stories listed under popular stories during this period, 21 were over 67 days old!

Days OldStory TitleURLPresent D/TOriginal Pub Date
281Rent ‘unaffordable’ in third of UKhttp://www.bbc.co.uk/news/business-2327344823/04/2014 00:0115/07/2013 06:35
852O’Donnell warns of UK challengeshttp://www.bbc.co.uk/news/uk-politics-1629542123/04/2014 00:0122/12/2011 18:10
67Pension system ‘is not working’http://www.bbc.co.uk/news/business-2617811323/04/2014 00:1114/02/2014 11:31
593Union joint strike action warninghttp://www.bbc.co.uk/news/business-1951419523/04/2014 00:2606/09/2012 23:40
854Government outlines pension dealhttp://www.bbc.co.uk/news/business-1625923823/04/2014 00:3620/12/2011 22:20
244Millions ‘worse off’ on new pensionhttp://www.bbc.co.uk/news/business-2377032723/04/2014 00:4621/08/2013 12:35
508Plain packs for Australia smokershttp://www.bbc.co.uk/news/world-asia-2055958523/04/2014 01:1201/12/2012 01:08
509Energy Bill for ‘cleaner economy’http://www.bbc.co.uk/news/business-2053998123/04/2014 01:1729/11/2012 15:14
615Australia court backs tobacco lawhttp://www.bbc.co.uk/news/business-1926424523/04/2014 01:3215/08/2012 01:56
887Doctors call for car smoking banhttp://www.bbc.co.uk/news/health-1574435223/04/2014 01:5217/11/2011 17:10
435Bedroom tax’s’ impact on the northhttp://www.bbc.co.uk/news/uk-england-tyne-2141282623/04/2014 02:0211/02/2013 14:01
317Labour ‘would cap welfare spending’http://www.bbc.co.uk/news/uk-politics-2278528223/04/2014 02:0709/06/2013 16:37
323Ed Balls seeks to restore Labour’s economic credibilityhttp://www.bbc.co.uk/news/uk-politics-2275304023/04/2014 02:1703/06/2013 13:24
281Benefit cap ‘leads to more in work’http://www.bbc.co.uk/news/business-2330609223/04/2014 02:1715/07/2013 17:51
818Britons ‘becoming more dishonest’http://www.bbc.co.uk/news/uk-1671487223/04/2014 02:4325/01/2012 08:54
218Benefit cheats face 10-year termshttp://www.bbc.co.uk/news/uk-2410474323/04/2014 02:4816/09/2013 12:49
811Family life on benefitshttp://www.bbc.co.uk/news/uk-1681218523/04/2014 02:5301/02/2012 12:47
135Most people in poverty are ‘in work’http://www.bbc.co.uk/news/uk-2528706823/04/2014 03:2308/12/2013 21:49
187Work ‘may be no way out of poverty’http://www.bbc.co.uk/news/uk-politics-2455361123/04/2014 03:2817/10/2013 14:59
692Professions ‘must be more open’http://www.bbc.co.uk/news/uk-politics-1825421923/04/2014 03:3930/05/2012 15:32
309Top unis ‘now less socially diverse’http://www.bbc.co.uk/news/education-2291260923/04/2014 04:0417/06/2013 07:38