20/03/2016

Rough guide to building UK election maps for D3.js

During my time at CITY A.M. I helped to build an interactive British General Election Map. This blog outlines how I created it.

Let's see the map then

Here's a simplified version of the map, hosted at Codepen.

See the Pen D3 Election Map by Stephen Scott (@sassquad) on CodePen.


Here, the map looks rather small, as it has been resized to fit the page. You can use your mouse wheel (or double-click/double-tap) to zoom in or out of the map, and click constituencies to view more detailed results on the left hand side. Each constituency is coloured according to party livery.

This blog post gives a very general guide as to how that map was created. I'll be adding further detail about the creation of the map in subsequent posts.

Acknowledgements

This project was the work of several people, in addition to myself. Here are the other former colleagues who collectively made the election map possible:
  • Michael King (back end programming of data capture system)
  • Billy Ehrenberg (for D3.js support and data journalism expertise)
  • Emma Haslett (who toiled through the night gathering the results)
  • Chris Parameter (design and feel of interactive visuals) 
  • Ben Barden (project management) 
Thank you all!

Disclaimer

There's a couple of things I need to mention beforehand:
  • A fair amount of time has passed since this project was completed, so forgive me if I appear to have skipped over the details. 
  • Early in March 2016, the CITY A.M. website was relaunched, so the election map doesn't display as well in the new layout. This is partly why I created the Codepen version.

Introduction

Many news organisations - such as BBC News or The Guardian - put a lot of resources into their election coverage, both offline as well as online. Way back in 2010, Adobe Flash was primarily used to produce interactive mapping solutions. Now we have at our disposal a wider range of tools with which to produce compelling content, without resorting to plug-ins.

To produce our map, we needed the following:
  1. a suitable back end with which to assimilate results data
  2. a map of all the 650 parliamentary constituencies which comprise England, Wales, Scotland and Northern Ireland.
  3. a way to bind the data to the map
  4. make the election map usable on mobile devices.
That's no mean feat. There's a lot of moving parts. This article will be a general overview. I'll then go into further detail in subsequent posts.

In the end the following technologies were used, in no order of importance:

D3.js

D3 - or Data Driven Documents - is a JavaScript library that binds data to elements of the Document Object Model (DOM) that are present - or yet to be created - on your web page. It's a piece of magic. You give it data, and inform it as to how you want the data to be displayed visually. D3.js then does the rest.

First released back in 2011 by Mike Bostock, it has grown exponentially in its popularity. The website alone is worth a visit, to see the myriad visualisations that are possible. It practically invented the recent career of data journalism, and lead to renewed interest in data science. The gradual open sourcing of governmental data and statistics has made the visualisation of previously impenetrable data far more compelling, and can reveal trends that would never have been seen previously.

D3.js follows web standards, and no proprietary solutions, so is extremely adaptable. It offers a deceptively simple syntax, but is more than capable of producing extremely compelling and beautiful data visualisations.

Building a map

For mapping, D3 supports various projections, in data formats such as GeoJSON or TopoJSON. If you care about file size or topology, then use TopoJSON. Otherwise, use GeoJSON.

Playing with map data is actually rather involved. Fortunately, Mike Bostock's D3.js article "Let's make a Map" is well worth a read. It happens to use the UK and Ireland as an example map, so some of the technical detail described within is very useful! My only problem was that I was using Windows to develop the map. Therefore, references to Homebrew had to be replaced with Windows equivalents, which proved to be a bit of a minefield.

However, most available mapping data comes in ESRI Shapefile format, for use in geographical mapping software, such as QGIS.  So, we need to get hold of this data. As Bostock's article attests, Natural Earth provide some great geographical data provided entirely by voluntary contributors.

The primary source of mapping data for parliamentary constituencies is the Ordnance Survey. Under their OS OpenData resource, you can download their datasets for free, under an open source licence. For authoritative data such as elections, it's important to get data from an official source.

However, the first thing to realise is that Northern Ireland mapping data is owned by a separate service - the Ordnance Survey of Northern Ireland. We therefore need to obtain maps from both services, then combine them. 

This is where the QGIS software comes into its own. QGIS is a hefty piece of software, and is well beyond the scope of this article, but is available for all major platforms as a free download. With QGIS, we can import both Shape files. We then export a new Shape file of the combined data. Once that is done, we can finally import the file into the Mapshaper.org site, and export a more simplified GeoJSON file, ready for our purposes.

The building of the map is the part of the journey I've had to skip through for the time being. I'll get into the nitty gritty of this in a future blog post.

Data binding

In order to marry the constituencies to the election result data, an unique identifier was required. In this case, the common element was that used by the Press Association. This id is a simple integer value, which enabled d3 to bind the data more quickly.

The main nub of the code is this:

queue()
    .defer(d3.json, "map file")
    .defer(d3.json, "election data")
    .await(ready);
...
function ready(error, uk, boundaries) {
    mapFeatures = topojson.feature(uk, uk.objects.subunits).features;
    var map = mapContainer
      .append("g")
      .attr("class","subunits")
      .selectAll("path")
      .data(mapFeatures),
      constituency = boundaries.data,...

The queue command allows for the loading of large datasets asynchronously. The await handler has a single argument, the function that builds the map, called ready(). The ready() function then maps the election result data to the constituency subunit features. 

The function call has three arguments, error, uk and boundaries. The last two correlate to the data sources referred to in the defer() handlers of the queue() command. In this instance, uk maps to the map file, while boundaries maps to the election data. The map variable is an instance of the SVG file which will contain all of the map data and election data combined.

Mobile and Tablet

As the election map had to be usable on mobile devices, the map file used on the webpage had to be simplified further. SVG files render very poorly on tablets and mobile devices, so the mapfile had to be reducd in size and compleity, a function the Mapshaper.org site offers. CSS breakpoints were then used to serve mobile and tablet specific versions which would work accordingly. D3.js supports touch and swipe events as standard.

Again, this will need to be the subject of another post, as it requires deeper explanation than can be provided for the time being.

Conclusion

This election map solution is adequate, but is merely one approach. There are sure to be other ways of producing a data visualization such as this. Much of the code was put together fairly quickly, with various iterations performed in initial research and discovery phase. Added complexity came in the form of mobile and tablet support. Much throttling had to be performed to prevent slow performance. SVG files don't work too well if they are too complex in structure and require features such as pan and zoom.

The D3.js library, and it's accompanying Queue library are available as Node NPM modules, which can be beneficial in preparing server side mapping solutions. The JavaScript itself is very messy in this implementation, and can be clearly improved.

As mentioned, I'll write further, more comprehensive accounts on some of the above in due course. Thanks for reading.

18/03/2016

Building BBC Micro game disc images using Linux


I wrote this following blog post around Christmas 2015, during that time I posted various BBC Micro games running via the JavaScript based jsBeeb emulator onto Facebook. These games were typically buried within compilation discs, so were not easy to access. Although jsBeeb provides an auto boot facility, you still had to go through any menus provided on the disc in order to get access to said game. This post details how you can build your own images, via a special tool, available to build on all platforms.

The article follows...

As it is Christmas, there's no better time to stoke the fires of nostalgia, and take part in some retro gaming. In the past, a fair amount of effort was required to satisfy your desire for playing of games of yesteryear. Thankfully, browser technology and the incredible abilities of the JavaScript language now make it much easier than ever to play old games.

The Internet Archive, for example, have their MS-DOS Games Archive, which contains a huge cache of games, some well known, others less so.

My particular area of interest is the BBC Micro. For those of you who are unaware, the BBC Micro was a British built computer, made by Acorn Computers. These machines made it into many of Britain's schools during the 1980s and 90s, and in some countries abroad. Acorn also created the ARM processor, which is now used in just about any electronic device in existence.

In any case, the Beeb has a great collection of games, most of which are now playable on jsBeeb, a JavaScript based emulator, created by Matt Godbolt. If you visit bbc.godbolt.org, and run the default disc image, you'll be able to play the original release of Elite, one of the most famous games of all time. This is Elite Dangerous' granddaddy!

Although jsBeeb has a link to various commercial arcade game images, provided by the Stairway to Hell website, you can in fact, boot up your own disc images. These files are usually zipped, and contain either an SSD or DSD file (single sided and double sided discs). These are based on the 5 ¼” floppy discs, that made great frisbees in their time, but could impale itself in necks if you were not careful.

One source of these great games are from the pages of computer magazines of the day. For the Beeb, we had titles such as A+B Computing, Acorn User, The Micro User and Electron User. If you wanted, you could type the games listings yourself, which could take some hours to do, or you could subscribe, and receive a disc in the post. These discs are what you now find on the Internet as zipped image files.

Although jsBeeb can automatically boot these discs for you, many of the discs required some further interaction before you could find the games in question to play. This inconvenience, along with the complexity of tools to edit/create these image files, made some of these games unreachable to a potential new audience.

So, I thought, why not bury myself in these tools, and create some brand new disc images, which could then auto boot and run the game straight away?

As I am using Linux, the best tool by far for disc image creation and editing is Walter Schouten's bbcim. Conceived in the mid 1990s, Walter revised the code in 2014, making it easier to compile and use for newer flavours of Linux, or any Unix based system.

[Note: since the time this was written, Schouten's website has been taken offline. The most recent archived version of the site does not include links to the binary for bbcim. I've uploaded a gzipped tarball of version 1.0 of the software from my website. You can download it here.] 

Following his sparse but functional instructions, I was able to create the binary executable, ready for use on the command line. First, I downloaded a disc image from the 8bs website, one of the subscription discs for The Micro User magazine. I was after a particular festive game, called Revenge of the Snowmen, from the January 1990 issue. We're talking about a game over 25 years old. But great games never age, right?

Written in BBC Basic by Jon Perry and Pali Rao, it made a big impression on me, in terms of how it worked. As a two player game involving the throwing of snowballs at each other, it used some fancy maths to calculate speed and trajectory of your snowballs. It's a great example of what BBC Basic, without the use of assembly language, could muster. The game uses brightly drawn, colourful graphics, and is actually great fun to play. A perfect candidate for festive fun.

Having obtained the image, and confirmed that it worked in jsBeeb, the time came to extract the game, and create a 'boot' file (called !BOOT) so it could auto run via a customised URL.

A fair amount of trial and error was required here. Building the binary was quite easy, the instructions were adequate. I did have to close my terminal and reopen it though for it to run.

To run it, I typed ./bbcim, which showed the copyright message, and asked for an option. The supplied instructions provide details of all the various commands and arguments you could use to create and modify images, plus extract files.

The disc image, called tmu90-01.ssd, could be inspected via:

./bbcim -c tmu90-01.ssd

This listed the image contents, and there it was, the game, saved as the filename $.revenge.

Now, to create a new image, I used:

./bbcim -new Revenge.ssd

A nice new disc image to fill with the game and a boot file.

Now, to extract the revenge file, I used this command:

./bbcim -e tmu90-01.ssd $.revenge

This extracted the file (saving it as tmu90-01.ssd.$.revenge), and also created a .inf file, which contains meta data required when merging the file into a new image. Now, to add the file to the new image:

./bbcim -a Revenge.ssd tmu90-01.ssd.$.revenge

A quick issue of the ./bbcim -c Revenge.ssd revealed that the game file was now present in my new disc image. Great!

Now, to make it bootable. This proved a little more hairy. You see, the fact that the boot file is called $.!boot plays havoc with the Unix command line. If I did:

./bbcim -e tmu90-01.ssd $.!boot

I would get the error message bash: !boot: event not found.

Damn!

I did a search on Google, and found the answer, I merely had to escape the ! with a backslash, thus:

./bbcim -e tmu90-01.ssd $.\!boot

This extracted the boot file successfully. Now of course, I had to edit it. I was initially hesitant, how would I edit the boot file on Linux? Turns out, I only had to open the !boot file in a text editor such as Gedit, and change the filename to “revenge”. Saved, it was done.

Now all I had to do, was one final command:

./bbcim -a Revenge.ssd tmu90-01.ssd.$.\!boot

This copied the boot file into the disc image. The new disc image was now complete.

Having moved the SSD file to my web server, I put together the URL to jsbeeb, complete with the URL of the image file, plus the &autoboot query string:

http://bbc.godbolt.org/?disc=http://www.sassquad.com/downloads/Revenge.ssd&autoboot

The image file now boots successfully, and the game is ready to play.

Brilliant!

I leave you with some choice links to the games I was able to bring to a new audience. Enjoy!

10/03/2016

Test your website in IE using Linux with Vagrant/Virtualbox

I've had an on-off relationship with Vagrant, having first used it for a short while at a previous job. However, in my current employment, we primarily use Linux based laptops, while a select number use Macs. Either way, testing our website developments in Internet Explorer is an added, but necessary headache.

However, Vagrant is actually quite brilliant for not only hosting development environments with ease, but for also hosting flavours of Windows, all with their different combinations of Internet Explorer. I remember back in 2010, using the likes of IETester to install multiple versions of IE within Windows. It was messy and unreliable.

Back to 2016, Vagrant works with Virtualbox to bring you multiple virtual machines that allow you to test your work easily.

I'll skip over installation of Vagrant and Virtualbox. Just visit their respective websites, and install as required.

If you're intention is only to test one version of IE, you could simply visit modern.ie and download a suitable box to your machine. Or, you could use this gist to grab all of the vms. This link contains a Vagrantfile to save into a given folder on your machine. I've set up a folder called windowsvms, and saved the Vagrantfile there.

I then access my terminal, and cd windowsvms, then type the following:

IE=Win7IE8 vagrant up

Assuming you have a decent internet connection, the box will get downloaded and unpacked to this location. Once everything has completed, Virtualbox will appear to boot your new virtual machine.

By default, you can browse the internet in your guest VM. However, you need to find out how to access your website running on the host. Assuming you have a site running on localhost, on Linux you can use the ifconfig command to get details of your host ip address.

Type ifconfig, and look for the inet addr under usually wlan0 (for wireless connection) – it will look like 10.0.2.110. You can test this by going to your web browser, and replacing 'localhost' part of the address with the ip address. You should still get your website.

Now, check your network connection in the Virtualbox menu. Look under Devices > Network > Network Settings. You should have only one adapter active. If not, then ensure it is enabled and set to NAT, with Cable Connected. Close the window, go to the Start menu, and type cmd, to bring up the command prompt.

We now need to find out the gateway ip address so we can view the website in IE. Type ipconfig, and look up the details for something like Ethernet adapter Local Area Connection 1. Find the IP address under the default gateway, it should resemble 10.0.2.2 or similar. This is the address we need.

Now open IE in your guest VM, and input the following URL:

http://10.0.2.2/website

replace website with whatever your address is, and you should now see your website within IE.

Once you get the hang of it, Vagrant and Virtualbox are an ideal set of tools in your armoury of browser testing.

09/03/2015

I've gone purple

After a good 8 years or so, I finally gave my site a reasonably good going over, and given it a distinctive purple appearance. I'm now wondering whether I've unwittingly given it the UKIP purple. I hope I haven't.

I've not had to change too much of the underlying HTML, but have given the CSS some extra love, including provision for a mobile and tablet version. It's by no means perfect, but it's certainly more pleasing on the eye than the old site. It wasn't great on mobiles at all. It was also, unwittingly painting a picture of someone who wasn't up to date with his front end skills. 

I've also had the chance to revisit some sections, and revise the text. You now know where I work, for instance. But you can also visit the profile page on my employers website, where you can see some of the d3.js work I've been specialising in this year.

Even the Games section has had content given greater prominence. Now that BBC Micro games can be played in the browser (thanks to Matt Godbolt's jsBeeb emulator), I've given the two games I wrote for it greater exposure. There's no excuse now to not cringe at my attempts at game programming on an 8-bit computer!

One thing I will look at is splitting the content on the RISC OS page into separate pages. It's rather long, and unfair on mobile devices (YouTube videos and images add to the pain when viewing the site on an intermittent connection, like on a train, for example). I'm sure there are other parts of the site that require some TLC, which I'll make a point of fixing in the near future.

Hope you enjoyed reading that - needed an excuse to update my blog after a year!

03/04/2014

Blowing off the dust

It's been a while since I last posted. The problem with blogs is that I always feel you have to write something worthwhile before posting. And the truth is, nothing much happens that can't be put into a tweet or a Facebook posting.

Today, I'm trying something different. I've had my brothers iPad for some months now, usually the kids are playing on it. Today I decided to take it with me to work. I then found out how to connect it to my Blackberrys 3G connection, giving me internet access, and therefore some time on the train, not distracted by homelike and the chaos which can and does ensue with young children.

After some months I am beginning to enjoy using the iPad. I viewed it with trepidation, wondering if the kids were going to be zombified with it. On some days, that fear has been realised! With great toys comes great parental responsibility. But for doing quick things like eBay, Skype, or even submitting meter readings, it's great.

Now I've got Twitter and Facebook on the iPad, I am most likely to be leaving bits and pieces over there. My website has a basic feed, but at some point, will be replaced by the new one I cobbled together using a template. I'm too busy to make my own!

That's all for now. Hopefully it won't be another two plus years before my next entry :-)

20/09/2011

RISC OS stuff for sale (or to give away)

I did have items of RISC OS software and books to sell or give away. As of March 2012, all these items have now been taken.

Thanks to everyone who expressed interest, and collected my items.

16/08/2011

BBC stuff (sold)

BBC Micro, TV and Cumana Floppy Disc Drive Combo - sold!

  • BBC Micro Model B (32K, with Watford Electronics Sideways RAM, power supply replaced early 2011)
  • Ferguson TX 14" colour TV (aerial broken, front door broken, but otherwise great for the Beeb)
  • Cumana 5 1/4" floppy disc drive
  • Various BBC games tapes and educational software (listed below).
  • 36 Challenging Games for the BBC Micro
  • Building Blocks for BBC Games
  • a handful of other books for the Beeb, of games listings.
  • Let's Compute! (all 12 issues, 1990-1991)
  • As the power supply cost about £25 to replace, I would expect offers for this to be at least above this amount.
This whole set has now been sold on eBay (http://www.ebay.co.uk/itm/260852403866). Thanks for all your interest; I'm sure it will now go to a good home!

08/07/2008

Ballantines Whisky

One of my more recent projects has finally come to fruition - a touchscreen kiosk for Ballantines Whisky. This kiosk features a webcam for users to take their picture, then incorporate onto a postcard design, for printing out, via the kiosk printer.

Work began late last year, and after jumping through many hoops and wiring (both metaphorically and physically) many disparate elements together, the kiosk arrived at its first venue of Madrid airport on 1st June 2008.

Every month or so, the kiosk will be moved to a different airport around Europe. The next destination is Frankfurt, after that, who knows?

I have to say I'm pretty proud of what was achieved. The main front end is done in Flash, while under the hood, PHP, MySQL and ImageMagick allow the picture to be taken, saved, and manipulated, thanks to the special effects and color filters that are available to the user.

The tricky part was translating a screen capture into a 300dpi print job. That amount of pixel wrestling takes time to assemble, so it had to be ready in time for the screen in which printing begins. A last minute bodge to insert the webcam at a 90 degree angle, due to lack of room in the kiosk meant scouring the Actionscript references for a quick way of re-orienting the webcam image back to its proper viewpoint onscreen.

If you're travelling around Europe, keep an eye out for a large blue kiosk, it looks a bit like the Tardis, from a distance. You'll only find it airside, so you'll need to be partaking a bit of air travelling. My website has some screen captures from the kiosk, I'll soon get some photos of the kiosk itself soon.

24/04/2008

Ruby on Rails

I've often considered whether to get into Ruby On Rails. Back in December, it reached version 2. I had a dabble with it over a year ago, and found it to be quite amazing. However, I lost enthusiasm for it, when I learned that my employer's hosting provider had no plans to support it.

Are there any good hosting providers out there for RoR?

15/04/2008

Ah wow - Blogger is great now

Wow, I haven't touched my blog in centuries. But what changes Blogger has done in my absence. A lot of the niggles I had concerns about have been addressed, especially how the archive is displayed down the right hand side.

So much has happened since last year. I am now a proud father - my son was born in May 2007, he's a little smiler! I'll put some pictures up at some point. You'll get to see them in that handy little photo panel on the right.

During November 2007 I spent some time getting Cavern Duel to work on 32-bit RISC OS platforms. I haven't been entirely successful as yet, should have something ready during the summer of 2008, time as always, is permitting, between fatherhood and project management.

As for work matters, my skills in PHP and Flash have been subjected to a very steep learning curve. As a result, I've been working with clients such as HMV, Communities and Local Government, Liquid Nightclubs and more, devising kiosks and intranets. Some very interesting stuff coming during 2008, which I can't reveal yet. Watch this space.

Hmm... I say watch this space. But as I've proved recently, blogging can be a love or hate affair. It's only as good as you want it to be.