Feed: rss rdf opml

November 29, 2015

Ben FowlerMaths and physics

One of the frustrations of doing physics at Birkbeck and then the Open University, is that they modularize the course in such a way, that the maths is split from the physics so they can present to a wider audience, purely for commercial and practical reasons. This forces the physics to be taught without the absolutely necessary maths it needs to do it proper justice.  I'd argue that the physics

November 29, 2015 10:59 PM

Ben FowlerSimple disk profiling under Linux

I have a NAS based on an ancient HP MediaSmart, that's had Windows Home Server wiped off it and replaced with Debian. It has a nice compact case, where the front is taken up with four hotswap cradles for 3.5" hard drives. Concerned that the disks were too slow, I asked around the HUMBUG mailing lists, and Paul Gearon replied, suggesting that I check out hdparm(1) and lsscsi(1) to see if

November 29, 2015 12:08 PM

November 28, 2015

Ben FowlerClean Code

Being through a redundancy and having now found myself in a new job where development practices are quite mature, I've been thinking much harder about the way I practice software development. I had noted that the development culture of the office is heavily influenced by the writings of Joel Spolsky and Bob Martin, and I've noticed a few dog-eared copies of Clean Code floating around. So my

November 28, 2015 01:32 PM

November 26, 2015

Ben FowlerRESTful API design

My colleague at work passed this onto me as part of some work we've kicked off. This is the best set of guidelines I've seen for doing JSON RESTful web services so far. Vinay Sahni: Best Practices for a Pragmatic RESTful API I obviously have been out of the loop, because the Link: header (RFC 5988) is news to me. As is the dirty rotten hack, which is JSONP.

November 26, 2015 02:16 PM

November 21, 2015

Ben FowlerUsing a Mac for serious development

One of the nicest things about the new job, is that we have been all given Macs for development. I have quite a comfortable setup: a maxed-out Mac Mini, with two 24" widescreen monitors. Despite being a Unix-head since I first had the chance to get my hands on Unix-powered machines, I've always found myself running Windows for development. Now actually finding myself having to code for a living

November 21, 2015 12:23 AM

November 19, 2015

Ben FowlerMy new gig

I probably didn't mention that I've had a change of scenery. I'm now working in a new role back at Thomson Reuters, building some neat new web applications. The three-month sabbatical was nice, but it was high time to tackle new challenges. Here's a video demo of the new hotness: World-Check One. It's throughly modern, massively scalable, and releases every few days. The difference between this

November 19, 2015 10:12 PM

November 14, 2015

Ben FowlerGitflow in 2015: no longer recommended?

ThoughtWorks have just released their 2015 technology radar, and it looks like consensus is building against gitflow -- the elaborate branching model invented as a set of best practices for using Git in big application development, where big releases are the norm. We firmly believe that long-lived version-control branches harm valuable engineering practices such as continuous integration, and

November 14, 2015 02:07 PM

November 13, 2015

Daniel DevineCentOS 7 Courier Repo & Work on Self-hosted Email

I've created a repository with Courier RPM packages built for CentOS 7 x86_64. The packages are compiled and signed automatically in a toolchain I have automated - but no particular reseach, tweaks or testing has been done to ensure that they are suitable for deployment. I created these packages as I want to work on Ajenti-V's Mail plugin and my platform of choice for servers is CentOS which does not have Courier packages it's repos and neither does EPEL.

Read more…

November 13, 2015 11:42 AM

November 11, 2015

Ben FowlerNew server build

I have a mind to upgrade my dinky old home server to something with a bit more grunt. At the moment, I have a HP MediaSmart headless media server, which was wiped and reinstalled with Debian, and is fully populated with big WD green drives. However, this little machine, while doing its job, has problems. Being such an underpowered little machine, it won't run ZFS well (at all?), and without

November 11, 2015 04:04 PM

Ben FowlerNewbie mistake: "nofail" and removable disks

I lost half an hour today because of a silly mistake. When adding removable disks to /etc/fstab under Linux, don't forget the nofail option. It's rather annoying when you reboot the system and then can't get into the machine remotely because a removable disk is missing. It's even more or a ballache, if said system is completely headless and doesn't have a VGA or keyboard port default (or

November 11, 2015 02:10 PM

November 09, 2015

Ben FowlerLinus Torvalds and that notorious code review

This happened recently: Christ people. This is just sh*t.The conflict I get is due to stupid new gcc header file crap. But whatmakes me upset is that the crap is for completely bogus reasons.This is the old code in net/ipv6/ip6_output.c:mtu -= hlen + sizeof(struct frag_hdr);and this is the new "improved" code that uses fancy stuff that wantsmagical built-in compiler support and has silly

November 09, 2015 03:41 PM

November 08, 2015

Ben FowlerBackups

My home computing setup has a Mac laptop and a Linux-based file server. Apart from using a server-hosted Time Machine backup which is brittle and occasionally fails, I have no backups. Seeing as I'd hate to lose all my wedding photos, I decided to sort this out today. On the hardware side, I bought a hard drive docking station, and plugged it into the server. Then I bought myself a pair of

November 08, 2015 09:31 PM

November 04, 2015

Ben FowlerMy LaTeX workflow

Working on my first assignment for "S217: Physics: from classical to quantum". Since I've got time on my hands, I've typeset it using Latex using a nice template (the jhwhw problem-sheet style), matplotlib to generate graphs, and Inkscape for the diagrams. TeXStudio is also very nice, for the stuff it gives you out-of-the-box (that'd otherwise take me hours to configure in Vim, my usual text

November 04, 2015 11:22 PM

Ben FowlerSwing

Swing. Just tried coding a toy example today -- a crude UI to calculate my GPA and degree classification for the degree I'm doing. I remember being shown it by Martin Pool as a technology preview back when it was the New Hotness, back in 1998, if memory serves correctly. It got Old and Busted in record time. Too hard to use correctly, too easy to use incorrectly, verbose as hell. IntelliJ

November 04, 2015 11:14 PM

October 26, 2015

Ben MartinESP8266 and a few pins

The new Arduino 1.6.x IDE makes it fairly simple to use the ESP8266 modules. I have been meaning to play around with a some open window detectors for a while now. I notice two dedicated GPIO pins on the ESP8266, which is one more than I really need. So I threw in an led which turns on when the window is open. Nothing like local, direct feedback that the device has detected the state of affairs. The reed switch is attached on an interrupt so as soon as the magnet gets too far away the light shines.

I will probably fold and make the interrupt set a flag so that the main loop can perform an http GET to tell the server as soon as it knows when a state has changed.

Probably the main annoying thing I've still got is that during boot it seems the state of both the gpio pins matters. So if the reed switch is closed when you first spply power then the esp goes into some stall state.

It will be interesting to see how easy OTA firmware updates are for the device.

October 26, 2015 12:41 PM

October 15, 2015

Ben MartinTerry & the start of a video project.

I did a test video showing various parts of Terry the Robot while it was all switched off and talking about each bit as I moved around. Below are some videos of the robot with batteries a humming and a little movement. First up is a fairly dark room and a display of what things look like just using the lighting from the robot itself. All the blinking arduino LEDs, the panel, and the various EL and other lights.

<iframe allowfullscreen="" frameborder="0" height="315" src="" width="560"></iframe>

The next video has a room light on and demonstrates some of the control of the robot and screen feedback.

<iframe allowfullscreen="" frameborder="0" height="315" src="" width="560"></iframe>

I got some USB speakers too, but they turned out to be a tad too large to mount onto Terry. So I'll get some smaller ones and then Terry can talk to me letting me know what is on its, err, "mind". I guess as autonomy is ramped up it will be useful to know if Terry is planning to navigate around or has noticed that it has been marooned by a chair that a pesky human has moved.

The talk over video is below. I missed talking about the TPLink wifi APs and why there are two, and might be only one in the future. The short answer is that Terry might become a two part robot, with a base station only one wifi AP is needed on the robot itself.

<iframe allowfullscreen="" frameborder="0" height="315" src="" width="560"></iframe>

October 15, 2015 12:27 PM

October 10, 2015

Blue HackersWorld Mental Health Day 2015

On this year’s World Mental Health Day, some info from Lifeline and Mental Health Australia:

Mental Health Begins with Me

Did you know 70% of people with mental health issues don’t seek help? […] As a community we can encourage others to take care of their mental health by breaking down the barriers that stop people seeking help when they need it.

How can you help?

Make your mental health promise and share it today.  Encourage your friends and family to do the same and share their promises here or on social media using the hashtag #WMHD2015.

Here are some great tips and promises to make to yourself this 10/10 (October 10th):

  1. Sleep well
  2. Enjoy healthy food
  3. Plan and prioritize your day
  4. Tune into the music you love
  5. Cut down on bad food and booze
  6. Switch off your devices and tune out
  7. Hangout with people who make you feel good
  8. Join in, participate and connect
  9. Exercise your body and mind
  10. Seek advise and support when you need it


October 10, 2015 01:28 AM

October 08, 2015

Adrian SuttonUse More Magic Literals

In programming courses one of the first thing you’re taught is to avoid “magic literals” – numbers or strings that are hardcoded in the middle of an algorithm. The recommended solution is to extract them into a constant. Sometimes this is great advice, for example:

if (amount > 1000) {

would be much more readable if we extracted a ADDITIONAL_AUTHORIZATION_THRESHOLD variable – primarily so the magic 1000 gets a name.

That’s not a hard and fast rule though.  For example:

value.replace(PATTERN_TO_REPLACE, token)

is dramatically less readable and maintainable than:

value.replace("%VALUE%", token)

Extracting a constant in this case just reduced the locality of the code, requiring someone reading the code to unnecessarily jump around the code to understand it.


My rule of thumb is that you should extract a constant only when:

Arbitrary tokens like %VALUE% above are generally unlikely to change – it’s an implementation choice – so I’d lean towards preserving the locality and not extracting a constant even when they’re used in multiple places.  The 1000 threshold for additional authorisation on the other hand is clearly a business rule and therefore likely to change so I’d go to great lengths to avoid duplicating it (and would consider making it a configuration option).

Obviously these are just rules of thumb so there will be plenty of cases where, because of the specific context, they should be broken.

October 08, 2015 09:40 PM

October 01, 2015

Adrian SuttonTravis CI

Probably the best thing I’ve discovered with my recent playing is Travis CI. I’ve known about it for quite some time, even played with it for simple projects but never with anything with any real complexity. Given this project uses rails which wants a database for pretty much every test and that database has to be postgres because I’m using it’s jsonb support, plus capybara and phantomjs for good measure, this certainly isn’t a simple project to test.

Despite that, getting things running on Travis CI was quick and easy. The container based builds give you a well controlled environment out of the box and they already have a heap of software you’re likely to need ready to go. Other things can be easily provided using a simple .travis.yml file.

While it’s common to have a build server these days, it’s quite uncommon for them to be so well controlled and able to be rebuilt quickly and easily – most wind up being special little snowflakes in some way or other. So building on Travis CI is immediately a massive leap forward in testing practices.

The two things we do at LMAX that I can’t see how to do with Travis CI (yet) is creating a database of historic test results that can be easily queried (at the individual test level, not the build result level), and running a build in parallel with the tests automatically load balanced across a number of hosts. I wouldn’t expect either to be provided as part of the free service anyway but they seem like the next major areas of value it could offer (with significant effort required I imagine).

Still, it’s been a real luxury to have a full CI setup for hacking on a little side project. So thanks for the generosity Travis CI.

October 01, 2015 01:26 AM

September 30, 2015

Adrian SuttonPlaying with Ruby on Rails

I’ve been playing around with ruby on rails recently, partly to play around with rails and partly to take a run at a web app I’ve been considering (which I’ve open sourced because why not?).

It turns out the last time I played with it was back in 2005 and slightly amusingly my thoughts on it haven’t changed all that much. The lack of configuration is still good, but the amount of magic involved makes it hard to understand what’s going on. The ease of finding documentation has improved dramatically – 10 years of blog posts really help. I’m still using TextMate and it’s still annoying that I can’t click a method name to jump to it’s definition – I hear good things about RubyMine but I’m not keen to invest that kind of money in what may be a very short-lived experiment.

The two big changes are that I’ve got 10 years more development experience and the quality of gems seems to have improved significantly. The extra experience means I pick things up a lot faster, understand them more deeply and am a lot less convinced that the way I’m used to is the only right way. The improved quality of gems makes it far less likely that I’ll waste a heap of time struggling with a poorly written gem and instead can drop it in and see benefits straight away.

While all the hipsters may consider rails old-hat now and moved onto node or go, from what I’ve seen it’s matured extremely well and is in a fairly ideal point in the hype-cycle – still getting a lot of attention, embedded enough to have loads of people with experience in it and code depending on it so it will never go away but old and boring enough to be well documented with good libraries and not changing dramatically every week.

September 30, 2015 10:47 AM

September 27, 2015

Blue HackersInteractive Self-Care Guide

Interesting find:

[…] interactive flow chart for people who struggle with self care, executive dysfunction, and/or who have trouble reading internal signals. It’s designed to take as much of the weight off of you as possible, so each decision is very easy and doesn’t require much judgement.

Some readers may find it of use. I think it’d be useful to have the source code for this available so that a broad group of people can tweak and improve it, or make personalised versions.

September 27, 2015 02:20 AM

September 24, 2015

Blue HackersSuicide doesn’t take away the pain, it gives it to someone else

"Suicide doesn't take away the pain, it gives it to someone else"This is something that I feel quite strongly about. Both of my parents have tried to commit suicide when I was young, at different times and stages of my life. The first one was when I was about 11 and I don’t remember too much about it, there was a lot of pain flying around the family at that time and I was probably shielded from the details. The second parent (by then long divorced from the other parent) tried when I was 21 and away at uni in a different city. That one I remember vividly, even though I wasn’t there.

My reactions to the second were still those of a child. Perhaps when it’s a parent, one’s reactions are always those of a child. For me the most devastating thought was a purely selfish one (as fits a child) “Do I mean that little to them? Am I not even worth staying alive for?” The pain of that thought was overwhelming.

At the time I was young, saw myself as an optimist and simply could not relate in any way to the amount of pain that would bring one to such an action. I was angry. I described suicide as “the most selfish act anyone could do”.

Now decades of time and a world of life experience later, I have stared into that dark abyss myself and I know the pain that leads one there. I know how all-encompassing the pain and darkness seems and how the needs of others fade. An end to the pain is all one wants and it seems inconceivable that one’s life has any relevance any more. In fact, one can even argue to oneself that others would be better off without one there.

In those dark times it was the certain knowledge of that pain I had experienced myself as one (almost) left behind that kept me from that road more firmly than anything else. By then I was a parent myself and there was just no way I was going to send my children the message that they meant so little to me they were not even worth living for.  Although living seemed to be the hardest thing I could do, there was no hesitation that they were worth it.

And beyond the children there are always others. Others who will be affected by a suicide, no matter of whom. None of us is truly alone. We all have parents, we may have siblings. Even if all our family is gone and we feel we have no friends, it is likely that there are people who care. The person at the corner shop from whom you buy milk on weekends and who may think “should I have known? Is there anything I could have done?” Even if you can argue that there is no-one that would notice or care, let’s be frank, someone is going to have to deal with the body and winding up of financial and other affairs. And I’m sure it’s really going to make their day!

Whenever I hear about trains being delayed because of incidents on the track I am immediately concerned for those on the train, not least of all the drivers. What have they ever done to that person to deserve the images that will now be impossible to erase from memory, which will haunt their nights and dark moments and which may lead them to require therapy.

There are many people, working for many organisations, some sitting at telephones in shifts 24 hrs a day, who want more than anything else to help people wrestling with these dark issues. They care. They really do. About everyone.

Help is always available. So let’s all acknowledge that suicide Always causes pain to others.

Need help?

September 24, 2015 04:21 AM

September 22, 2015

Anthony TownsLightning network thoughts

I’ve been intrigued by micropayments for, like, ever, so I’ve been following Rusty’s experiments with bitcoin with interest. Bitcoin itself, of course, has a roughly 10 minute delay, and a fee of effectively about 3c per transaction (or $3.50 if you count inflation/mining rewards) so isn’t really suitable for true microtransactions; but pettycoin was going to be faster and cheaper until it got torpedoed by sidechains, and more recently the lightning network offers the prospect of small payments that are effectively instant, and have fees that scale linearly with the amount (so if a $10 transaction costs 3c like in bitcoin, a 10c transaction will only cost 0.03c).

(Why do I think that’s cool? I’d like to be able to charge anyone who emails me 1c, and make $130/month just from the spam I get. Or you could have a 10c signup fee for webservice trials to limit spam but not have to tie everything to your facebook account or undergo turing trials. You could have an open wifi access point, that people don’t have to register against, and just bill them per MB. You could maybe do the same with tor nodes. Or you could setup bittorrent so that in order to receive a block I pay maybe 0.2c/MB to whoever sent it to me, and I charge 0.2c/MB to anyone who wants a block from me — leechers paying while seeders earn a profit would be fascinating. It’d mean you could setup a webstore to sell apps or books without having to sell your sell your soul to a corporate giant like Apple, Google, Paypal, Amazon, Visa or Mastercard. I’m sure there’s other fun ideas)

A bit over a year ago I critiqued sky-high predictions of bitcoin valuations on the basis that “I think you’d start hitting practical limitations trying to put 75% of the world’s transactions through a single ledger (ie hitting bandwidth, storage and processing constraints)” — which is currently playing out as “OMG the block size is too small” debates. But the cool thing about lightning is that it lets you avoid that problem entirely; hundreds, thousands or millions of transactions over weeks or years can be summarised in just a handful of transactions on the blockchain.

(How does lightning do that? It sets up a mesh network of “channels” between everyone, and provides a way of determining a route via those channels between any two people. Each individual channel is between two people, and each channel is funded with a particular amount of bitcoin, which is split between the two people in whatever way. When you route a payment across a channel, the amount of that payment’s bitcoins moves from one side of the channel to the other, in the direction of the payment. The amount of bitcoins in a channel doesn’t change, but when you receive a payment, the amount of bitcoins on your side of your channels does. When you simply forward a payment, you get more money in one channel, and less in another by the same amount (or less a small handling fee). Some bitcoin-based crypto-magic ensues to ensure you can’t steal money, and that the original payer gets a “receipt”. The end result is that the only bitcoin transactions that need to happen are to open a channel, close a channel, or change the total amount of bitcoin in a channel. Rusty gave a pretty good interview with the “Let’s talk bitcoin” podcast if the handwaving here wasn’t enough background)

Of course, this doesn’t work very well if you’re only spending money: it doesn’t take long for all the bitcoins on your lightning channels to end up on the other side, and at that point you can’t spend any more. If you only receive money over lightning, the reverse happens, and you’re still stuck just as quickly. It’s still marginally better than raw bitcoin, in that you have two bitcoin transactions to open and close a channel worth, say, $200, rather than forty bitcoin transactions, one for each $5 you spend on coffee. But that’s only a fairly minor improvement.

You could handwave that away by saying “oh, but once lightning takes off, you’ll get your salary paid in lightning anyway, and you’ll pay your rent in lightning, and it’ll all be circular, just money flowing around, lubricating the economy”. But I think that’s unrealistic in two ways: first, it won’t be that way to start with, and if things don’t work when lightning is only useful for a few things, it will never take off; and second, money doesn’t flow around the economy completely fluidly, it accumulates in some places (capitalism! profits!) and drains away from others. So it seems useful to have some way of making degenerate scenarios actually work — like someone who only uses lightning to spend money, or someone who receives money by lightning but only wants to spend cold hard cash.

One way you can do that is if you imagine there’s someone on the lightning network who’ll act as an exchange — who’ll send you some bitcoin over lightning if you send them some cash from your bank account, or who’ll deposit some cash in your bank account when you send them bitcoins over lightning. That seems like a pretty simple and realistic scenario to me, and it makes a pretty big improvement.

I did a simulation to see just how well that actually works out. With “Alice” as a coffee consumer, who does nothing with lightning but buy $5 espressos from “Emma” and refill her lightning wallet by exchanging cash with “Xavier” who runs an exchange, converting dollars (or gold or shares etc) to lightning funds. Bob, Carol and Dave run lightning nodes and take a 1% cut of any transactions they forward. I uploaded a video to youtube that I think helps visualise the payment flows and channel states (there’s no sound):

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="375" src="" width="500"></iframe>

It starts off with Alice and Xavier putting $200 in channels in the network; Bob, Carol and Dave putting in $600 each, and Emma just waiting for cash to arrive. The statistics box in the top right tracks how much each player has on the lightning network (“ln”), how much profit they’ve made (“pf”), and how many coffees Alice has ordered from Emma. About 3000 coffees later, it ends up with Alice having spent about $15,750 in real money on coffee ($5.05/coffee), Emma having about $15,350 in her bank account from making Alice’s coffees ($4.92/coffee), and Bob, Carol and Dave having collectively made about $400 profit on their $1800 investment (about 22%, or the $0.13/coffee difference between what Alice paid and Emma received). At that point, though, Bob, Carol and Dave have pretty much all the funds in the lightning network, and since they only forward transactions but never initiate them, the simulation grinds to a halt.

You could imagine a few ways of keeping the simulation going: Xavier could refresh his channels with another $200 via a blockchain transaction, for instance. Or Bob, Carol and Dave could buy coffees from Emma with their profits. Or Bob, Carol and Dave could cash some of their profits out via Xavier. Or maybe they buy some furniture from Alice. Basically, whatever happens, you end up relying on “other economic activity” happening either within lightning itself, or in bitcoin, or in regular cash.

But grinding to a halt after earning 22% and spending/receiving $15k isn’t actually too bad even as it is. So as a first pass, it seems like a pretty promising indicator that lightning might be feasible economically, as well as technically.

One somewhat interesting effect is that the profits don’t get distributed particularly evenly — Bob, Carol and Dave each invest $600 initially, but make $155.50 (25.9%), $184.70 (30.7%) and $52.20 (8.7%) respectively. I think that’s mostly a result of how I chose to route payments — it optimises the route to choose channels with the most funds in order to avoid payments getting stuck, and Dave just ends up handling less transaction volume. Having a better routing algorithm (that optimises based on minimum fees, and relies on channel fees increasing when they become unbalanced) might improve things here. Or it might not, and maybe Dave needs to quote lower fees in general or establish a channel with Xavier in order to bring his profits up to match Bob and Carol.

September 22, 2015 07:46 AM

September 20, 2015

Ben MartinTerry Motor Upgrade -- no stopping it!

I have now updated the code and PID control for the new RoboClaw and HD Planetary motor configuration. As part of the upgrade I had to move to using a lipo battery because these motors stall at 20 amps. While it is a bad idea to leave it stalled, it's a worse idea to have the battery have issues due to drawing too much current. It's always best to choose where the system will fail rather than letting the cards fall where the may. In this case, leaving it stalled will result in drive train damage in the motors, not a controller board failure, or a lipo issue.

One of the more telling images is below which compares not only the size of the motors but also the size of the wires servicing the power to the motors. I used 14AWG wire with silicon coating for the new motors so that a 20A draw will not cause any issues in the wiring. Printing out new holders for the high precision quadrature encoders took a while. Each print was about 1 hour long and there was always a millimetre or two that could be changed in the design which then spurred another print job.

Below is the old controller board (the 5A roboclaw) with the new controller sitting on the bench in front of Terry (45A controller). I know I only really needed the 30A controller for this job, but when I decided to grab the items the 30A was sold out so I bumped up to the next model.

The RoboClaw is isolated from the channel by being attached via nylon bolts to a 3d printed cross over panel.

One of the downsides to the 45A model, which I imagine will fix itself in time, was that the manual didn't seem to be available. The commands are largely the same as for the other models in the series, but I had to work out the connections for the quad encoders and have currently powered them of the BEC because the screw terminal version of the RoboClaw doesn't have +/- terminals for the quads.

One little surprise was that these motors are quite magnetic without power. Nuts and the like want to move in and the motors will attract each other too. Granted it's not like they will attract themselves from any great distance, but it's interesting compared to the lower torque motors I've been using in the past.

I also had a go at wiring 4mm connectors to 10AWG cable. Almost got it right after a few attempts but the lugs are not 100% fixed into their HXT plastic chassis because of some solder or flux debris I accidentally left on the job. I guess some time soon I'll be wiring my 100A monster automotive switch inline in the 10AWG cable for solid battery isolation when Terry is idle. ServoCity has some nice bundles of 14AWG wire (which are the yellow and blue ones I used to the motors) and I got a bunch of other wire from HobbyKing.

September 20, 2015 04:48 AM

September 16, 2015

Ben Martin10 Foot Pound Boots for Terry

A sad day when your robot outgrows it's baby motors. On carpet this happened when the robot started to tip the scales at over 10kg. So now I have some lovely new motors that can generate almost 10 foot pounds of torque.

This has caused me to move to a more rigid motor attachment and a subsequent modofication and reprint of the rotary encoder holders (not shown above). The previous motors were spur motors, so I could rotate the motor itself within its mounting bracket to mate the large gear to the encoders. Not so anymore. Apart from looking super cool the larger alloy gear gives me an 8 to 1 reduction to the encoders, nothing like the feeling of picking up 3 bits of extra precision.

This has also meant using some most sizable cables. The yellow and purple cables are 14 AWG silicon wires. For the uplink I have an almost store bought 12AWG and some hand made 10 AWG monsters. Each motor stalls at 20A so there is the potential of a noticable amount of current to flow around the base of Terry now.

September 16, 2015 12:40 PM

September 12, 2015

Clinton RoyKiwi PyCon quick reviews




Filed under: Uncategorized

September 12, 2015 03:53 AM

September 07, 2015

Blue HackersThe Effect of Screens Before Bedtime

Staring at screens right before sleep turns out to be a lot worse than previously thought. Dr. Dan Siegel, clinical professor of psychiatry at the UCLA School of Medicine, lays out all of the negative effects that bedtime screen viewing can have on the brain and body.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="" width="560"></iframe>

And of course, that’s aside from the effects of the contents! If you read any news, you may find yourself all riled up and annoyed at something you can do absolutely nothing about (or at least not at that moment). If you check your work email, just as problematic.

September 07, 2015 03:00 AM

September 02, 2015

Blue Hackerstears in rain | Wil Wheaton

I walked out of the loading dock, through a cloud of rotting garbage, and into the alleyway behind the theater. A curtain of rain fell between me and my destination, a little over a block away.

“Do you want to wait here, while I get you an umbrella?” Liz, the producer from Wizards of the Coast, asked me.

“No,” I said, stepping into the rain, extending my arms outward and turning my palms and face to the sky, “it’s been so long since I felt rain fall on my body, I’m not going to let this opportunity pass me by.”

I walked down the sidewalk, surrounded by other PAX attendees. Some were not bothered by the rain, while others held up programs and newspapers and other things to keep it away. A man walked his dog next to me. The dog was unperturbed by the weather. We got to the corner and waited for the light to change. The rain intensified and it was glorious.

“Are you sure this is okay?” She said.

“Oh yes, this is so much more than okay,” I answered, “this is perfect.”


I’ve been feeling pretty much the opposite of awesome for several weeks, now, and actually getting to sit down, face to face, in a semi-quiet few moments with real people who wanted to be there with me was … restorative, I guess is the best word. One player told me, “Thank you for everything you do. From Tabletop to Titansgrave — which is the best thing I’ve ever seen — to talking so openly about anxiety and depression.”


Read Wil’s entire post at

September 02, 2015 12:05 AM

August 31, 2015

Ben MartinInspecting ODF round trips for attribute retention

Given an office application one might like to know which attributes are preserved properly across a load and save cycle. For example, is the background color or margin size mutated just by loading and saving an ODF file with OfficeAppFoo version 0.1.

The odfautotests project includes many tests on simple ODF documents to see how well each office application preserves the information in the document. Though testing ODF attribute preservation might not be as simple as one might first imagine. Consider the below document with a single paragraph using a custom style:

  <text:p text:style-name="style">hello world</text:p>

In the styles.xml file one might see something like the following:

     <style:text-properties fo:background-color="transparent" />


This input is obviously designed to see how well the fo:background-color style information is preserved by office applications. One thing to notice is that the style:family attribute in the above is paragraph.

If one loads and saves a document with the above fragments in it using LibreOffice 4.3.x then they might see something like the following in the output ODF file. In content.xml:

<text:p text:style-name="TestStyle">hello world</text:p>

And in the styles.xml file the background-color attribute is preserved:

<style:style style:name="TestStyle"
      <style:text-properties fo:background-color="transparent"/>

One can test if the attribute has been preserved using XPath selecting on the @style-name of the text:p and then making sure that the matching style:style has the desired fo:background-color sub attribute.

The XPath might look something like the below, which has been formatted for display:

  or (not(@s:display-name) and @s:name='TestStyle')]

Performing the load and save using Word 2016 is quite interesting. The resulting content.xml file might have:

<style:style style:name="P1"
     <style:paragraph-properties fo:break-before="page"/>
<office:text text:use-soft-page-breaks="true">
  <text:p text:style-name="P1">hello world</text:p>

and in styles.xml the background-color setting is pushed up to the paragraph style level.

<style:style style:name="TestStyle"
      <style:text-properties fo:hyphenate="false"/>

<style:default-style style:family="paragraph">
<style:text-properties ... fo:background-color="transparent"

So to see if the output ODF has the fo:background-color setting one has to consider not just the directly used style "P1" but also parent style elements which might contain the attribute instead. In this case it was pushed right up to the paragraph style.

For the Word output the above XPath doesn't necessarily work. If the attribute we are looking for has been pushed up to paragraph then we should look for it there instead. Also, if we are looking at the paragraph level then we need to be sure that there is no attribute directly at the lower, TestStyle, level. Also it helps to ensure in the selection that the paragraph is really a parent of the TestStyle, or P1 in the above.

After a bit of pondering I found an interesting solution that can evaluate using plain XPath1.0. To test the value I pick off the fo:background-color from both the TestStyle and also the paragraph level. If those values are passed to concat() then, if the attribute is only at the TestStyle or paragraph level we get something that can be used to test the value. If the attribute appears at both levels are are in trouble.

For example:

<style:style style:name="TestStyle"
<style:text-properties ... fo:background-color="transparent"  />
<style:default-style style:family="paragraph">
<style:text-properties ... fo:background-color="#FF0000"/>

Considering the semantic XPath query of concat( TestStyle/@fo:background-color, paragraph/@fo:background-color ) the result would be  transparent#FF0000 which would not match a string comparison with 'transparent'.

The trick is to use an array selector on the second item in the concat() call. If we only return the paragraph/@fo:background-color value if there is no value associated with the TestStyle then the concat will effectively only return one or the other (directly on TestStyle or nothing on TestStyle and the attribute from paragraph).

With this the query can allow the office application to move the information to a parent of the style and still match for a test.

August 31, 2015 10:17 AM

August 30, 2015

Daniel DevineAn Update

Over the past 8 or so months I have been quite busy. I decided to take a semester off uni to do some government work, following that I had enough money saved to take a trip to the USA (and an unexpected deviation from the plan). I was intending on re-enrolling for the second semester of university but found that it was impractical... So I'm currently job hunting!

Read more…

August 30, 2015 12:16 PM

August 27, 2015

Adrian SuttonTesting@LMAX – Replacements in DSL

Given our DSL makes heavy use of aliases, we often have to provide a way to include the real name or ID as part of some string. For example, an audit record for a new account might be:

Created account 127322 with username someUser123.

But in our acceptance test we’d create the user with:


someUser is just an alias, the DSL creates a unique username to use and the system assigns a unique account ID that the DSL keeps track of for us. So to write a test that the new account audit record is written correctly, we need a way to build the expected audit string.

Our initial attempt was straight forward enough:

"expression: Created account <accountId> with username <username>.",
"account: someUser",

The DSL substitutes <accountId> and <username> with the real values for us. Simple, neat, worked well for quite a while. Except that over time we kept finding more and more things that needed to be substituted and encountered situations where an audit log would reference to two different accounts or multiple instruments, leading us to have <accountId2> and <accountId3> replacements.

So a little while back some of my clever colleagues changed things around to a slightly more complex but much more flexible syntax:

"expression: Created account <accountId:someUser> with username <username:someUser>."

Essentially, the replacement now contains both the type of value to insert as well as the alias to look up. This has been a minor revolution for our DSL – it’s now much easier to handle all sorts of complex cases and it’s much clearer which alias is being used for each replacement.

The biggest win though, is that because all the required information to perform the expansions is inside the one string – not requiring any extra DSL parameters – the replacement algorithm can be easily shared across all the different places in the DSL that need to support replacements. New types of things to be replaced can easily be added in one place and are then available everyone as well.

It’s surprising how much benefit you can get from such a simple little change.

August 27, 2015 10:29 AM

August 25, 2015

Adrian SuttonMiller – Command Line CSV Tool

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV.

Leaving this here for the next time I have to work with CSV files on the command line.  It can do a lot more than just CSVs of course, but that’s likely to be the most useful to me.

August 25, 2015 10:27 AM

August 24, 2015

Blue HackersThe Legacy of Autism and the Future of Neurodiversity

The New York Times published an interesting review of a book entitled “NeuroTribes: The Legacy of Autism and the Future of Neurodiversity”, authored by Steve Silberman (534 pp. Avery/Penguin Random House).

Silberman describes how autism was discovered by a few different people around the same time, but with each the publicity around their work is warped by their environment and political situation.

This means that we mainly know the angle that one of the people took, which in turn warps our view of Aspergers and autism. Ironically, the lesser known story is actually that of Hans Asperger.

I reckon it’s an interesting read.

August 24, 2015 11:50 PM

August 08, 2015

Adrian SuttonDisplay Port Monitor Wakes Up a Few Seconds After Being Suspending on Linux

I have a dual monitor setup with Linux where one display is connected via DisplayPort and the other via DVI, both from an Nvidia graphics card. When the screen is locked both displays go black, the DVI monitor says it’s entering power save mode and the DisplayPort monitor says “No input” at which point both monitors turn back on displaying the unlock screen.

Playing with xset dpms force standby|suspend|off gave a variety of effects, sometimes the DisplayPort stayed off but the DVI turned back on, sometimes the DisplayPort went black but didn’t turn off etc.

Ultimately though the solution was to disable the “DP Auto Switch” and “HDMI Auto Switch” setting on the DisplayPort monitor. I imagine on other monitors this is equivalent to auto detecting the input source. Now the default Linux settings do the right thing and turn both monitors off after a while, they stay off until I move the mouse and then both turn back on perfectly.

August 08, 2015 09:41 PM

July 17, 2015

Ben MartinOSX Bundling Soprano and other joys

Libferris has been moving to use more Qt/KDE technologies over the years. Ferris is also a fairly substantial software project in it's own right, with many plugins and support for multiple libraries. Years back I moved from using raw redland to using soprano for RDF handling in libferris.

Over recent months, from time to time, I've been working on an OSX bundle for libferris. The idea is to make installation as simple as copying to /Applications. I've done some OSX packaging before, so I've been exposed to the whole library paths inside dylib stuff, and also the freedesktop specs expecting things in /etc or whatever and you really want it to look into /Applications/YouApp/Contents/Resources/.../etc/whatever.

The silver test for packaging is to rename the area that is used to build the source to something unexpected and see if you can still run the tools. The Gold test is obviously to install from the app.dmz onto a fresh machine and see that it runs.

I discovered a few gotchas during silver testing and soprano usage. If you get things half right then you can get to a state that allows the application to run but that does not allow a redland RDF model to ever be created. If your application assumes that it can always create an in memory RDF store, a fairly secure bet really, then bad things will befall the app bundle on osx.

Plugins are found by searching for the desktop files first and then loading the shared libary plugin as needed. The desktop files can be found with the first line below, while the second line allows the plugin shared libraries to be found and loaded.

export SOPRANO_DIRS=/Applications/
export LD_LIBRARY_PATH=/Applications/

You have to jump through a few more hoops. You'll find that the plugin ./lib/soprano/ links to lib/librdf.0.dylib and librdf will link to other redland libraries which themselves link to things like libxml2 which you might not have bundled yet.

There are also many cases of things linking to QtCore and other Qt libraries. These links are normally to nested paths like Library/Frameworks/QtCore.framework/Versions/4/QtCore which will not pass the silver test. Actually, links inside dylibs like that tend to cause the show to segv and you are left to work out where and why that happened. My roll by hand solution is to create softlinks to these libraries like QtCore in the .../lib directory and then resolve the dylib links to these softlinks.

In the end I'd also like to make an app bundle for specific KDE apps. Just being able to install okular by drag and drop would be very handy. It is my preferred reader for PDF files and having a binary that doesn't depend on a build environment (homebrew or macports) makes it simpler to ensure I can always have okular even when using an osx machine.

July 17, 2015 01:01 PM

Ben MartinTerry && EL

After getting headlights Terry now has a lighted arm. This is using the 3 meter EL wire and a 2xAA battery inverter to drive it. The around $20 entry point to bling is fairly hard to resist. The EL tape looks better IMHO but seems to be a little harder to work with from what I've read about cutting the tape and resoldering / reconnecting.

I have a 1 meter red EL tape which I think I'll try to wrap around the pan/tilt assembly. From an initial test it can make it around the actobotics channel length I'm using around twice. I'll probably print some mounts for it so that the tape doesn't have to try to make right angle turns at the ends of the channel.

July 17, 2015 06:49 AM

July 16, 2015

Ben MartinTerry - Lights, EL and solid Panner

Terry the robot now has headlights! While the Kinect should be happy in low light I found some nice 3 watt LEDs on sale and so headlights had to happen. The lights want a constant current source of 700mA so I grabbed an all in one chip solution do to that and mounted the lights in series. Yes, there are a load of tutorials on building a constant current driver for a few bucks around the net, but sometimes I don't really want to dive in and build every part. I think it will be interesting at some stage to test some of the constant current setups and see the ripple and various metrics of the different designs. That part of he analysis is harder to find around the place.

And just how does this all look when the juice is flowing I hear you ask. I have tilted the lights ever so slightly downwards to save the eyes from the full blast. Needless to say, you will be able to see Terry coming now, and it will surely see you in full colour 1080 glory as you become in the sights. I thought about mounting the lights on the pan and tilt head unit, but I really don't want these to ever get to angles that are looking right into a person's eyes as they are rather bright.

On another note, I now have some EL wire and EL tape for Terry itself. So the robot will be glowing in a sublte way itself. The EL tape is much cooler looking than the wire IMHO but the tape is harder to cut (read I probably won't be doing that). I think the 1m of tape will end up wrapped around the platform on the pan and tilt board.

Behind the LED is quite a heatsink, so they shouldn't pop for quite some time. In the top right you can just see the heatshrink direct connected wires on the LED driver chip and the white wire mounts above it. I have also trimmed down the quad encoder wires and generally cleaned up that area of the robot.

A little while ago I moved the pan mechanism off axle. The new axle is hollow and setup to accomodate a slip ring at the base. I now have said slip ring and am printing a crossover plate for that to mount to channel. Probably by the next post Terry will be able to continuiously rotate the panner without tangling anything up. The torque multiplier of the brass to alloy wheels together with the 6 rpm gearmotor having very high torque means that the panner will tend to stay where it is. Without powering the motor the panner is nearly impossible to move, the grub screws will fail before the motor gives way.

Although the EL tape is tempting, the wise move is to fit the slip ring first.

July 16, 2015 02:39 AM

June 24, 2015

Clinton RoyPyCon Australia 2015 Programme Released

PyCon Australia is proud to release our programme for 2015, spread over the weekend of August 1st and 2nd, following our Miniconfs on Friday 31 July.

Following our largest ever response to our Call for Proposals, we are able to present two keynotes, forty eight talks and two tutorials. The conference will feature four full tracks of presentations, covering all aspects of the Python ecosystem, presented by experts and core developers of key Python technology. Our presenters cover a broad range of backgrounds, including industry, research, government and academia.

We are still finalising our Miniconf timetable, but we expect another thirty talks for Friday. We’d like to highlight the inaugural running of the Education Miniconf whose primary aim is to bring educators and the Python community closer together.

The full schedule for PyCon Australia 2015 can be found at

PyCon Australia has endeavoured to keep tickets as affordable as possible. We are able to do so, thanks to our Sponsors and Contributors. Registrations for PyCon Australia 2015 are now open, with prices starting at AU$50 for students, and tickets for the general public starting at AU$240. All prices include GST, and more information can be found at

We have also worked out favourable deals with accommodation providers for PyCon delegates. Find out more about the options at

To begin the registration process, and find out more about each level of ticket, visit
Important Dates to Help You Plan

June 29: Financial Assistance program closes.
July 8: Last day to Order PyCon Australia 2015 T-shirts
July 19: Last day to Advise Special Dietary Requirements
July 31 : PyCon Australia 2015 Begins

About PyCon Australia

PyCon Australia is the national conference for the Python Programming Community. The sixth PyCon Australia will be held on July 31 through August 4th, 2015 in Brisbane, bringing together professional, student and enthusiast developers with a love for developing with Python. PyCon Australia informs the country’s Python developers with presentations, tutorials and panel sessions by experts and core developers of Python, as well as the libraries and frameworks that they rely on.

To find out more about PyCon Australia 2015, visit our website at or e-mail us at

PyCon Australia is presented by Linux Australia ( and acknowledges the support of our Platinum Sponsors, Red Hat Asia-Pacific, and Netbox Blue; and our Gold sponsors, The Australian Signals Directorate and Google Australia. For full details of our sponsors, see our website.

Filed under: Uncategorized

June 24, 2015 03:43 AM

June 23, 2015

Adrian SuttonEnd to End Tests @ LMAX Update

A little while back I said that LMAX ran around 11,000 end to end tests in around 50 minutes. Since then we’ve deployed some new hardware to run our continuous integration on, plus continued building new stuff and are now running about 11,500 tests in under 20 minutes.

A large part of the speed boost is extra VM instances but also the increased RAM allocation available to each VM has allowed us to increase a number of limits in the system and we can now run more tests concurrently against each VM instance.

We’re currently running 61 instances of the exchange using virtual machines hosted by four Dell FX2s chassis three-quarter populated with FC630s. That gives us 480 cores and 4.5TiB RAM. That’s certainly no small investment, but we consider it excellent value for money because of the boost in productivity and confidence it gives our development team (not to mention the boost in confidence and reliability it gives our clients).

June 23, 2015 10:54 PM

June 16, 2015

Ben MartinAbide the Slide

The holonomic drive robot takes it's first rolls! This is what you get when you contort a 3d printer into a cross format and attach funky wheels. Quite literally as the control board is an Arduino Mega board with Atmel 2650 MCU and a RAMPS 1.4 stepper controller board plugged into it. The show is controlled over rf24 link from a hand made controller. Yes folks, a regression to teleoperating for now. I'll have to throw the thing onto scales later, but the steppers themselves add considerable weight to the project, but there doesn't seem to be much problem moving the thing around under it's own power.

<iframe allowfullscreen="" frameborder="0" height="315" src="" width="560"></iframe>

The battery is a little underspeced, it will surely supply enough current, and doesn't get hot after operation, but the overall battery capacity is low so the show is over fairly quickly. A problem that is easily solved by throwing more dollars at the battery. The next phase is to get better mechanical stability by tweaking things and changing the software to account for the fact that one wheel axis is longer than the other. From there some sensor feedback (IMU) and a fly by wire mode will be on the cards.

<iframe allowfullscreen="" frameborder="0" height="315" src="" width="560"></iframe>

This might end up going into ROS land too, encapsulating the whole current setup into being a "robot base controller" and using other hardware above to run sensors, navigation, and decision logic.

June 16, 2015 10:39 PM

June 12, 2015

Clinton RoyPyCon Australia 2015 Early Bird Registrations Now Open!

We are delighted to announce that online registration is now open for PyCon Australia 2015. The sixth PyCon Australia is being held in Brisbane, Queensland from July 31st – 4th August at the Pullman Brisbane and is expected to draw hundreds of Python developers, enthusiasts and students from Australasia and afar.

Starting today, early bird offers are up for grabs. To take advantage of these discounted ticket rates, be among the first 100 to register. Early bird registration starts from $50 for full-time students, $180 for enthusiasts and $460 for professionals. Offers this good won’t last long, so head straight to and register right away.

PyCon Australia has endeavoured to keep tickets as affordable as possible. We are able to do so, thanks to our Sponsors and Contributors.

We have also worked out favourable deals with accommodation providers for PyCon delegates. Find out more about the options at

To begin the registration process, and find out more about each level of ticket, visit
Important Dates to Help You Plan

June 8: Early Bird Registration Opens — open to the first 100 tickets
June 29: Financial Assistance program closes.
July 8: Last day to Order PyCon Australia 2015 T-shirts
July 19: Last day to Advise Special Dietary Requirements
July 31 : PyCon Australia 2015 Begins

About PyCon Australia

PyCon Australia is the national conference for the Python Programming Community. The sixth PyCon Australia will be held on July 31 through August 4th, 2015 in Brisbane, bringing together professional, student and enthusiast developers with a love for developing with Python. PyCon Australia informs the country’s Python developers with presentations, tutorials and panel sessions by experts and core developers of Python, as well as the libraries and frameworks that they rely on.

To find out more about PyCon Australia 2015, visit our website at or e-mail us at

PyCon Australia is presented by Linux Australia ( and acknowledges the support of our Platinum Sponsors, Red Hat Asia-Pacific, and Netbox Blue; and our Gold sponsors, The Australian Signals Directorate and Google Australia. For full details of our sponsors, see our website.

Filed under: Uncategorized

June 12, 2015 08:35 AM

June 08, 2015

Adrian SuttonTesting@LMAX – Aliases

Even after a magnum opus on the DSL LMAX uses for acceptance tests, there’s one crucial feature that I haven’t mentioned: the use of aliases to allow tests to use simple, meaningful names while ensuring that uniqueness constraints are met.

Creating a user with our DSL looks like:


You might expect this to create a user with the username ‘user’, but then we’d get conflicts between every test that wanted to call their user ‘user’ which would prevent tests from running safely against the same deployment of the exchange.

Instead, ‘user’ is just an alias that is only meaningful while this one test is running. The DSL creates a unique username that it uses when talking to the actual system. Typically this is done by adding a postfix so the real username is still reasonably understandable e.g. user-fhoai42lfkf.

We do the same thing for instruments, venues, currencies and anything else that needs unique names.

This relatively simple trick gives us a great deal of isolation between tests that may run against the same server instance, even allowing us to run the same test multiple times without interfering with itself.

June 08, 2015 02:44 AM

June 05, 2015

Adrian SuttonTesting@LMAX – Abstraction by DSL

At LMAX, all our acceptance tests are written using a high level DSL. This gives us two key advantages:

The DSL that LMAX uses is probably not what most people think of when hearing the term DSL – it doesn’t attempt to read like plain English, just simplifies things down significantly. We’ve actually open sourced the simple little library that is the entrance-way to what we think of as the DSL – creatively named simple-dsl. It’s essentially the glue between what we write in an acceptance test and the plain-java implementation behind that.

As a simple example here’s a test that creates a user, instrument and places an order:

tradingAPI.placeOrder("instrument", "quantity: 5", "type: market",
"expectedStatus: UNMATCHED");

Overall, the aim is to have the acceptance tests written at a very high level – focussing on what should happen but leaving the how to the DSL implementation. The tradingAPI.placeOrder call is a good example of this, it’s testing that when the user places an order on an instrument with no liquidity, it won’t be matched. In the DSL that’s actually a two step process, first place the order and receive a synchronous OK response to say the order was received, then when the order reaches the matching engine an asynchronous event will be emitted to say the order was not matched. We could have made that two separate calls in the acceptance test but that would have exposed too much detail about how the system works when what we really care about is that the order is unfilled, how that’s reported is an implementation detail.

However that does mean that the implementation of the DSL an important part of the specification of the system. The acceptance tests express the user requirements and the DSL expresses the technical details of those requirements.

Model the Types of Users and Interactions

All our acceptance tests extends a base class, DslTestCase, that exposes a number of public variables that act as the entry points to the system (registrationAPI, adminAPI and tradingAPI in the example above). Each of these roughly represent a way that certain types of users interact with the system. So registrationAPI works with the API exposed by our registration gateway – the same APIs that our sign-up process on the website talks to.  adminAPI uses the same APIs our admin console talks to and tradingAPI is the API that both our UI uses and that many of our clients interact with directly.

We also have UI variants like adminUI and tradingUI that use selenium to open a browser and test the UI as well.

Our system tends to have a strong correlation between the type of user and the entry point to the system they use so our DSL mostly maps to the gateways into our system, but in other systems it may be more appropriate to focus more on the type of user regardless of what their entry point into the system is. Again the focus should be on what happens more than how. The way you categorise functions in the DSL should aid you in thinking that way.

That said, our top level DSL concepts aren’t entirely restricted to just the system entry point they model. For example the registrationAPI.createUser call in the example will initially talk to the system’s registration API, but since a new account isn’t very useful until it deposits funds, it then talks to the admin console to approve the registration and credit some funds into the users account. There’s a large dose of pragmatism involved in the DSL implementation with the goal being to make it easy to read and write the acceptance tests themselves and we’re willing to sacrifice a little design purity to get that (but only a little).

Top level concepts often further categorise the functionality they provide, for example our admin console that adminAPI drives has a lot of functionality and is used by a range of types of users, so it sub-categorises into things like marketOperations, customerServices, risk, etc.

Add Reusable Components to the DSL

One of the signs that people don’t understand the design of our DSL is when they extract repetitive pieces of tests into a private method within the test itself. On the surface this seems  like a reasonable idea, allowing that sequence of actions to be reused by multiple tests in the file. If the sequence is useful in many test cases within one file and significant enough to be worth the indirection of extracting a method it’s almost inevitably useful across many files.

Instead of extracting a private method, put reusable pieces into the DSL itself. Then they’ll be available to all your tests.  More importantly though, you can make that method fit into the DSL style properly – in our case, using simple-dsl to pass parameters instead of a fixed set of method parameters. 

One of our top level concepts in the DSL is ‘workflows’. It bundles together broader sequences of actions that cut across the boundaries of any one entrance point. It’s a handy home for many of the reusable functions we split out. The down side is it’s currently a real grab bag of random stuff and could do with some meaningful sub-categorisation. Naming is hard…

Design to Avoid Intermittency

The way the DSL is designed is a key weapon in the fight against intermittency. The first rule is to design each function to appear synchronous as much as possible. The LMAX Exchange is a highly asynchronous system design but our DSL hides that as much as possible.

The most useful pattern for this is that whenever you provide a setter-type function it should automatically wait and verify that the effect has been fully applied by checking the equivalent getter-type API. So the end of the DSL implementation for registrationAPI.createUser is a waiter that polls our broker service waiting for the account to actually show up there with the initial balance we credited. That way the test can carry on and place an order immediately without intermittently being rejected for a lack of funds.

The second key pattern applies when verifying values. We produce a lot of reports as CSV files so originally had DSL like:"date: today", "rememberAs: myCsvFile");"csvFile: myCsvFile", "amount: 10.00");

Apart from being pretty horrible to read, this leads to a lot of intermittency because our system doesn’t guarantee that cash flows will be recorded to the database immediately, it’s done asynchronously so is only guaranteed to happen within a reasonable time. Instead it’s much better to write:"date: today", "amount: 10.00");

Then inside the DSL you can use a waiter to poll the cashflow CSV until it does contain the expected value or whatever you define as a reasonable time elapses and the test times out and fails. Again, having the test focus on what and the DSL dealing with how allows us to write better tests.

Don’t Get Too Fancy with the DSL

The first thought most people have when they see our DSL is that it could be so much better if we used static types and chained method calls to get the compiler to validate more stuff and have refactoring tools work well. It sounds like a great idea, our simple string based DSL seems far too primitive to work in practice but we’ve actually tried it the other way as well and it’s not as great as it sounds.

Inevitably when you try to make the DSL too much like English or try to get the compiler more involved you add quite a lot of complexity to the DSL implementation which makes it a lot harder to maintain so the cost of your acceptance tests goes up – exactly the opposite of what you were intending.

The trade offs will vary considerably depending on which language you’re using for your tests and the best style of DSL to create will vary significantly. I strongly suspect though that regardless of language the best DSL is a radically simple one, just that different things are radically simple in different languages.

DSL’s Matter

This was meant to be a quick article before getting on to what I really wanted to talk about but suddenly I’m 1500 words in and still haven’t discussed anything about the implementation side of the DSL.

It turns out that while our DSL might be simple and something we take for granted, it’s a huge part of what makes our acceptance tests easily maintainable instead of gradually becoming a huge time sink that prevents any change to the system. My intuition is that those people who have tried acceptance tests and found them too expensive to maintain have failed to find the right style of abstraction in the DSL they use, leaving their tests too focused on how instead of what.

June 05, 2015 06:34 AM

May 04, 2015

Elspeth ThorneCatching up with myself.

It's been quite a while since my last entry on this blog.

Since then, many things have happened.

Starting with the physical stuff, I'm an inch taller. This is a result of some pretty intense physiotherapy and rehabilitation work. My thoratic spine was frozen in place; that's been addressed, finally, and I'm slowly improving. I've even been able to start addressing my lower and upper spine issues. I have to maintain a pretty extensive stretch and exercise regimen, but even with the DOMS I have less back pain than I have had in decades.

Speaking of, I'm weightlifting twice a week. It's a simple split; generally on Mondays I do squats and assisted chin ups; on Thursdays I do deadlifts and some form of upper-body-push exercise, like a pushup or shoulder press. I got up to a working weight of 55kgs when doing squats (generally in four sets of eight reps). That's a pretty good start. I still have a very long way to go before I'm as strong as I'd like to be, especially in my upper body.

As far as flexibility goes, I can touch my toes most of the time, without extensive warm-up, for the first time in my life. Which is awesome. The downside is that combined with other things, there's a distinct possibility I may also have a hyperflexibility disorder. Apparently, when I'm not frozen up, or have muscular tension blocking things, I bend really really well. Too well. One of the implications is that it takes me a lot more effort to do pretty much anything, because I have to use muscles for support and control rather than ligaments and tendons. Which would also explain why and how I strain muscles so easily and often, and some of the fatigue I deal with. Also why I bruise so easily. The long term implications mean that I'm going to be doing an awful lot of strength training and flexibility/mobility work for as long as it's physically possible, just to maintain the level of functionality most people get from sitting in a chair all day. On the upside, knowing what the situation is makes it a whole lot easier to manage, instead of fumbling around and hoping that something will work.

Pain-wise, well, I'm not in pain 24/7 anymore. There are some weeks where I don't even take any painkillers. Day to day I probably experience more pain than most people, but it's rather less than I was accustomed to at most points, which I am very grateful for.

Mentally speaking - well, I'm on antidepressants again. The grey dog struck with a vengeance last year. Food was tasteless, I couldn't care about anything, especially not my own wellbeing, and things got not so great. I made the decision to address this, and I'm doing rather better, which is nice.

My memory has improved significantly. While I'm not back to my previous level of function, I no longer get lost on the way to places I've been several times before, and my memory for people is also much better. There's an entry of its own in there, but I'll leave that for after this one.

Socially speaking, I actually have a social life again! Which is amazing and wonderful. I miss all my Australian friends, but the isolation over here seems to have ended. I certainly hope it has; I'm pretty good with being solitary, but after 3 years of it, I'd like to be a social butterfly for a while. It hasn't all been sunshine and unicorn farts. Readjusting to being around people has been a little tricky - balancing the sensory overload with actually getting lonely when I can't go out is a current challenge. Still, the awesome new people in my life are more than worth it!

I have yet to take up any form of study. That's annoying, but in all honesty, something that has to wait until I'm sure I have the intellectual, emotional, and physical reserves to not run myself into the ground again. Or to recover when I do so the first few times before I manage to calibrate myself accurately enough.

That's pretty much it for now. I'm hoping that this is the start of me actually writing about my mindstate a little more often, but well, we'll have to see.

May 04, 2015 11:19 AM

May 02, 2015

Clinton RoyOne Week Left for PyCon Australia 2015 Proposals!

Under a week to go – closes Friday 8th May

With just under a week to go until the PyCon Australia 2015 Call for Proposals closes, we thought it would be a good idea to give everyone an update and a reminder. We’re very happy with the proposals we’ve already received, but we’re eager to receive more! We hope our proposal writing working bees in Brisbane have been of help, and hope to roll them out to more cities next year. If you’ve got any questions please get in touch (numerous contact details are up on We would like to give a special shout out for the Education MiniConf, which is new this year: if you know people teaching and using computing in the education realm, please forward this CFP on.

The deadline for proposal submission is Friday 8th May, 2015.


The conference this year will be held on Saturday 1st and Sunday 2nd August 2015 in Brisbane. PyCon Australia attracts professional developers from all walks of life, including industry, government, and science, as well as enthusiast and student developers. We’re looking for proposals for presentations and tutorials on any aspect of Python programming, at all skill levels from novice to advanced.
Presentation subjects may range from reports on open source, academic or commercial projects; or even tutorials and case studies. If a presentation is interesting and useful to the Python community, it will be considered for inclusion in the program.
We’re especially interested in short presentations that will teach conference-goers something new and useful. Can you show attendees how to use a module? Explore a Python language feature? Package an application?


Four Miniconfs will be held on Friday 31st July, as a prelude to the main conference. Miniconfs are run by community members and are separate to the main conference. If you are a first time speaker, or your talk is targeted to a particular field, the Miniconfs might be a better fit than the main part of the conference. If your proposal is not selected for the main part of the conference, it may be selected for one of our Miniconfs:
DjangoCon AU is the annual conference of Django users in the Southern Hemisphere. It covers all aspects of web software development, from design to deployment – and, of course, the use of the Django framework itself. It provides an excellent opportunity to discuss the state of the art of web software development with other developers and designers.
The Python in Education Miniconf aims to bring together community workshop organisers, professional Python instructors and professional educators across primary, secondary and tertiary levels to share their experiences and requirements, and identify areas of potential collaboration with each other and also with the broader Python community.
The Science and Data Miniconf is a forum for people using Python to tackle problems in science and data analysis. It aims to cover commercial and research interests in applications of science, engineering, mathematics, finance, and data analysis using Python, including AI and ‘big data’ topics.
The OpenStack Miniconf is dedicated to talks related to the OpenStack project and we welcome proposals of all kinds: technical, community, infrastructure or code talks/discussions; academic or commercial applications; or even tutorials and case studies. If a presentation is interesting and useful to the OpenStack community, it will be considered for inclusion. We also welcome talks that have been given previously in different events.

First Time Speakers

We welcome first-time speakers; we are a community conference and we are eager to hear about your experience. If you have friends or colleagues who have something valuable to contribute, twist their arms to tell us about it! Please also forward this Call for Proposals to anyone that you feel may be interested.

The most recent call for proposals information can always be found at:

See you in Brisbane in July!

Important Dates

Call for Proposals opens: Friday 27th March, 2015
Proposal submission deadline: Friday 8th May, 2015
Proposal acceptance: Monday 25 May, 2015

Filed under: Uncategorized

May 02, 2015 06:54 AM

April 30, 2015

Adrian SuttonMaking End-to-End Tests Work

The Google Testing Blog has an article “Just Say No to More End-to-End Tests” which winds up being a rather depressing evaluation of the testing capabilities, culture and infrastructure at Google. For example:

Let’s assume the team already has some fantastic test infrastructure in place. Every night:

  1. The latest version of the service is built. 
  2. This version is then deployed to the team’s testing environment. 
  3. All end-to-end tests then run against this testing environment. 
  4. An email report summarizing the test results is sent to the team.

If your idea of fantastic test infrastructure starts with the words “every night” and ends with an email being sent you’re doomed. Bryan Pendleton does a good job of analysing and correcting the details so I won’t recover that ground. Instead, let me provide a view of what reasonable test infrastructure looks like.

At LMAX we’ve recently reached the milestone of 10,000 end to end acceptance tests. We’ve obviously invested a lot of time in building up all those tests but they’re invaluable in the way they free us to try daring things and make sweeping changes, confident that if anything is broken it will be caught. We’re happy to radically restructure components in ways that require lots of changes to unit tests because of those end-to-end tests.

We also have huge numbers of unit tests, integration tests, performance tests, static analysis and various other forms of tests, but the end-to-end tests are far more than a sanity-check, they’re a primary form of quality control.

Those end-to-end tests, or acceptance tests as we call them:

That’s pretty much entry-level for doing end-to-end testing (or frankly any testing). We’ve also got a few extra niceties that I’ve written about before:

Plus the test results are displayed in real-time, so we don’t even have to wait for the end of the test run to see any failures. Tests that failed on the previous run are run first to give us quick feedback on whether they’ve been fixed or not.

There’s lots of great stuff in there, but we have more work to do. We have an intermittency problem. When we started out we didn’t believe that intermittency could be avoided and accepted a certain level of breakage on each build – much like the Google post talks about expecting 90% of tests to pass. That attitude is a death-knell for test reliability. If you don’t have all the tests passing consistently, gradually and inevitably more and more tests become intermittent over time.

We’ve been fighting back hard against intermittency and making excellent progress – we’ve recently added the requirement that releases have no failures and green builds are the norm and if there are intermittent failures it’s usually only one or two per run. Currently we’re seeing an intermittent failure rate of around 0.00006% of tests run (which actually sounds pretty good but with 10,000 tests that’s far too many runs with failures that should have been green).

But improvements come in waves with new intermittency creeping in because it can hide in amongst the existing noise. It has taken and will take a lot of dedication and commitment to dig ourselves out of the intermittency hole we’re in but it’s absolutely possible and we will get there.

So next time you hear someone try to tell you that end-to-end tests aren’t worth the effort, point them to LMAX. We do end-to-end testing big time and it is massively, indisputably worth it. And we only expect it to become more worth it as we reduce the intermittency and continue improving our tools over time.

April 30, 2015 05:46 AM

April 27, 2015

Ben MartinUnbrick the NUC

It seems there are many folks with the suspend of death on the NUC. When you suspend to RAM you can't get back. When you disconnect power for a while you can't turn it on again. Welcome to brickland, population: you. I found that following the advice on the forums if I disconnect the CMOS battery for a bit then I could turn on the NUC again.

The downside is that the CMOS battery is installed under the motherboard, so you have to remove the motherboard which is no easy task the first time. Then each subsequent time that the NUC bricks you have to take it apart again to such a great extent.

Luckily I found these extension leads which let me bring out the battery from the case. So hopefully now a debrick isn't going to involve a system teardown anymore.

April 27, 2015 11:14 AM

April 14, 2015

Adrian SuttonPrinting Only Part of a grep Match

It’s not uncommon to want to find a particular line in a file and print just a part of it.  For example, if we wanted to find the total memory on a machine we could use:

grep '^MemTotal:' /proc/meminfo | 
sed -e 's/^MemTotal:\s*\([0-9]*\).*/\1/'

Which does the job but duplicates a bunch of the matching regex. If you’re using a reasonably recent gnu grep, you can use it’s support for perl regex syntax and the \K operator:

grep -Po '^(MemTotal:\s*)\K[0-9]*' /proc/meminfo

The -P is for perl regex, and the o option causes grep to output only what is matched. You can see though that we’re actually matching the MemTotal: at the start of the line yet somehow it doesn’t wind up in the output.  That’s the magic of \K which excludes the token immediately prior to it from the matched text (without rewinding the stream and attempting a different match for that text).


Alexandre Niveau emailed me to point out that in this particular case (and probably most others where I’d be tempted to use \K with grep) grep is completely superfluous and we can just use sed with the -n flag:

sed -n -e 's/^MemTotal:\s*\([0-9]*\).*/\1/p' /proc/meminfo

The -n disables automatic printing of lines and the ‘p’ at the end of the regex prints the matching lines. Definitely good to know – thanks Alexandre.

April 14, 2015 11:02 PM

April 10, 2015

Ben MartinTiny Tim improves and gets Smaller

I finally switched Tiny Tim over to a lipo battery. Almost everything worked when I tested the new battery, the only thing that failed in a major way were the two 2812 LEDs which, either didn't come on or came on for a very quick moment and went dark. So Tim is now smaller again without the "huge" AA battery pack at it's tail.

The 2812 story was interesting. It wasn't going to be happy jumping to the 7.6v of the 2S lipo. So I tried various voltage divider setups which didn't work either. I ended up using a common 5v regulator and the lights work fine again. I think I was maybe using too high resistor values in the divider and the 2812s didn't like it. At any rate, they apparently want a good regulated power source, and I wasn't giving it one before I switched over to using the regulator.

On the whole, going from 5-6v of the AA pack to 7.6v has made it a snappier mover. I tried it initially with the battery on the bench and found it would lift the back off the desk under hard break.

Next up is probably attaching a claw or drop mechanism and ultrasound sensor and then take on the Sparkfun autonomous ping-pong ball into cup challenge. I'll probably control it via wireless from a second on board micro-controller. The drop, ultrasound, and autonomous navigation micro (and additional battery) can all be put into a single "module" that I can then bolt to Tim. All the navigation micro needs to do is control the differential drive like a remote control would. This way, the existing micro etc on Tim doesn't change at all in order for the challenge to be accepted.

April 10, 2015 01:11 PM

Clinton RoyPyCon Australia 2015 Proposal Writing Working Bee

Writing your first conference proposal can be difficult, so we’re running a working bee at UQ on Saturday 11th (in conjunction with Humbug). If you’ve never written a conference proposal before, or you’d like yours given the once over, please come along, register over at meetup.

Filed under: Uncategorized

April 10, 2015 08:57 AM

April 09, 2015

Adrian SuttonEmoji One – Open source emoji designed for the web.

Mostly so I can find this again later when I inevitably need it, Emoji One is a creative commons licensed collection of Emoji and related tools for the web.

April 09, 2015 12:55 AM

March 30, 2015

Blue HackersSleep: How to nap like a pro | BBC Future

March 30, 2015 11:43 PM

March 28, 2015

Clinton RoyPyCon Australia 2015 Call for Proposals is Open!

PyCon Australia 2015 is pleased to announce that its Call for Proposals is now open!
The conference this year will be held on Saturday 1st and Sunday 2nd August 2015 in Brisbane. We’ll also be featuring a day of Miniconfs on Friday 31st July.

The deadline for proposal submission is Friday 8th May, 2015.

PyCon Australia attracts professional developers from all walks of life, including industry, government, and science, as well as enthusiast and student developers. We’re looking for proposals for presentations and tutorials on any aspect of Python programming, at all skill levels from novice to advanced.

Presentation subjects may range from reports on open source, academic or commercial projects; or even tutorials and case studies. If a presentation is interesting and useful to the Python community, it will be considered for inclusion in the program.

We’re especially interested in short presentations that will teach conference-goers something new and useful. Can you show attendees how to use a module? Explore a Python language feature? Package an application?


Four Miniconfs will be held on Friday 31st July, as a prelude to the main conference. Miniconfs are run by community members and are separate to the main conference. If you are a first time speaker, or your talk is targeted to a particular field, the Miniconfs might be a better fit than the main part of the conference. If your proposal is not selected for the main part of the conference, it may be selected for one of our Miniconfs:

DjangoCon AU is the annual conference of Django users in the Southern Hemisphere. It covers all aspects of web software development, from design to deployment – and, of course, the use of the Django framework itself. It provides an excellent opportunity to discuss the state of the art of web software development with other developers and designers.

The Python in Education Miniconf aims to bring together community workshop organisers, professional Python instructors and professional educators across primary, secondary and tertiary levels to share their experiences and requirements, and identify areas of potential collaboration with each other and also with the broader Python community.

The Science and Data Miniconf is a forum for people using Python to tackle problems in science and data analysis. It aims to cover commercial and research interests in applications of science, engineering, mathematics, finance, and data analysis using Python, including AI and ‘big data’ topics.

The OpenStack Miniconf is dedicated to talks related to the OpenStack project and we welcome proposals of all kinds: technical, community, infrastructure or code talks/discussions; academic or commercial applications; or even tutorials and case studies. If a presentation is interesting and useful to the OpenStack community, it will be considered for inclusion. We also welcome talks that have been given previously in different events.

Full details:

Filed under: Uncategorized

March 28, 2015 05:31 AM

March 17, 2015

Ben MartinGoogle Breakpad and the post crash experience

Google Breakpad has many components to it, but at the basic level it lets you capture information at the time a crash occurs and upload that to the net. A really cute part of Breakpad is that the binary doesn't need to have the debug symbols in it, you don't even need to have them on the client machine at any location. When you build version $githash then you use a breakpad tool to copy out the debug symbols into separate files. When the user discovers a crash they upload a minidump file to a server of your selecting. Then you can combine the extracted symbols from build time and the minidump file to generate a backtrace with line number information. So software users don't have to know about gdb or lldb or whatnot and how to make a backtrace and where to paste it.

I recently updated FontForge's use of breakpad to use a small server on localhost to report the bug. The application dmg file for fontforge will soon also include the extracted symbols for the build. By telling breakpad to use a local server, that server can lookup the symbols that are shipped and generate a human readable backtrace with line number information. Because its also a web interface and running locally, it can spawn a browser on itself. So instead of getting the Mac dialog supplied by the osx crash reporter app telling you that there was a crash, you get a web page telling you the same thing. But the web page can use jQuery/Bootstrap (or $ui tool of choice) and ask what the user was doing and offer many ways to proceed from there depending on how the user wants to report things. The site can be used to report without any login or user accounts. It's also rather handy as a place to checking larger backtraces that might be, maybe, 50-100kb.

But once you can upload to gist, you can get a http and other URL links to the new gist. So it makes sense from there to offer to make a new github issue for the user too. And in that new issue include the link to the gist page so that developers can get at the full backtrace. It turns out that you can do this last part, which requires user login to github, by redirecting to github/.../issues/new and passing title and body GET parameters. While there is a github API, to report a new issue using it you would need to do OAuth first. But in the libre world it's not so simple to have a location to store the OAuth secure token for next time around. So the GET redirect trick nicely gets around that situation.

For those interested in this, the gist upload and callback to subsequently make a github issue are both available. The Google Breakpad hands over the minidump to a POST method which then massages the minidump into the backtrace and spawns a browser on itself. The GET serves up all the html, css, js, and other assets to the browser and that served html/js is what I link to at the start of the paragraph which is where the actual upload/reporting of the backtrace takes place.

The only thing left to do is to respond to the backtraces that come in and everybody gets a more stable FontForge out of the deal. It might be interesting to send off reports to a Socorro server too so that statistics month on month can be easily available.

March 17, 2015 04:33 AM

March 15, 2015

Blue HackersDalek Relaxation for Humans

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src=";showinfo=0" width="420"></iframe>

March 15, 2015 04:08 AM

March 12, 2015

Clinton RoyThursday 12th March

Walked into work fairly early.

Dead tired by the end of the day.

Signed onto sleepio to give them a burl. Looks like they’d like me to get a fitbit..

Filed under: diary

March 12, 2015 07:47 AM

March 11, 2015

Adrian SuttonPatterns are for People

Avdi Grimm in Patterns are for People:
Patterns aren’t tools for programming computers; they are tools for programming people. As such, to say that “patterns are a language smell” makes very little sense. Patterns are a tool for augmenting language: our language, the language we use to talk to each other about a problem and its solutions; the language we use to daydream new machines in our minds before committing them to code. Patterns aren’t a language smell; rather, patterns are by definition, [optional] language features.
There’s so much we can learn if only we stop making everything into a battle between “new” tools or approaches and “old” ones. Patterns are relevant and useful in all forms of programming and we shouldn’t discard them just because they’re not “cool” anymore. Sadly we seem to throw out nearly all our experience and learning each time a new approach comes along rather than learning from and taking advantage of both.

March 11, 2015 10:01 PM

Clinton RoyWednesday 11th March

Awful day.

Sleep was disrupted by neighbours moving their cars around at two thirty am to make way for builders coming. Why two thirty? fucked if I know.

Finally dragged myself out of bed tired and headachy, headed into work.

Did manage to get some stuff done at work before the jack hammering started, and I left immediately in the early afternoon.

Struggling to stay awake till a sensible hour while catching up on a few of these here diary entries.

Filed under: diary

March 11, 2015 08:16 AM

Clinton RoyTuesday 10th March

Walked home.

Conference stuff.

Filed under: diary

March 11, 2015 08:12 AM

Clinton RoyMonday 9th March

Walked into work stupidly early.

Home early, straight to bed.

Filed under: diary

March 11, 2015 08:12 AM

Last updated: November 30, 2015 06:30 PM. Contact Humbug Admin with problems.