My Blog, Reborn

This blog has been broken for a while and I've finally gotten around to fixing it. However rather than just reviving the old blog, I thought it was time for a change. Whilst it may look thhe same, under the hood its a total rewrite.

I didn't dislike the previous Wordpress incarnation, althought there were some nasty hacks in that. But viven I've developed Balsa, a fast and lightweight Java web application framework, I figured I should use it for my own stuff.

So, this blog is now being served from a Balsa application, the content is written in Markdown and stored in Git. It only took a day to knock up the application, Balsa has good support for rendering Markdown content.

Creating a post is now as simple as:

  • Firing up Kate (a rather lovely text editor)
  • Attempting to extract my toughts in a coherent manner (the hardest part for me)
  • Finally the git commit; git push origin

Its refreshingly simple to add a post now, just write, commit and push.

I've even put the code behind my blog on GitHub for people who are really interested.

Java 8

I've been using Java 8 since it was released earlier this year and have found some of the new features game changing. To the extent I've now moved most of my projects to make use of new Java 8 features. Support for Java 8 in Eclipse Luna is good and I've not run into any major issues using Java 8.

Lambda Expressions

The single biggest feature in Java 8 is support for Lambda expressions, this further supports more functional programming styles in Java. Java has always had closures via its anonymous classes functionality, these have often been used to implement callbacks.

Lambda expressions are extremely useful when working with collections. As a simple example, filtering a List prior to Java 8 would require something along the lines of:

// List<String input;
List<String> filtered = new LinkedList<String>();
for (String e : input)
{
    if ("filter".equals(e))
        filtered.add(e);
}

With Java 8 we can transform this into:

// List<String input;
List<String> filtered = input.stream()
                        .filter((e) -> { return "filter".equals(e); })
                        .collect(Collectors.toList());

Certainly a major improvement in semantics and readability. While using Lambda expressions would be slightly slower. Java 8 has taken care to implement them as efficiently as possible, by making use of InvokeDynamic and compiling each Lambda expression to a synthetic method.

Having been using Java 8 for the last few months, I can honestly say that Lambda expressions have changed how I code. The addition of Lambda expressions has made as much of an impact as adding generics in Java 5 did.

Default Methods

Default methods allow concrete functionality to be added to interface methods.

Prior to Java 8 methods of an interface could only be abstract. Interfaces defined how objects should be interacted with only, they were Java's solution to multi-inheritance while attempting to avoid some of the issues with it.

I've always liked the simplicity of the Java object model, however at times it was a straitjacket for certain use cases.

Default methods seem like a good compromise between flexibility and simplicity.

I've found them useful for avoiding having to copy and paste trivial code.

For example:

public interface Parameterised
{
    List<Parameter> getParameters();

    default Parameter getParameter(String name)
    {
        return this.getParameters().stream()
                .filter((p) -> {return name.equals(p.getName());})
                .findFirst()
                .get();
    }
}

Repeatable Annotations

I really like annotations in Java, they allow metadata to be added to code elements. This is really handy for frameworks which can then use this information to customise how objects are interacted with, allowing for more declarative coding.

Since annotations were added in Java 5, I've never understood why they were not repeatable, it seems obvious that they should be. Its a shame that it has taken until Java 8 to address this limitation.

I make heavy use of annotations in Balsa to declare routes (a route handles a specific HTTP request for an application). Annotations give a rather nice way to declare this routing information, making it simple and readable to declare routes. Allowing developers to focus upon the actual functionality of the application.

Prior to Java 8 to make annotations repeatable, you needed to define another annotation to contain them. The user would then need to define both annotations on whatever they were annotating.

For example, the API developer defines the following annotations:

public @interface RequirePermission
{
    String value();
}

public @interface RequirePermissions
{
    RequirePermission[] value();
}

To consume the API, we would then do:

@RequirePermissions({
    @RequirePermission("ui.access"),
    @RequirePermission("ui.read")
})
public void myHttpRoute()
{
}

With Java 8, the API developer only needs to annotate the singular annotation as repeatable:

@Repeatable(RequirePermissions.class)
public @interface RequirePermission
{
    String value();
}

This has the advantage for the API developer that it doesn't alter how they process the annotations. However for the API consumer life is a little easier, as we can now do:

@RequirePermission("ui.access")
@RequirePermission("ui.read")
public void myHttpRoute()
{
}

That makes things a fair bit easier and doesn't have any backwards compatibility problems, quite a clever solution really.

Nashorn

Nashorn is a new Javascript engine for Java, it is fast and easy to work with.

It boasts performance comparable to that of Google's V8 and has the massive advantage of being able to make use of any Java APIs from Javascript, including threading. Again it makes use of InvokeDynamic for performance. It is usable via the ScriptEngine API as well as directly from the command line.

The quickest way to have a play with Nashorn is via jjs on the command line:

jjs> print("Hello World");
Hello World
jjs> exit();

It isn't that hard to execute a script from Java either:

// create the script engine
ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine script = factory.getEngineByName("nashorn");
// execute
script.eval("print(\"Hello World\");");

To pass variables into the ScriptEngine, we need to setup some bindings:

SimpleBindings bindings = new SimpleBindings();
bindings.put("message", "Hello World");
script.setBindings(bindings, ScriptContext.ENGINE_SCOPE);

Variables are contained by a ScriptEngine context, and are not shared across different ScriptEngine instances, we can change the previous example to:

// create the script engine
ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine script = factory.getEngineByName("nashorn");
// bindings
SimpleBindings bindings = new SimpleBindings();
bindings.put("message", "Hello World");
script.setBindings(bindings, ScriptContext.ENGINE_SCOPE);
// execute
script.eval("print(message);");

As mentioned Nashorn allows Javascript to inter-operate with Java, Javascript can invoke Java methods and Java can invoke Javascript functions. Nashorn also automatically maps functions to single method interfaces. For example, we can create a new thread to print Hello World twice a second:

jjs> new java.lang.Thread(function() { while (1) { print("Hello World"); java.lang.Thread.sleep(500); } }).start();

About SNMP-IB

SNMP-IB is a minimalist non-blocking asynchronous SNMP V1, V2c and V3 client implementation. It implements enough to be able to query information from things and receive traps from things.

I started writing SNMP-IB as a way to get to understand SNMP better, I thought: it calls its self simple it can't be that hard. To an extent getting version 2c implemented was simple, I had a working implementation after an evenings work. Version 3 took a little longer, mainly getting my head around the bat shit crazy design by committee which is version 3. I'll post a bit more on this some time.

I wanted the library to be clean and simple to use. It makes use of Java NIO at the network level and is non-blocking, asynchronous, callback based. One instance (and thread) is capable of efficiently communicating with many devices.

What does it support

SNMP-IB currently supports: Get, GetNext, GetBulk and Set requests for both V1, V2c and V3. It also supports receiving Traps for V1, V2c and V3.

Only the user security model of V3 is supported. MD5 and SHA1 are supported for authentication and DES (56bit) and AES (128bit) are supported for privacy.

The core code is fairly stable, however it's real world exposure to devices is somewhat limited. Mainly being tested against 3COM switches, Aerohive access points and Cisco switches, basically what ever devices I have / have access to.

Using SNMP-IB

A key design goal was creating a simple, clean API which is easy to use, the following will fetch the system description and uptime from two devices:

// Create the transport which will be used to send our SNMP messages
SNMPTransport transport = SNMPTransport.open();

// A context represents an Agent we are going to contact, or which is going to contact us
SNMPV2Context lcAgent  = transport.openV2Context("127.0.0.1").setCommunity("public");
SNMPV2Context swAgent  = transport.openV2Context("172.30.12.1").setCommunity("public");

// Use the context to send messages
// The callback will be executed when a response to a request is received
lcAgent.get(new OnResponse.LoggingAdapter(), new OnError.LoggingAdapter(), 
            "1.3.6.1.2.1.1.1.0", "1.3.6.1.2.1.1.3.0");
swAgent.get(new OnResponse.LoggingAdapter(), new OnError.LoggingAdapter(), 
            "1.3.6.1.2.1.1.1.0", "1.3.6.1.2.1.1.3.0");

// Run our transport to send and receive messages
transport.run();

The SNMPContext is the key abstraction, use it to send requests to a device and recieve a callback when the response has been received. The callback classes are designed to be useable from Java 8, without taking a dependency on Java 8.

Where can I get it

You can find the code on GitHub it is licensed under the LGPL V3.

Children Of The Grave

This is a bit of a rehash of a post I originally wrote a 18 months ago, before the revelations of Edward Snowden and the more recent DRIP débâcle.

We've seen time and time again, the power of the Internet in disseminating information, especially during chaotic times, the most recent example being in the Ukraine.

The Internet offers us a utopia where information can be freely shared. Where it can be shared without restriction. Where people can communicate with each other. Where geography does not exist. It is by definition transnational. It offers all of us freedom.

By enabling anyone to communicate, without prejudice, without inference, with anyone else. The Internet represents the single most powerful tool humanity has.

This power, these freedoms. Seem even more significant given the recent events.

The Internet can provide me and you directly with raw information, when it happens, where it happens. No media organisation, what ever there motives can compete with that.

Recent events in the UK has reignited debate over the interception and retention of communications data.

My view remains unchanged, my thoughts have not been swayed. I remain firmly opposed to any capability for Government to snoop on its citizens.

To paraphrase a book I recently read:

This post is not a manifesto. There is no time for that. This is a warning

The book in question: Cypherpunks - Freedom and the future of the Internet. It was a book I found immense satisfaction in reading, and a book I would recommend every Internet user to read.

The Internet is humanities best chance at a transnational utopia. Yet it is a paradox. It is also a transnational surveillance dystopia.

The Internet's offer of free communication also offers total surveillance, leading to totalitarian control by an elite.

Its in encumbered upon every single person whom uses the Internet to: realise and understand their freedom and most importantly defend it.

We must not enable a minority to snoop on our activities, to ultimately control and dominate us. Lets us understand our freedom, lets us embrace it, lets us be defiant.

We must not let our politicians react disproportionally. Existing powers are too wide reaching and invasive, we must fight to get these reduced. But above all we need debate. We cannot let politicians to collude and pass these draconian powers without any consultation or proper consideration.

The title of this post, is a track by Black Sabbath. What does Heavy Metal have to do with politics I hear you ask. Its merely my interpretation of the lyrics, but I feel it embodies my point. Even given it was composed in a different era (has that much really changed), I feel it is still relevant.

"Revolution in their minds - the children start to march 
 Against the world in which they have to live

 Show the world that love is still alive you must be brave 
 Or you children of today are 
 Children of the Grave"

About Bergamot

Bergamot is a scalable, distributed monitoring system which offers an easy migration path from Nagios. Bergamot is written in Java and makes use of RabbitMQ to distribute checks. It is able to read the Nagios object configuration and is able to execute Nagios checks.

More to come...

50 years on

This weekend, 27th October 2012, is 50 years on from the Cuban missile crisis.

Arguably the closest the world has come to the Mutually Assured Destruction of a nuclear war.

To my generation the thought of a nuclear war seems mad. It seems unimaginable, a relic of history. I never lived through the events my parents did. I never felt the fear that their generation did. While, I'm interested, and somewhat well informed on the subject. I'm in many ways atypical.

Yet, while the world has moved on, progressed. Nuclear weapons are still reality. Britain still has Trident II D5 missiles, stationed on board Vanguard class SSBNs (submarines). Right now there is probably 16 missiles somewhere in the Atlantic, each with up to 14 warheads.

With tensions between the west and Iran growing. Is the ignorance of my generation acceptable? I suspect not, my generation will have to tackle these issues head on in the near future.

Soon a decision over Trident will need to be made. Should we be renewing, extending, decommissioning our nuclear deterrent. This is a massively complex decision. A question to which I don't think there is an easy answer.

I would rule out, renewing Trident. I don't think we should be creating any more nuclear weapons. But I honestly remain undecided as to whether we should decommission Trident.

It would be a bold move by Britain if we decided to decommission Trident. That might be a good thing.

Ultimately it's probably time we had serious international accords to completely disarm all states with nuclear weapons and finally lay to rest the cold war.

How the Iran situation is handled has significant impact of this. I've read about proposals, for an independent international Uranium bank, which could provide fuel for peaceful nuclear projects.

Whatever happens, we must ensure that we have an intelligent, well informed discussion over the issues leading to a carefully thought out decision.

Shropgeek (R)evolution 2012

On Friday I attended Shropgeek (R)evolution (a web conference in Shrewsbury). It was a excellent way to spend the evening, a friendly and fun atmosphere, with beer and interesting conversation. There was a number of talks throughout the evening and plenty of time for networking / drinking.

On arrival, we were given some goodies, a name badge (with lanyard) and a Sharpie keyring, to write our names and Twitter handles (I'm @intrbiz ) on our badges. Followed by a trip to the bar to get a pint of EPA.

Neil Kinnish got the event under way with an interesting talk on Shropshire Screen . A simple site to aggregate rural cinema listings across Shropshire. A neat site, which shows how easily WordPress can be abused to fit a wide array of needs.

The next hour allowed for some drinking and networking, a chance for me to catchup with a number for friends I don't get to see that often.

Next up was Jake Smith talking about the D&AD awards system they built. It was a fascinating talk, even if I'm still not sure who D&AD is and the videos dragged on a little. It sure looked an interesting project to have been involved in. Their attitude of the project is more important than the client is refreshing. Their approach of fully involving the client during the analysis and model stages of a project is something I agree with.

Neil Kinnish and Mike Kus followed on with a brave presentation on WorkFu and how/why it failed. It's not often that people stand up and talk about how they failed, making it an unusual topic for a conference. However they made some very interesting points. One point which stuck was launching the least viable product, in order to minimise your losses in case it doesn't work.

The last talk of the evening was Paul Annett about gov.uk .

While at Shropshire I had followed the gov.uk project closely. It was amusing listening to how Government has to an extent resisted gov.uk, it's good to finally see the government focusing on the user, for a change. Directgov, had the glimmer of a good idea, just implemented utterly the wrong way. It was interesting hearing aobut how gov.uk is using their design principles to change the fundamental workings of government.

Following the talks, a number of people hung around to socialise into the night. I even got to see Paul Annett do some magic tricks.

Throughout the evening, one topic seemed to keep getting mentioned: Iterate quick and often. Make it easy to deploy your application. Listen to feedback. Code, test, deploy, listen and repeat.

Lastly, thanks to the Shropgeek team , sponsors and speakers for putting on a great evening. I look forward to next year

Setting the CPUID of a XEN guest

After some reading around I've discovered it is possible to configure the CPU vendor and model information that a XEN guest sees.

Most Linux sysadmins will be familiar with cat /proc/cpuinfo to get information about the systems processors. This gives information about the CPU vendor, model and features. For example my desktop gives:

[cellis@cedesktop ~]$ cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 67
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 5200+
- snip -

This information is actually exposed via cpuid instruction . This instruction takes a command in the EAX register and populates the EAX, EBX, ECX and EDX registers with the requested data.

If EAX is set to zero, the CPU vendor string is returned. This is a 12 byte ASCII string. Which is stored in the EBX, EDX, ECX (that is certainly logical, I suspect a hangup from the circuit complexity).

The CPU model string is a little more complex, it is a 48 byte ASCII string.

This string is obtained by executing the cpuid instruction 3 times, with EAX set to: 0x80000002, 0x80000003 and 0x80000004.

XEN has the cpuid config option , which defines the values of EAX, EBX, ECX and EDX for specific values of EAX.

Lets take the following XEN config:

cpuid=['0:eax=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,ebx=01110010011101000110111001001001, 
ecx=00000000000000000000000000000000,edx=00000000011110100110100101100010']

This causes the XEN guest to see the CPU vendor string 'Intrbiz', by providing the cpuid register values when EAX = 0.

As such, my XEN VM now reports:

xenvm1:~ # cat /proc/cpuinfo
processor       : 0
vendor_id       : Intrbiz
cpu family      : 0
model           : 0
model name      : Intrbiz XEN virtual CPU
stepping        : 0

The hard part of setting this, is working out the values of the registers. It takes time to convert the text to binary and get the orderings correct.

Instead, you can generate the XEN config right here:

Simply enter your desired CPU vendor and model string and the XEN config will be generated

CPU Vendor: Model:


							

Leaping CPU

As have many people around the world, I've found a number of my computers having high CPU load since the Leap Second.

Specifically I've found, the following to be using well over 100% CPU usage:

  • Java
  • Ruby
  • MySQL
  • Firefox
  • Akonadi (Uses MySQL)

Researching the issue, it seems there is an issue in the Linux kernel affecting futexes with the Leap Second. This is a new issue, not to be confused with other issues which have previously been patched. Futexes are a form of userspace lock, which are used heavily by the likes of Java, etc. This flaw seems to be in essentially every kernel since 2.6.22.

Note: This is a kernel bug, it is not a bug in Java or any other application.

A patch is already on the LKML: [PATCH] [RFC] Potential fix for leapsecond caused futex related load spikes

There is a work around for the issue, which is to simply set the date on the server, using the following:

date `date +\"%m%d%H%M%C%y.%S\"`

or (if you prefer)

reboot

Setting the time certainly sorted my problems, that took Firefox from 163% CPU to 1% and similar for Ruby and Java. Note that just restarting the Java or whatever process will not solve the problem.

This is because setting the system time, will call a kernel function: clock_was_set() ensuring the hrtimer subsystem is correct. Futexes often use the hrtimer subsystem in a loop, these sub-second waits are expiring immediately, causing high CPU usage. More detail