Developing HTML Forms

Developing useful and professional HTML forms requires a lot more work than simple examples might suggest. There are a whole range of problems such as handling initial data, client-side validation, server-side validation, optimistic concurrency and performing the final action. This post explores the various steps.

The initial GET request

The initial GET request is a simple subset of what we need to consider. This is made up of two parts:

  • Load any initial data and convert it into a form suitable for rendering.
  • Render the form as HTML and transmit that to the client.

Gathering initial data is simpler for forms that are initially blank. Forms that provide editing will need to get the original data, convert it to a values that are suitable for HTML inputs and then render the form.

The POST request

When users fill in the form they post their results back to the server. Regardless of the client-side code, we need to do a number of things.

  • Parse and validate the entered values.
  • If valid, perform the intended action.
  • If invalid, render the form with error information.

If the intended action is an edit to an existing logical record, we might need some of the original data from the database, and we might need to convert the POST data into a form suitable for our application or database.

If the data is invalid then error information needs to be rendered back with the fields. You need to know which fields are invalid and how to help the user to correct what they’ve entered, and this needs handling in a way that can be rendered back into the form.

Converting data

HTML forms support key-value pairs or text, that’s it! Any structure is purely coincidental and is not understood by the browser. There is also support for files, where the value is binary and comes with a content type, but that’s beyond the scope of this post.

Often, when working with structures data such as objects and arrays field names can be generated according to a pattern. Using Node JS modules such as flat help with bi-directional conversion of structures. More complex conversions will involve numbers, boolean and dates.

The conversion only needs to be done on valid data. If you are converting invalid data it is often a mistake.

Rendering and validation

Rendering is needed for both the GET and POST requests so it makes sense to use the same trunking to do this. The only real difference is that the POST request involves rendering error information. This is where forms get complicated as you might add classes to fields to show their status and include a header with more complex information.

You might also have some optional client-side helpers that make the user experience better and provide client-side validation and assistance.

The validation rules are often rendered to the client as HTML5 or ECMAScript but must also be enforced on the server to avoid security issues. Inconsistencies in these rules create opportunities for users to exploit the system and for valid data to be rejected.

Keeping an abstract data structure that represents the fields, their validation rules, rendering hints, current value and any error status, makes it easier to develop rendering templates field and provide consistent rendering and validation for all conditions and field types. It also  makes debugging much easier as you can inspect this structure more easily that looking at the complex HTML output.

Optimistic Concurrency

Optimistic concurrency is extremely important when editing data on the web. It allows people to modify data out-of-process without the server keeping locks.

There are two common models, although sometimes something more complex is done, these are:

  • Fail if modified and
  • Last one wins.

The first approach requires sending information with the form that lets you determine if the data on the server has been changed by another user or process since the form was presented. This might mean a hash or revision identifier of a logical record, such as that used by Couch DB, or it might be a complete copy of the original data so that data can be individually checked.

The second approach is simpler and often the most appropriate, depending on the granularity and nature of the data. With either approach, one user loses out, the difference with this approach is that the first user loses and doesn’t get to know about it. When considering how and why data is updated there are many scenarios where forcing the second user to lose and try again gives the same result as ignoring what the first user did.

Some database technology includes optimistic concurrency systems. It is often useful to combine the initialisation step into the POST request to get what we need to overwrite existing data without error, especially when implementing a last-one-wins approach.


This post has been written to cover some of the issues that can arise when developing HTML forms, it is not comprehensive and assumes the usual level of support in your HTTP server for dealing with general HTTP request handling issues.

Quick and Dirty Swap File (Linux)

Need a swap file on a Virtual Machine that didn’t come with one? Don’t care about the size, name or location? This is for you…

fallocate -l 1G /swap
mkswap /swap
chmod 0600 /swap
swapon /swap
echo "/swap none swap sw 0 0" >> /etc/fstab

I use lots of tiny virtual servers for development and testing and on most hosting providers they don’t come with a swap file by default. Sometimes I need a little more memory to get over a hump, such as installing or compiling software, but not for general operation so a bigger machine is overkill. I found lots of step-by-step guides online but I couldn’t find a full cut & paste script very easily online so I’ve posted one here.

Not for use in production, unless the size and location all happen to match your requirements.

Shrink QEMU/QCOW2 Images

Needed to shrink a full directory of QEMU QCOW2 images today that had grown over time with snapshots and general use. While there is information around on the web about it, the focus is on single images and the code provided are just examples.

Here’s a script to keep around that shrinks and compresses all images in a folder, and shows progress:

for imageFile in *.qcow2
    echo "Processing $imageFile"
    mv "$imageFile" "$imageFile.orig"
    qemu-img convert -p -c -O qcow2 "$imageFile.orig" "$imageFile"
echo "Warning: Test images before deleting the .orig files"

For convenience I stuck this in my QEMU image directory and made it executable. Be sure to test images and then delete the “.orig” files to save space.

Software 101: Understanding the Different Types of Software in the IT Landscape.

The Software landscapeSoftware used to be installed on the computer in front of you, but the internet has led to cloud computing, SaaS and more. What does this mean for your business?
If your business used computers 20 years ago, you might only have encountered one type of software. Desktop apps were installed onto individual machines by way of floppy disks and CD-ROMs, and that was it. However, now there are multiple categories of program, website, app and more.
If you are unsure about what each category of software entails, then look no further; read on for a handy guide that should clear up the differences.

On-premise Software

This covers several sub-categories; however, the key thing that links on-premise applications is that they are installed and run on computers that are physically in your building. The software and the computer are on the premises, and most of the time it’s the computer sat right on your desk in front of you.

There are variants of this. For example, a group of people can all connect through their own computer to another computer, often called a server. However, the server is still geographically close by. We’re going to start off by explaining the various types of on-premise software and then move on to the other main kind: web applications.

Desktop Applications

First of all, desktop apps. These are the programs that were installed on machines before the internet, and are still widely in use now. All the name means is that these are installed on individual machines, usually working independent of any internet connection.

Examples include Microsoft Word and Adobe Photoshop. You may have got these from a disk or a download, but either way the actual software program is to be found on your computer. These apps have some advantages, such as not needing to be connected to the internet to use them. This makes them convenient, and can make them more responsive or richer in features.

For some intensive jobs, such as video editing, a desktop application is still the norm. However this is all changing as technology improves. Desktop apps can be more customisable, as you could in theory modify programs to fit the needs of each user. In practice this is rare except in large enterprises, due to the high costs involved.

Client Server Applications

Some business software is built to run as a ‘client server’ application. This means some of the software is installed on another computer and several people access it through a network using a separate desktop app.

It’s called client server software because part of the software is on an individual’s computer (the client) and another part on a central computer (the server). All the computers are typically on the same office network and there could be anything from hundreds of people to just a handful using it.

A server is simply a computer that you can access remotely, so you can install a program on it that multiple people can use. However, you will need a specific network setup for this; it is not enough to simply connect to the internet.

Client server applications have been common in offices since the 1980s, especially for things like financial accounting, order processing or group email. Since the explosion of the internet this is changing, as we will explain below.

Mobile apps

Finally, mobile apps are essentially desktop apps built for installation on a smartphone rather than a full sized computer. Many mobile apps will access data or other systems through the internet, but importantly they must be downloaded and installed. This is typically done through an app store controlled by one of the platform vendors like Apple or Google. Because of the ubiquity of these devices, there has been an explosion of mobile apps since the launch of the iPhone in 2007 and fortunes have been made.

There are good reasons to build a mobile app to meet this demand, but it is worth bearing in mind that there are three major code bases for mobile: Android, iOS and Windows Phone. Creating the app for two or three of these can increase the cost of ownership significantly, with coding, maintenance and updates all taking longer and costing more.

On-premise Pros and Cons

Until the internet came along everything was an on-premise app with the vast majority created to run on Windows computers. Buying a software licence means owning something tangible, which perhaps felt good. However, with that ownership comes responsibilities and costs. It’s your job to install the software and also install updates over time.

That may not sound bad, but with an office full of computers it soon becomes time-consuming. When installing updates you may encounter compatibility issues with other apps you own and you may need to test how new versions work together, which takes more time and is more expensive.

In addition, backing up your data is your responsibility; not to mention guarding against viruses and recovering should you encounter a malicious attack. All this means that the cost of the licence isn’t the same as the cost of ownership, and for many businesses this lifetime expense is critical. All these factors have led to an industry shift away from on-premise software.

Web applications

Web apps often look the same as desktop apps, except they are installed on a remote server and – importantly – run inside a web browser like Google Chrome or Mozilla Firefox. This means anyone with access to the internet can use them and they never need to install the application on their computer.

To use a new piece of software is no harder than going to a different website. Gmail, Microsoft Office 365, accounts system Xero, Salesforce CRM, project-management app Basecamp, and social media tool Buffer are all examples of services where you don’t need to install anything to use them; they are simply accessed online. The features these applications provide would have traditionally only been available through one of the desktop apps described earlier.

Web Application or Website

Whether something is a web application or a website is a fuzzy line. Suffice to say that a pure website generally only contains pages of information, much like a book, while a pure web app delivers pure functionality to get a job done much like a desktop app. However many websites, perhaps most websites, provide a blend of both.

Users browse through pages of text and look at pictures or videos, but they can also log on and send each other messages, buy theatre tickets, order flowers or even submit an expense form through their employers web based finance system. Many web applications have become popular websites and many popular websites have added functionality that makes them in some sense web applications.

Technically web applications and websites differ in their complexity. They are both hosted on remote servers, possibly even in a different country. At their most basic level, websites are a collection of documents with little or no programming code other than HTML. However the complexity rises rapidly with the functionality being delivered and modern web applications can be formidable.

Content Management Systems

A content management system (CMS) is a popular kind of web app that allows businesses to write and maintain the content of a website themselves, without having to learn HTML or rely on a web design company to make changes. One of the most popular examples is WordPress. Through this, you can design a website, edit it and continually update it without using any kind of programming knowledge.

There are many varieties of CMS, and these are often highly customisable with plugins that provide complementary features or functionality. Sometimes content management systems are used as the basis to deliver specific business functionality through creating bespoke plugins for them.

Because only the plugin is bespoke this approach can often be cheaper. However, when the functionality you require is extensive these savings soon disappear. They are replaced by compromises imposed by the CMS architecture that often increase costs.

Software as a Service

Software as a Service or SaaS is actually a business model rather than a technology and means using a web app and paying for it through a subscription instead of owning a licence. This makes a lot of sense, as it removes a lot of the disadvantages of owning apps. For example, you never have to worry about maintenance or updates, as the company that owns the software takes care of all of this. You just have to pay to access it.

For SMEs, this has become a very popular option. You can access up-to-date software for as long as you need it, without any associated hardware costs. These apps are typically easy to learn, and they are designed for in-browser use and most people already know how to use a browser, the documentation is just a click away and as it’s much easier for customers to switch to alternative services it’s important for SaaS companies to provide good levels of customer service.

The Cloud

You might have heard about ‘the cloud’ when referring to web apps, as it seems like lots of technology companies are offering cloud services or the opportunity to move your systems into the cloud. An article like this wouldn’t be complete without a short explanation.

It’s really about the shift to computing capacity being offered as a commodity service by some of the big technology vendors like Amazon, Microsoft and Google. A host of smaller vendors have now joined in with similar services.

These businesses offer access to computers and storage through the web and allow other technology companies to run their software applications on them, paying for what they use through metered usage much like an electricity bill. All the computer hardware and data centres are owned by the cloud provider and the precise location may not even be known by the customers.

This allows these cloud providers to drive down costs through economies of scale, offering very low prices. The companies using these services don’t need to buy any hardware or rent space in a data centre, and they can quickly increase or reduce their computer processing power or storage to suite their needs and demand.

In a similar way to SaaS, the cloud is a change in business model rather than any particular technology. In fact you may also hear the similar term “Platform-as-a-Service” which is used synonymously by some vendors.

The Future of Business Software

The world is becoming increasingly computerised, and software will continue to evolve. At the moment, SaaS has grown massively in popularity and seems to be the future. More and more applications are being created for this, as it seems like a great financial option for both users and creators.
This makes now a great time to be a tech entrepreneur, as the web provides a low-cost mechanism to reach customers globally. The demand for unique software is there and the SaaS model means more people can afford it more than ever before.

Building an Agile Team

Teamwork in action

It’s possible to build successful software in all kinds of ways but the need for better commercial outcomes has led to a shift towards modern work practices referred to as lean or agile.

Adopting a lean or agile product development process has its greatest benefits when including the entire organisation is included. This wholesale approach means that everyone from the CEO to the most junior members of staff are involved. Any decision making – from roadmap ideas to actual production deployments – is based on the key principles of agile, aiming to improve flow and waste elimination.

However, this isn’t always easy to do, especially when you need people from different departments and backgrounds to work in the same way. Due to the collaborative nature of software development, many businesses choose to start with the development team when introducing agile work practices.

Whether you are a startup with just a handful of people working together or a huge organisation with thousands of employees, agile management can help make the most of your business and make sure the work you deliver is of the highest standard.
But how do you go about building an agile team? There are some key principles that can help you make the most of the foundations of this working style. Here we cover the basics of creating an agile team and how this can help your business:

Agreeing to Change as a Team?

There are differing opinions on how to adopt an agile working approach; whether all members need to be in the same place, or how people’s roles may change to get the best performance. However, the foundation of any agile team is agreeing to focus on good outcomes and ways to further improve a steady flow of working, production quality, and individual features. Everyone involved must agree to improve continually.

Agile teams often consist of professionals that have a broad range of complementary skills, with emphasis on them collaborating to deliver one main project together. This helps avoid dependencies on outside people or other teams, which can often cause delays.

In software, this means you’ll have web developers working alongside designers, with project managers liaising and supporting them. By identifying and focusing on the overarching goals, the team can be united and make sure they are all working towards the same result.

For any business, good commercial outcomes are everything and for this reason agile working methods often implicitly prefer these as a measurement of success. This helps to encourage cooperation and trust, which is a vital part of running a successful agile team and delivering a high-quality product for a competitive price.

Encourage Autonomy

A steady flow of working features is the sign of a functioning agile team and this often means trying to make progress with incomplete information. Traditional teams follow pre-specified requirements and may stop if they don’t have enough guidance. This can significantly hinder work production. Encouraging acts of leadership and autonomy helps a team solve its own problems and keep progressing forward. A lack of autonomy can lead to hand offs and work being stalled while approval is sort increasing costs and lead times.

It’s also important that the team has a culture where making mistakes is a point for discussion, and not a finger-pointing exercise. A key part of agile working is that at each stage of the project, practices are evaluated, and suggestions for improvements are taken into account. If people are worried about the consequences of admitting they made a mistake or that they could have done better, it’s much more difficult to find areas to improve.

Visualise and Measure What is Actually Happening

This may sound obvious but it’s not something that is generally done outside lean / agile circles. What’s important is that data is collected and easily seen by everyone so that the team can reflect on their performance proactively in real time. This differs from the traffic lights often used in traditional weekly project reports, which show a trailing indicator of performance. Agile teams commonly use a shared board, which is updated throughout the day and referred to constantly by everyone. Another common practice is to hold a short daily meeting, often standing in front of the board, with the whole team.

Be Willing to Change and Adapt Constantly

Although there will be a goal for the project, it’s important that teams have the ability to revise or even completely change aspects of the work. This may be adjusting behaviour or priorities through collaboration with project stakeholders or changing work practices and is a crucial part of being agile.

This flexibility helps a team make improvements when they are needed and try things out that they may later reverse. It’s common to have regular sessions to reflect on the process and discuss things to either stop or start doing. Adopting continual small improvements is a practice adapted from lean manufacturing and in particular the Japanese motor industry where it’s more commonly called Kaizen.

Build Small Pieces and Release Them Quickly

At their core lean / agile methods help improve the commercial outcome of projects. They do this by getting working features into the hands of customers quickly for them to verify that the right things are being built. If customers aren’t happy, money is saved by replanning to rectify the situation instead of soldering on in the wrong direction. To work in this way it’s critical to build small pieces of the system at a time and strive to release very regularly to real users. If progress is blocked for some reason experienced agile teams will swarm around the problem to unblock things instead of waiting for a solution and starting another task in the meantime.

Building an agile development team takes time and patience. Many industry veterans have successfully build software for many years using traditional project methods. However that doesn’t mean that those project couldn’t have saved money or been closer to the users needs through the use of lean / agile methods of working. The key is to gather willing participants, be open to change and keep making small improvements as learning continues.

How to Run a Code Club – We’re Hosting a Volunteer Session at our Offices

Code Club event at NewRedoWe’re hosting a Code Club community training evening to allow anyone who is thinking of becoming a Code Club volunteer an opportunity to find out more about Code Club and get an insight into what to expect and how to get involved. Linda Broughton is the Code Club Regional Coordinator and will be running the session and doing all the talking. The straight fact is that Leeds doesn’t have a Code Club running in every school and it should have. The only way that can happen is with more industry volunteers.


Code Club is a nationwide network of after-school coding clubs for children. All the clubs are led by volunteers and this training is designed to give volunteers all the information they need before they start running a Code Club.

The training will cover:

  • What is Code Club?
  • Volunteering with Code Club
  • Working effectively with schools
  • Safeguarding
  • An introduction to Scratch & the Code Club projects

Who is the training for?

  • Anyone who is considering running a Code Club and would like some more information before they sign up
  • Anyone who has signed up as a volunteer but has not yet started running a Code Club

Volunteering for Code Club will be the best thing you do this year. Guaranteed.

Building a Test Email Server

This post explains how to set up an email server on GNU/Linux that can be used for testing applications. It allows you to test that emails are correctly addressed (except for BCC) and allows you to receive the emails and view them in any POP3 email client just as if they’d been received normally.

These instructions assume Ubuntu 12.04 LTS, and that we are installing our SMTP and POP3 server on the same computer that is running our application, and that the application uses the local SMTP relay.

First, install postfix, which will be your SMTP server.

sudo apt-get install postfix

When asked what type choose local only, it doesn’t matter what you choose as a domain name.

Next, add the following line to /etc/postfix/

virtual_alias_maps = regexp:/etc/postfix/virtual

This sets up our use of virtual alias maps which we’ll use to redirect all the mail in the next step.

Then create the file /etc/postfix/virtual and add the following line to it:

/.*/ mailsink

We’ll need a user creating for this mail dump account so run the following:

sudo adduser mailsink

Be sure to give it a reasonable password as this will be used to access emails later.

You may also wish to block spam if the server is public facing, in which case add the following line to /etc/postfix/

smtpd_recipient_restrictions = reject_rbl_client, reject_rbl_client

Now we need a POP3 server installing:

sudo apt-get install dovecot-pop3d

The following command enables clear text authentication over POP3:

sudo sed --in-place "s|#disable_plaintext_auth = yes|disable_plaintext_auth = no|" /etc/dovecot/conf.d/10-auth.conf

The following command sets the mail_location in Dovecot:

sudo sed --in-place "s|#mail_location = |mail_location = mbox:~/mail:INBOX=/var/mail/%u|" /etc/dovecot/conf.d/10-mail.conf

The following command ensures that Dovecot has permission to delete emails from the inbox:

sudo sed --in-place "s|#mail_privileged_group =|mail_privileged_group = mail|" /etc/dovecot/conf.d/10-mail.conf

Some e-mail clients will require you to specify an out-going SMTP server too. To make this easier you can enable SMTP access to this server. However, this will allow junk mail to be delivered to the test mailbox so avoid it if you can. To enable SMTP access run the following command:

sudo sed --in-place "s|inet_interfaces = loopback-only|inet_interfaces = all|" /etc/postfix/

Any mail sent using this SMTP server it will be routed to the mailsink in-box.

If you are using UFW then enable POP3 access:

sudo ufw allow pop3
sudo ufw allow smtp

Now restart the affected services:

sudo service dovecot restart
sudo service postfix restart

You can now configure your email client to access the server using POP3, you should configure it to leave messages on the server so that you can test using different email clients. If the server is accessible over the internet then you can configure GMAIL and HotMail to access the mail using POP3.

Run the following command to send a test email (to verify that all email addresses get re-routed just use a real email address that you have access to, if this email doesn’t reach that address then everything is OK):

echo $'Subject: TEST\nHELLO\n.\n' | sendmail -t

Product Backlog Tools – How to Prioritise Using the Cost of Delay

Whats next image

Have you wondered how to arrange a to-do list in the correct order? Probably not. There’s nothing to learn, right? Clearly, prioritising what to do next shouldn’t be hard. But don’t underestimate how costly a casual attitude to this can be. Most of the time prioritisation is done on gut feel, but a sound mental model of the economics will speed up planning, help resolve differences of opinion and bring in money sooner.

We use an input queue, often called a backlog, to pull work items from. The backlog is nothing more than an ordered list of things to do, with the next most important thing at the top.

Given a list of work, most people go about prioritisation by choosing what’s important to them or by identifying what’s important to users based on gut feel or customer feedback. Gut feel or feedback is fine but it’s not the whole story and comparing the right criteria can make the ordering easier and more profitable.

A cleaner, more measurable approach is to focus on the opportunity cost. The opportunity cost is fancy way to say, what will it cost not to do something. Another phrase often used is the Cost of Delay or COD. A commercially clear way to compare two backlog items is to ask, “What will it cost to delay doing this item compared to the item below it”. By placing the more expensive of the two above the other, then moving down the list a Cost of Delay ranking will emerge.

The Cost of Delay could take the form of lost sales or subscribers, in other words market share. Or, in the case of back office process automation, the perceived savings in time and labour not kicking in. Putting exact financial figures to these isn’t easy and is generally unnecessary. Reducing the comparison to the single cost variable and using whatever past data is available to make a judgement, is normally enough. There are some interesting techniques and collaboration games that can improve speed and objectivity still further; but that will be covered in a future article.

Tracing everything back to money may seem cold and ruthless, but in the world of commercial software it is reality. Even cool things like beautiful, contemporary design benefit from this Cost of Delay comparison. Users tend to place greater trust in pleasing and nice to use websites and this trust factor will influence sales and market share.

Sometimes the order of work seems inevitable and based on perceived feature dependencies. For example, it may seem essential to for users of an on-line web service to be able to create an account profile before doing anything. With care, creativity and engineering jujitsu this could be reversed allowing other revenue generating features to be released earlier. If you’re sceptical checkout how Doodle handle no user accounts.

Ultimately breaking functionality down into mini features that may be delivered early and crafting the build schedule with an eye on the Cost of Delay is the mark of a canny and experienced product development team.

Building RaceBest

RaceBest LogoThis year saw the launch of a new service from Leeds-based start-up RaceBest Ltd. NewRedo were responsible for building their online system and we continue to host and support it.

In building the system we utilised some of the latest open-source technologies to build an interactive website for the running community. The service includes a race calendar, online entry, results and reviews, all managed by RaceBest staff using a custom-built administration section.

The front-end uses mainly client-side rendering powered by AngularJS, backed up with server-side pre-rendering for clients that don’t have JavaScript or search engine spiders. The back-end uses document-oriented database system CouchDB and node.js (proxied through NGINX) as the application server. Running on a small server, keeping the carbon footprint low, the website is really fast and was barely under load when it took the 750+ entries for their first major event.

Agile Yorkshire Software Development Community

Every month NewRedo plays host to the Agile Yorkshire community group. Fifty people come together to listen and swap ideas. It could be more but we’ve only room for that many.
At NewRedo we’re very keen that the process of product development a team effort and not just about writing programming code, and what’s great about Agile Yorkshire is it attracts people from all disciplines. From engineers and UX designers to senior managers, they all come together in the same room for the same reason – to open their minds to better ways of work together and creating things with software.
To keep that diverse group happy requires a careful mix of topics and levels. Helping project managers with their knowledge of BDD frameworks and hard core programmers appreciate the cost of delay provides value to everyone in the end. Another key mission of Agile Yorkshire is to promote new ways of working to a new audience, so having some entry level content also plays a part.
This month Grant Crofton and Mike Burrows provided contrasting slots on F# and Kanban respectively. It was a balmy summer evening and discussion flowed on afterwards into the pub next door. The best part about any community gathering.