Specification by Example: Tooling Recommendations

Specification by example has a history that is closely linked to Ruby. The main software tools and development methods came from work on Ruby projects. As the method has started to attract a wider audience development tools and links to Microsoft .Net platforms have begun to appear.

At Intechnica our approach has been from two directions. An important aspect of the methodology is the use of continuous integration and code build, and that unit tests and inter-module testing is built into the software framework. This will prove that the code providing the functionality for the stories is working properly. So the development teams are looking to build continuous integration and unit tests into their software development processes. At the same time we don’t want to be tied to a specific and expensive development platform. Current processes are predicated on using Team Foundation Server – if the testing components are added to this it becomes an expensive solution.

The area that the solution assurance team are working in is the setting out the scope of the project and ensuring that the requirements reflect customer business requirements. Most importantly, it is then taking that living documentation and collaborating with the development team, PM’s and the customer to refine the details and identify the key business flows. Initially the outline ideas are put together using whiteboard sessions. These are then recorded into a mind-map software solution. There are many different products out there – many of which are open source. An important part of the selection criteria is that they are collaborative and that they have flexible import and export capabilities. It is no good having meetings and having working groups and then finding that they can only print output and can’t have shared input. The solution we have chosen is Mindmeister. Like many tools used for business analysis, testing and development, Mindmeister is a Software as a Service (SaaS) solution. This provides great flexibility and reduces the need to have software installed onto closed PCs or to use licensing dongles and allows easier upgrades. The software is free to use with limited number of mindmaps in the basic version; the personal version with greater number of maps and export formats is £4.99 a month, and the pro version is £9.99.

Once the project has a satisfactory outline and the scope has been captured with a rough outline of the solution we then use the Speclog solution to start to write the user stories. Speclog is a tool that is designed to look at business requirements and allow them to be described fully. For the story based process of specification by example this allows us to breakdown an application into its constituent actor goals, business goals and user stories. Each core feature is broken out into different user stories. Again it has strong collaboration characteristics. We have chosen a server version that allows a number of people to work on the same project. Linking back to specification by example, the output produced is in Gherkin language.

As part of this process, this is the time when the requirements traceability and the summary of requirements documentation is written. This is so there is an agreed understanding of the scope of the project. The tools are put in place for later test phases that will validate the delivery of the business capability.

This takes us back to our initial introduction to specification by example where we worked on user stories and used a Ruby tool called Cucumber that allowed the project to be detailed. Cucumber also produced and validated Gherkin code. It is a descriptive language that allows requirements to be described in the Given When Then format.

Given that I am a user of the system
I want to enter a valid order
So that products can be dispatched

In this example a number of characteristic scenarios would be used when describing the order entry feature. An order should have a valid product, reject certain types of product, have correct dispatch details, have a correct order scheme, contain shipping details, reject incorrect quantity amount and produce an order summary. From this the Gherkin user stories would then be produced and the developer would start to work on the features that provide this software capability.

And this brings us to our last software tool. In order that user stories can be discussed with the users and reviewed internally during development and testing, we need a tool that can bring together the Gherkin output in a Wiki style web site. This is key to the specification by example method where the output needs to be the living documentation and reflect the improvements and changes that are made to the requirements and how they are met through the project. Most importantly, both the SpecLog description (connected to the development cycle) and the tool used to monitor the progress of the requirements and display their maturing needed to be closely aligned. In theory this can be done by putting the Gherkin files into a version control system where the software that provides the capability will be developed, but because we don’t have the right connectivity and the development processes are not yet in place, we have put in a simpler solution. The user stories are being published into a file share and then put out into a Wiki style display tool called Relish. This can be used in two ways, directly by writing Gherkin feature files using simple editors such as Notepad++ or Sublime – plugins to these two products allow them to validate Gherkin code.

The linking together of the solution assurance and development processes are the next area for examination. When developing features there won’t necessarily be a direct one-to-one relationship between developed code modules and the stories. Some modules could be used across a number of stories – but in different functional capabilities. There will also be a much larger number of software modules and the relationship between the stories and the underlying code needs to be monitored. This area will require a lot of detailing work.

The final step will be to put the agreed user stories into our test management software tool (we use Practitest) as requirements and the write the core end-to-end process flows as test scenarios – with the test cases described within reflecting the different functional capabilities.

Conclusion

There are a lot of good quality software tools to help work with Specification by Example. Collaboration and flexibility are key – as is basic reliability. A time can be seen when there is a end-to-end link up to take requirements and produce testable artefacts. The next step in the process will be to work out how the development tools will hook into this process and more importantly, if we can follow the process fully, is it possible to produce a hybrid approach that combines traditional methods with this much leaner approach? The biggest challenge with this process is that it works from a top-down perspective – where traditional software methods use a combination of top-down and bottom-up to ensure that no gaps are inadvertently produced. It is addressing this issue that will be the most pressing challenge for specification by example.

See the other posts in this series by visiting David’s profile page.

Agile and QA approaches – Requirements Traceability

Following on from my post two weeks ago about specification by example and application maturity, this piece is about requirements traceability.

Software development processes have traditionally worked from signed artefacts and signing up to agreed pieces of work. This would normally describe a set amount of development time, some testing time and some support time to resolve defects and help bed-in the new application. An important part of this process is a description of what will be delivered. This is a key document in the specification by example process. It is important when creating the original statement of requirements and describing the user stories in tools such as Speclog that all requirements have been captured and that there is a common understanding between the project sponsors and the suppliers of how this functionality will be used. By describing key business flows in terms of the behaviour of the application an opportunity is provided for areas of discussion and the understanding of the supplier and the customer can be explored.

These activities drive the test approach. Traditional testing analysis would follow on from here. Often this would be long weeks of Business Analysts creating specification documents and from there test analysts would follow the same path and develop test packs which examine and validate these requirements. In the mean-time developers will work with the documents and the outline discussions, and begin their development approach. In the old days this would have created detail specification of requirements, a detailed development approach and detailed test approach – all of which would have to be approved and signed off. This produces a lot of documents and the big problem with them is that it assumes the requirements are fully understood at the beginning of the process. Any changes are managed through change requests, which require impacting and updating of all these documents. It is not a flexible process but although slow, delivers good quality software.

In the same way that Agile has helped to remove all the unnecessary documentation in development and testing of new applications, specification by example is looking to broaden/develop this concept to address the requirements itself. In the area of Quality Assurance the big challenge with ever more complex systems is measuring what areas of the requirements are being met by which development activities and then validated by which testing activities. In its broadest sense specification by example attempts to address this by a straight-forward approach that states that tests will be developed for each requirement specified in the user story.

The challenge is what artefacts we need to create and how they fit in with the user stories. For instance can/will test approach documents be taken straight from the stories that are created in the analysis. The examples and systems that have been discussed in Agile conferences and across the web have been small with simple business processes which have limited number of functional combinations of application modules. And these systems have been created from scratch or are small improvements. This work hasn’t taken place with existing systems that may not have full documentation or where new software and changes to existing applications take place.

This is where testing theory and previous processes can supplement the given process. It is important that the relationships between the requirements and stories are understood. What are the most important part of the applications ? Which are the most complex? By using the traditional requirements traceability it is possible to create a an application relationship map – which then can be used to drive the test plan and more importantly be used to manage the delivery of the application functionality. This will help with deciding the critical path and the main release points where key modules from a testable business process.

Where defects are being seen then it can steer back to the stories and allow us to understand whether there is something missing from the requirements or something has changed or details need to be more fully understood. This takes us back to the largest flaw in current development/test methodology. When the initial analysis takes place assumptions are made using the business/test/developer analyst experience and understanding of the business. The won’t necessarily align and more importantly they will change as more work is done. It is vital that all that information is shared – and that is at the heart of collaborative processes such as specification by example try to engender. But at the moment the process seeks to implement a one-size-fits-all approach and only looks at basing everything on the user stories. There may well need to be additional process activities.

What we want to find out over the next few months is how requirements traceability works in the Specification by Example area. It has proved to be a valuable tool in Agile projects – where there isn’t the time to record all the details of all the requirements and in Waterfall where the long time periods of development and testing activity can be managed through the matrix. It has weaknesses – it doesn’t work well across a large number of conflicting requirements and doesn’t work well when the business capability being developed splits into small components. But it will be interesting to see what it produces and future blogs we will be recording what we see.

Specification by Example: Defect management in Agile projects

It is often thought that Testing/Quality Assurance is just about finding defects – or as some people think – playing with the software and seeing what works and what doesn’t. As usual the reality is a lot more involved. The finding of defects is only one aspect of what QA does. When preparing a test approach you need to ask some fundamental questions that appear to have obvious answers, but when you think about them a bit more – you are not so sure.

• How much testing needs to be carried out before the application is ready to deploy?
• What quantity of defects found in a test phase demonstrates that the application meets customers’ acceptance criteria?
• What business activities need to be evaluated to prove that the application will be able to carry out business tasks as required?

The third question is actually the key. Any project will have a set of opportunities or a capability that a business needs. They will measure the benefits of employing people to create code and time to deliver the project by the value benefits these capabilities add to the profits of the company. This could be from consolidating a number of accounting systems, being able to sell via an electronic market place a new service or being able to manage more effectively the routes that delivery drivers take to deliver stock. All add value to the bottom line. To achieve this the project has to be very clear about what new software or changes to existing applications in a language which there can be no ambiguity. Working with the developers, Quality Assurance exercise the software and measure whether the code meets these needs. And this is done through looking at the behaviour of the application rather than trying to take apart all the constituent parts. Because specification-by-example builds testing into the development process – the lower level code modules will be tested and validated with the requirements. So low level noise (syntax errors, validation errors etc.) shouldn’t be present. So the QA focus when using the application will look at validating the user-stories and from that examining the requirements. So how many defects found in a test phase mean that the software is acceptable for use? The key measure used by QA is to look at the spread of defects by functional area. Traditionally these areas will have been mapped in the early part of the project via a traceability document. This document measures the business criticality and complexity of the requirements and relates that to use stories on how the new functionality will be used. The challenge is going to be that in an Agile context, making it lightweight enough so that enough information is provided but it doesn’t take a long time to produce. The drivers behind the Agile methodology emphasise only doing those tasks that are necessary – to drive the release forward and understanding what tasks need to be done on the critical path to deliver a release. Traditionally the readiness of an application for service was proved by the level of defects seen in each functional area. Traditional QA activities come from a waterfall past where specific quality gates must be achieved and everything moves in slow sequence. In the new methodology it is necessary to only do the minimum amount of testing that proves that project requirements can be met and that the application will perform provide a reliable and effective level of service. Without the old techniques of testing all parts of the application the focus moves to testing those areas that are critical to the purpose of the change and support the business capability being delivered. So it is possible to use the old technique but it is necessary to put it in a new context making the process lighter and more flexible. This is just the start of some big changes in software development methodologies. When defects are seen they will have much higher impact and could have slipped unnoticed through a regression change or integration point where two modules have been brought together for the first time. The new processes will need to give this information context so that critical issues are fixed quickly without impacting overall delivery times. Testing will focus more on end-to-end process flows (business stories) and less on small components such as data validation and software branch variation. This will not mean that the testing won’t be done, but it will be done at an earlier point. This is only one area where “Specification by Example” will make subtle – but fundamental changes to application delivery. Important Agile behaviours such as “fix-first-time” – “active collaboration” will need to be enforced to drive through quality. But old measures such as defect latency, burn rate on feature delivery and making decisions on what will be a realistic time for launch of the application are going to be equally important. These are going to be exciting times for project methodologies and it will be interesting to see where we are in 6 months time.

The Changing World of Testing

I’m part of Intechnica’s (fairly new) QA team, who are bringing in new perspectives to Intechnica, as we are from the testing world and Intechnica is traditionally a software development (as well as performance assurance) focused company. My background is not only test leadership but also test culture, architecture, working on methodologies and helping them align with the places a business is going to go.

In our world there have been big changes since the banking crash in 2008. In that time projects were large and all software testing was big budget to go with big software and big applications. But increasing pressure to make application changes faster & smarter has resulted in radical changes in how software is developed. Now applications are developed strategically and are being devolved into web services and cloud applications – Software as a Service (SaaS) – where focus is purely on the service, not how it is served.

The software engineering world is now focused on application capability, making sure requirements are clear and that they are tested as early as possible. The old job roles of test analyst, lead, and the follow on role of Quality Assurance, where people like myself would validate the work of out-sourced testing units to ensure that requirements are in line with business expectations, are melting away. Now test is being driven throughout the software life-cycle – from initial requirements to business validation. So what will these roles turn into ?

There are quite a few discussion pieces going on out there now; Some predicting that formal test and test groups will be go the same way as operations and capacity planning (what I used to do). Why would we need specific testing activity when testing is part of all project work? This article discusses this possibility: The Shrinking Role of QA

I agree that testing is changing. The question is, how do we become part of the new conversation? I think the old test/QA skill sets still remain important. How will the new software function be used? Will the expected behaviour of the application match the business opportunities outlined in the proposal? More importantly (and this is the bit that is certainly beyond current defensive method thinking right now) are there opportunities to give the users of the project a better product which brings together a couple of requirements in a more structured useful mechanism? Can we deliver exactly on time with just the requirements the customer wants instead of waiting for all the functionality to come together?

Test Analysts, Leads and Architects need to keep doing what they do best – talk through the requirements with all parties involved – work with development teams and help make sure the bigger picture of the overall solution isn’t lost – and working with implementation teams on how to get this out to the customers with the appropriate business proving trials – and, most importantly, tracking and validating the requirements, making sure any changes are done in a structured way that keeps all groups’ view of the changes aligned.

The names will change, but the role will continue to be planted at the heart of the project delivery mechanism.