Delivery Best Practice: The value of “Rambo” testing

This is Part 2 in a series on Kanban’s delivery best practices.

At Kanban, we create test scripts that verify the proper display and behavior of every item that the client signed off on in our functional specification. When we plan our QA stage, we also take into account two other types of testing: regression tests of existing functionality, and Rambo testing.

Rambo testing is somewhat of a misnomer: it’s not always about doing whatever it takes to break the new functionality. That would be too easy. It is about going off script in a thoughtful manner. The idea of Rambo testing at Kanban is to find legitimate ways that the customers or users might work with the system that are not explicitly documented in our functional spec, and thus wouldn’t be covered in our regular scripts.

For example, there are some situations where we can assume that users will try to hack the system. Game players will want to cheat. Students will want to change their grades. Bloggers will want product images before they’re available to the general public. After we test certain specific methods of hacking based on security best practices, we will need to Rambo test, creatively trying to discover other ways to get around our security measures. This is best done by a software engineer who did not work on the project.

Another example is testing content entry in a content management system, or CMS. This is where we need to be especially thoughtful in our Rambo testing. It’s not fair to file a bug that says the wrapping in a narrow column fails to accommodate “fjkeowfjiwopamkverjipvjtghioweiomnweknktl” (assuming your site is not in German). But it is fair to file a bug if, for example, the narrow column could legitimately be expected to contain a URL without hyphens. This is best done by planning ahead to have as much real content available during QA as the client can provide.

In the same vein, we wouldn’t want to Rambo test the display of media with assets that are nothing like what the client would realistically upload. So what do we do if the client doesn’t have real examples yet? We don’t Rambo test it. We test against only what was specified, and we declare from the start of the project that there should be a phase after this launch where the system is refined to meet needs that we can’t know until it’s put into use.

Where Rambo testing really gets the use that its name implies is in testing a tool or application. This is a great time to get someone who wasn’t on the project to try to break your app. It’s a bonus if that person can be similar to the target audience, but it’s more important that they’re creative about testing. Does the tool work while the user is doing other things on the site? What if they click very quickly? Hit the Back button? Have the same page open in two browsers at once?

We document the results of Rambo testing by having the QA tester briefly describe what they did and if it passed. From bugs filed during Rambo testing, we can see if any methods that the Rambo tester used should become a part of our normal QA process next time. With this approach, we are consistently improving our quality assurance process while delivering precisely what our customers and users expect.

Perils of Open Source Software

We love Open Source code and attempt to use it wherever possible. This is much easier on the frontend (HTML, JavaScript, CSS) than it is on the backend (Depending on your platform of course). In almost every frontend project we have worked on we have automatically included one Open Source JavaScript library: jQuery.

In a recent project, one of our goals was to speed up page load & rendering time on the product listing and detail pages (The product detail page used to take 15 seconds to render). One way to do this is to dynamically load in data/DOM only when it is needed. Doing so either required the backend to learn how to render data for each section separately, or to use frontend templates. With the particular backend system we are using, making it learn how to render each section individually would not be worth the effort, so we settled on jQuery Templates_[1]. Even though it was in beta it was being officially backed by the jQuery team and Microsoft. Though we understood that beta software will have bugs, it was an acceptable risk.

The first bug we experienced was with templates that called other templates. The code was designed to not allow more then 1 level of nesting on templates (template1 -> template2 !-> template3) as they automatically assumed the second level was a function call. This was a relatively easy bug to fix: just check before you call that it is indeed a function and send it though the template parser again if it is not. And like any good Open Source citizen, we filed a bug report and supplied a patch_[2].

The second bug we experienced was with their parser and the way we had done some function calls. Several of the functions we wanted to call took an object as a parameter, and the contents of this object were being built with data being passed into the template:

    <script type="jQuery/template" id="template1">
        {{if $data.variants }}
            {{each $data.pnos_m_et }}
                {{tmpl({array: $data.pnos_m_et, value: $value, obj: $data.variants[$value]})
                                brand.cft.templates.c48.tile_view.block }}
            {{/each}}
        {{else}}
            {{tmpl({array: [], value: null, obj: $data}) brand.cft.templates.c48.tile_view.block }}
        {{/if}}
    </script>

We initially tried to allow nested objects {array:[], value:$value, data: {name:$data.name, other:$value.something,…}}, but this broke their parser, because it was based on paired {{ && }}. So we accepted this and reworked things a little to ensure we never needed to pass nested data structures.

This worked for a while, but then we got to the point where the template code itself would need to call some form of initialization function when it was injected into the page, and we did not want the code that was injecting it to require knowing what was being injected. So we tried adding code at the end of each template that looked like:

    <script type="jQuery/template" id="template1">
       …
         <div class="script">
            {{=
                jQuery.tmpl_queue.add(function() {
                    nikon.normalizeProductTileHeight({
                        "id": "{{data.id}}",
                        "region": "product-details"
                    })
                })
            }}
        </div>
    </script>

But again this broke their parser, as you could not have a line break between {{ && }}. So at this point we sat down and looked at their parser, and looked at other javascript template systems: JavaScript Micro-Template_[3], Pure_[4], and EJS_[5]. We did not want to go back and have to manually update ~1500 JavaScript templates, but we liked how JSMT & EJS were using <% && %> as their section identifiers. A quick test of that showed that the simplistic XSLT engine our backend uses would break if there were any < || > that were not part of a tag, so we settled on {% && %}.

We spoke privately with the jQuery Templates team and they did not like this, because supporting the type of code we were trying to write was outside their goals. So we just modified our local version to support this. To make this backwards compatible we added another regex that matched our new opening and closing tags. We also set it up so that it would do console.info calls for every template it parsed that was using the old system, so if we happen to be working in an area using old templates we could see that and update them as we go.  

So far we have only addressed known perils, bugs, and project divergence due to use-case differences. The major peril of Open Source Software is project abandonment. Two months after all the work was finished for us to be able to use jQuery Templates the way we needed to, the jQuery Templates team decided to abandon the project in favor of including templates in jQueryUI at some future date. So for the time being there is no official template support for jQuery, and we are left supporting code for a client that we expected to have external support and bug fixes for.

Delivery Best Practice: Single Comment Repository

This is Part 1 in a series on Kanban’s delivery best practices.

Much of a project manager’s job requires a wrangling of tiny details. The more complexity there is in the design, the more details will come out over the entire course of the project. Every touch point adds to our understanding of how the client expects their functionality to work.

Some project managers keep a record of everything that occurs in every meeting. This serves a CYA purpose, but it can create too much documentation to take action on. A comment repository only tracks decisions and remarks about the functionality of what is being built. We use it for creating and updating our functional specs and for refining our QA test scripts.

At Kanban, the PM adds to the repository after every client presentation: wireframes, concepts, comps, functional specs, and sneak peeks. The client point-of-contact is then asked to revise and clarify what the PM captured. They add consolidated comments from other stakeholders.

We always meet together with the POC and our design partners after receiving the consolidated comments. Everyone gets on the same page. Prior to the next presentation, the PM reviews the new deliverables to confirm that every comment was addressed as agreed. Nothing gets lost.

Because the client is involved in clarifying the comments, and because they are tracked the same way in the same place across the entire project, the client knows and remembers the details of what they’re getting. Maintaining a single repository of comments is vital to creating a final deliverable that makes the client say, “This works exactly how I always thought it would.”

POV: Six Principles of Software Development

Kanban development process ensures that the enterprise software it develops is robust, reliable, and bug free. The following six key principles of Kanban’s development process are core to our engineering and software architecture philosophy.

Design Patterns and Object-Oriented Design

Kanban emphasizes the importance of solid object-oriented design though the implementation of design patterns. Object-oriented design helps architects to map a software system’s required behavior to structures or objects within that system and to define how those objects will communicate. This promotes greater flexibility and maintainability of the software. Software architects familiar with design patterns can easily identify recurring problems and their corresponding solutions. Object-oriented design and programming enables the reuse of those solutions and avoids the duplication of effort by system architects and developers.

Code Reuse

Code reuse saves time and effort. Achieving a high level of reuse is perhaps the most difficult goal to accomplish in developing a software system. The reuse of code and designs is considered a major benefit of using object-oriented technologies. To leverage the reuse possibilities that object-oriented programming provides requires forethought and planning. Kanban has developed extensive code libraries that can be leveraged as a solution for future projects. Leveraging existing code both increases quality and decreases cost. Communication is fundamental to code reuse and is the responsibility of the software architects.

Avoiding Unnecessary Complexity

Doing "the simplest thing that could possibly work" is central to Extreme Programming (XP), and it is a concept that Kanban holds in high regard. Due to the range of components available in software development, it is tempting for some to over-engineer solutions, accepting greater complexity for capabilities irrelevant to the business requirements. Complexity adds to cost throughout the software lifecycle. We take the time to thoroughly analyze the requirements so as not to over-simplify or over-complicate the solution.

Performance and Scalability

It is of the utmost importance for an enterprise application to meet performance and load expectations and exhibit sufficient scalability (the potential for an application to support increased load, given appropriate hardware). Scalability is an important consideration during the architecture phase of software applications. Kanban ensures that its applications are designed to operate efficiently in a clustered environment. Kanban’s solid understanding of J2EE and .NET infrastructure is essential to meeting both performance and scalability requirements.

Maintenance and Extensibility

Maintenance is the most expensive phase of the software development lifecycle. Because of this, Kanban pays particular attention to maintainability when designing an enterprise software solution. A custom software application is likely to be a key part of an organization's software mix for years, and must be able to accommodate new business needs without the need for a redesign. Maintainability and extensibility depend largely on a well-planned and well-executed architecture. Kanban engineers design solutions to ensure that each component of the application has a clear responsibility, and that maintenance is not hindered by tightly-coupled components.

Testing and Quality Assurance

At Kanban, testing is not an afterthought; it is an integral part of the software development cycle. Testing is done proactively; test cases are created based on requirements and iteratively updated during the development cycle. The Kanban QA team is not solely responsible for the quality of the software product; unit tests are created by our developers, often even before the code is written. Unit tests are written from a developer's perspective. Unit tests ensure that a particular method of a class successfully performs a set of specific tasks. Each unit test confirms that a method produces the expected output when given a known input. Kanban combines functional tests and unit tests to ensure a comprehensive process that greatly improves the quality of the software product.