help  namespaces  categories  help  rss feeds

TestBox

<< Back to Dashboard

Contents

TestBox : BDD & Testing Framework

Covers up to version 2.1.0

logo

Introduction

TestBox is a next generation testing framework for ColdFusion (CFML) that is based on BDD (Behavior Driven Development) for providing a clean obvious syntax for writing tests. It contains not only a testing framework, runner, assertions and expectations library but also integrates with MockBox for mocking and stubbing. It also supports xUnit style of testing and MXUnit compatibilities.

Features At A Glance

Here are a simple listing of features TestBox brings to the table:

  • BDD style testing
  • xUnit style testing
  • Testing life-cycle methods
  • MockBox integration for mocking and stubbing
  • Ability to extend and create custom test runners
  • Ability to extend and create custom test reporters
  • Extensible reporters, bundled with tons of them:
    • JSON
    • XML
    • JUnit 4 XML
    • Text (80's style)
    • Console
    • TAP (Test Anything Protocol)
    • Simple HTML
    • Min - Minimalistic Heaven
    • Raw
    • CommandBox
  • Asynchronous testing
  • Multi-suite capabilities
  • Test skipping
  • Suite skipping
  • Dynamic skipping support via runtime executions
  • Test one or more suites exclusively
  • Test one or more tests/specs exclusively
  • Test labels and tagging
  • Testing debug output stream
  • Clickable suite titles to filter test execution
  • Much more!

System Requirements

If you are going to be using the xUnit style of testing which includes MXUnit compatibility, then you don't need the latest CFML engines. BDD uses closures to declare your specs, so it it is only available on newer versions of your CFML engine.

xUnit and MXUnit Compat

  • ColdFusion 9+
  • Railo 3+

BDD Style (Requires closures)

  • ColdFusion 10+
  • Railo 4.1+

Installing TestBox

TestBox can be downloaded from http://www.coldbox.org/download or can be installed via CommandBox CLI (http://www.ortussolutions.com/products/commandbox):

// latest stable version
box install testbox

// latest bleeding edge
box install testbox-be

This will install TestBox in a /testbox folder from where you called the command. You can also extract the download zip and place it anywhere you like and create a mapping called /testbox that points to testbox in the distribution folder, this is the most secure approach. However, you can just place it in the webroot if you like.

this.mappings[ "/testbox" ] = expandPath( "C:/frameworks/testbox/" );

You can also clone from Github:

git clone git://github.com/ortus-solutions/testbox testbox

What's Included

The download structure includes:

  • apidocs : The API docs for TestBox
  • tests : Several sample tests and runners which actually are used to build TestBox
  • system : The main system framework folder
  • test-browser : This is a little utility to facilitate navigating big testing suites. This helps navigate to the suites you want and execute them instead of typing all the time.
  • test-runner : The pre-built TestBox Global Runner. You can pass in a directory mapping or bundles and select labels and boom run the tests.
  • test-harness : A simplified version of the TestBox runner that can be placed anywhere in your application. It includes an ANT build that allows you to execute your tests and produce results via ANT and also JUnit compliant reports via the junitreport task.

The ONLY required folder is "system" and it does not need to be web-accessable. The other folders are all optional tools to help you that expect to be web-accessible. You can safely remove everything except the system folder if you want to build your own test browsers. If you are placing TestBox outside of your web root, the "/testbox" mapping needs to point to the parent folder containing the "system" directory as all TestBox models start with testbox.system.

Primers

We have created two primers to get you started quickly and left this document to be the boring and detailed documentation. We also have awesome PDF ref cards for each of them.

MXUnit Compatiblity

TestBox is fully complaint with MXUnit xUnit test cases (if not, let us know). In order to leverage it you will need to create or override the /mxunit mapping and make it point to the /testbox/system/compat folder. That's it, everything should continue to work as expected. Note, you will still need TestBox to be in the web root, or have a /testbox mapping created even when using the MXUnit compat runner.

this.mappings[ "/mxunit" ] = expandPath( "/testbox/system/compat" );

After this, all your test code remains the same but it will execute through TestBox's xUnit runners. You can even execute them via the normal URL you are used to. If there is something that is not compatible, please let us know and we will fix it.

Expected Exceptions

We also support in the compatibility mode the expected exception MXUnit annotation: mxunit:expectedException and the expectException() methods. The expectException() method is not part of the assertion library, but instead is inherited from BaseSpec.cfc. Please refer to MXUnit's documentation on the annotation and method for expected exceptions, but it is supported with one caveat. The expectException() method can produce unwanted results if you are running your test specs in TestBox asynchronous mode since it stores state at the component level. Only synchronous mode is supported if you are using the expectException() method. The annotation can be used in both modes.

Testing Styles

TestBox comes with two flavors of testing: BDD and xUnit. BDD stands for behavior driven development and is highly based on creating specifications and expectations of results in a readable DSL (Domain Specifc Language). xUnit style of testing is the more traditional TDD or test driven development approach where you are creating a test case CFC that matches the software under test and for each method in the SUT you create a test method in the test case CFC. We cover these two topics in our primers:

No matter what testing style you pick you, TestBox will execute the tests for you and then send yoru results to a reporter that can produce awesome testing reports for you.

Test Bundles

No matter what style you decide to use, you will still end up building a Testing Bundle CFC. This CFC will either contain BDD style suites and specs or xUnit style method tests. These components are simple components with no inheritance that can contain several different life-cycle callback methods. Internally we do the magic of making this CFC inherit all capabilities and methods from our BaseSpec class (testbox.system.BaseSpec).

component{
     // easy right?
}

Optional Inheritance

At runtime we provide the inheritance via mixins so you don't have to worry about it. However, if you want to declare the inheritance you can do so and this will give you the following benefits:

  • Some IDEs will be able to give you introspection for methods and properties
  • You will be able to use the HTML runner by executing directly the runRemote method on the CFC Bundle
  • Your tests will run faster
component extends="testbox.system.BaseSpec"{
}

Injected Variables

At runtime, TestBox will inject several public variables into your testing bundle CFCs to help you with your testing:

  • $mockbox : A reference to MockBox
  • $assert : A reference to our Assertions library
  • $utility : A utility CFC
  • $customMatchers : A collection of custom matchers registered
  • $exceptionAnnotation : The annotation used to discover expected exceptions, defaults to expectedException
  • $testID : A unique identifier for the test bundle
  • $debugBuffer : The debugging buffer stream

Injected Methods

At runtime, TestBox will also decorate your bundle CFC with tons of methods to help you in your testing adventures. Dependening on your testing style you will leverage some more than others. For the latest methods please visit the API Docs for more information.

// quick assertion methods
assert( expression, [message] )
fail(message)

// bdd methods
beforeEach( body )
afterEach( body )
describe( title, body, labels, asyncAll, skip )
xdescribe( title, body, labels, asyncAll )
it( title, body, labels, skip )
xit( title, body, labels )
expect( actual )

// extensions
addMatchers( matchers )
addAssertions( assertions )

// runners
runRemote( testSpecs, testSuites, deubg, reporter );

// utility methods
console( var, top )
debug( var, deepCopy, label, top )
clearDebugBuffer()
getDebugBuffer()
print( message )
printLn( message )

// mocking methods
makePublic( target, method, newName )
querySim( queryData )
getmockBox( generationPath )
createEmptyMock( className, object, callLogging )
createMock( className, object, clearMethods, callLogging )
prepareMock( object, callLogging )
createStub( callLogging, extends, implements )
getProperty( target, name, scope, defaultValue )

Life-Cycle Methods

Each style has its own life-cycle methods you can discover in their primers. They are callbacks that TestBox calls at specific points in time of testing. They are a great way to setup or teardown things before the entire testing suite or one test case. Here is a short synopsis for each style.

xUnit

  • beforeTests() - Executes once before all tests for the entire test bundle CFC
  • afterTests() - Executes once after all tests complete in the test bundle CFC
  • setup( currentMethod ) - Executes before every single test case and receives the name of the actual testing method
  • teardown( currentMethod ) - Executes after every single test case and receives the name of the actual testing method
component{
     function beforeTests(){}
     function afterTests(){}

     function setup( currentMethod ){}
     function teardown( currentMethod ){}
}

BDD

  • beforeAll() - Executes once before all specs for the entire test bundle CFC
  • afterAll() - Executes once after all specs complete in the test bundle CFC
  • run( testResults, TestBox ) - Executes once so it can capture all your describe and it blocks
  • beforeEach( currentSpec ) - Executes before every single spec in a single describe block and receives the currently executing spec.
  • afterEach( currentSpec ) - Executes after every single spec in a single describe block and receives the currently executing spec.
  • aroundEach( spec, suite ) - Executes around the executing spec so you can provide code surrouding the spec.
component{
     function beforeAll(){}
     function afterAll(){}

     function run( testResults, testBox ){
          describe("A Spec", function(){

               beforeEach( function( currentSpec ){
                    // before each spec in this suite
               });

               afterEach( function( currentSpec ){
                    // after each spec in this suite
               });

               aroundEach( function( spec, suite ){
                    // execute the spec
                    arguments.spec.body();
               });

               describe("A nested suite", function(){

                    beforeEach( function(){
                         // before each spec in this suite + my parent's beforeEach()
                    });

                    afterEach( function(){
                         // after each spec in this suite + my parent's afterEach()
                    });
          
                }); 
          
          });

          describe("A second spec", function(){

               beforeEach( function( currentSpec ){
                    // before each spec in this suite, separate from the two other ones
               });

               afterEach( function( currentSpec ){
                    // after each spec in this suite, separate from the two other ones
               });
          
          }); 
     }
}

The great flexibility of the BDD approach is that it allows you to nest describe blocks or create multiple describe blocks. Each describe block can have its own life-cycle methods as well. Not only that, if they are nested, TestBox will walk the tree and call each beforeEach() and afterEach() in the order you declare them.

Suites, Tests & Specs (Oh My!)

Test suites are a collection of testing methods or specifications (specs) inside of a testing bundle CFC. Here is a synopsis for each style.

xUnit

The testing bundle CFC is actually the suite in xUnit style as it contains all the test methods you would like to test with. Usually, this CFC represents a test case for a specific software under test (SUT), whether that's a model object, service, etc. This component can have some cool annotations as well that can alter its behavior.

component displayName="The name of my suite" asyncAll="boolean" labels="list" skip="boolean"{
  
}

Bundle Annotations

Argument Required Default Type Description
displayName false --- string If used, this will be the name of the test suite in the reporters.
asyncAll false false boolean If true, it will execute all the test methods in parallel and join at the end asynchronously.
labels false --- string/list The list of labels this test belongs to
skip false false boolean/udf A boolean flag that makes the runners skip the test for execution. It can also be the name of a UDF in the same CFC that will be executed and MUST return a boolean value.

Important: If you activate the asyncAll flag for asynchronous testing, you HAVE to make sure your tests are also thread safe and appropriately locked.

Tests

TestBox discovers test methods in your bundle CFC by applying the following discovery rules:

  • Any method that has a test annotation on it
  • Any public method that starts or ends with the word test

// Via inline annotation
function shouldBeAwesome() test{}

/**
* Via comment annotation
* @test
*/
function shouldBeAwesome(){}

// via conventions
function testShouldDoThis(){}
function shouldDoThisTest(){}

Each test method will test the state of the SUT (software under test) or sometimes referred to as code under test. It will do so by asserting that actual values from an execution match an expected value or condition. TestBox offers an assertion library that you have available in your bundle via the injected variable $assert. You can also use our expectations library if you so desire, but that is mostly used in our BDD approach.

function testIncludes(){
       $assert.includes( "hello", "HE" );
       $assert.includes( [ "Monday", "Tuesday" ] , "monday" );
}

Each test function can also have some cool annotations attached to it.

Argument Required Default Type Description
labels false --- string/list The list of labels this test belongs to
skip false false boolean/udf A boolean flag that makes the runners skip the test for execution. It can also be the name of a UDF in the same CFC that will be executed and MUST return a boolean value.

An expectation or an assertion is a nice DSL that TestBox exposes so you can pretty much read what should happen in the testing scenario. A test will pass if all assertions pass. A spec with one or more assertions that fail will fail the entire test.

BDD

Suites

A test suite begins with a call to our TestBox describe() function with at least two arguments: a title and a closure within the life-cycle method called run(). The title is the name of the suite to register and the function is the block of code that implements the suite. There are more arguments which you can see below:

Argument Required Default Type Description
title true --- string The title of the suite to register
body true --- closure/udf The closure that represents the test suite
labels false --- string/array The list or array of labels this suite group belongs to
asyncAll false false Boolean If you want to parallelize the execution of the defined specs in this suite group.
skip false false Boolean A flag or a closure that tells TestBox to skip this suite group from testing if true. If this is a closure it must return boolean.
function run( testResults, testBox ){

     describe("A suite", function(){
          it("contains spec with an awesome expectation", function(){
               expect( true ).toBeTrue();
          });
     });

}

In BDD, suites can be nested within each other which provides a great capability of building trees of tests. Not only does it arrange them in tree format but also TestBox will execute the life-cycle methods in order of nested suites as it traverses the tree.


describe("A spec", function() {
     
     beforeEach(function() {
          coldbox = 22;
          application.wirebox = new coldbox.system.ioc.Injector();
     });

     afterEach(function() {
          coldbox = 0;
          structDelete( application, "wirebox" );
     });

     it("is just a function, so it can contain any code", function() {
          expect( coldbox ).toBe( 22 );
     });

     it("can have more than one expectation and talk to scopes", function() {
          expect( coldbox ).toBe( 22 );
          expect( application.wirebox.getInstance( 'MyService' ) ).toBeComponent();
     });

     describe("nested inside a second describe", function() {
          
          beforeEach(function() {
               awesome = 22;
          });

          afterEach(function() {
               awesome = 22 + 8;
          });

          it("can reference both scopes as needed ", function() {
            expect( coldbox ).toBe( awesome );
          });
     });

     it("can be declared after nested suites and have access to nested variables", function() {
          expect( awesome ).toBe( 30 );
     });

});

Specs

Specs are defined by calling the TestBox it() global function, which takes in a title and a function. The title is the title of this spec or test you will write and the function is a block of code that represents the test/spec. A spec will contain most likely one or more expectations that will test the state of the SUT (software under test) or sometimes referred to as code under test.

Argument Required Default Type Description
title true --- string The title of the spec
body true --- closure/udf The closure that represents the spec
labels false --- string/array The list or array of labels this suite group belongs to
skip false false Boolean A flag or a closure that tells TestBox to skip this suite group from testing if true. If this is a closure it must return boolean.

An expectation is a nice assertion DSL that TestBox exposes so you can pretty much read what should happen in the testing scenario. A spec will pass if all expectations pass. A spec with one or more expectations that fail will fail the entire spec.

function run( testResults, testBox ){

     describe("A suite", function(){
          it("contains spec with an awesome expectation", function(){
               expect( true ).toBeTrue();
          });

          it("contains a spec with more than 1 expectation", function(){
               expect( [1,2,3] ).toBeArray();
               expect( [1,2,3] ).toHaveLength( 3 );
          });
     });

}

Assertions

TestBox supports the concept of assertions to allow for validations and for legacy tests. We encourage developers to use our BDD expectations as they are more readable and fun to use (Yes, fun I said!). The assertions are modeled in the class testbox.system.Assertion, so you can visit the API for the latest assertions available. Each test bundle will receive a variable called $assert which represents the assertions object. Here are some common assertion methods:

fail( [message] )
assert( expression, [message] )
isTrue( actual, [message] )
isFalse( actual, [message] )
isEqual( actual, expected, [message] )
isNotEqual( actual, expected, [message] )
isEqualWithCase( actual, expected, [message] )
null( actual, [message] )
notNull( actual, [message] )
typeOf( type, actual, [message] )
notTypeOf( type, actual, [message] )
instanceOf( actual, typeName, [message] )
notInstanceOf( actual, typeName, [message] )
match( actual, regex, [message] )
matchWithCase( actual, regex, [message] )
notMatch( actual, regex, [message] )
key( target, key, [message] )
notKey( target, key, [message] )
deepKey( target, key, [message] )
notDeepKey( target, key, [message] )
lengthOf( target, length, [message] )
notLengthOf( target, length, [message] )
isEmpty( target, [message] )
isNotEmpty( target, [message] )
throws(target, [type], [regex], [message])
notThrows(target, [type], [regex], [message])
closeTo( actual, expected, delta, [datePart], [message] )
between( actual, min, max, [message] )
includes( target, needle, [message] )
notIncludes( target, needle, [message] )
includesWithCase( target, needle, [message] )
notIncludesWithCase( target, needle, [message] )
isGT( target, [message])
isGTE( target, [message])
isLT( target, [message])
isLTE( target, [message])

Custom Assertions

TestBox comes with a plethora of assertions that cover what we believe are common scenarios. However, we recommend that you create custom assertions that meet your needs and criteria so that you can avoid duplication and have reusability. A custom assertion function can receive any amount of arguments but it must use the fail() method in order to fail an assertion or just return true or void for passing.

Here is an example:

function isAwesome( required expected ){
     return ( arguments.expected == "TestBox" ? fail( 'TestBox is always awesome' ) : true );
}

Assertion Registration

You can register assertion functions in several ways within TestBox, but we always recommend that you register them inside of the beforeTests() or setup() life-cycle method blocks, so they are only inserted once.

Inline Assertions You can pass a structure of key/value pairs of the assertions you would like to register. The key is the name of the assertion function and the value is the closure function representation.

function beforeTests(){

     addAssertions({
          isAwesome = function( required expected ){
               return ( arguments.expected == "TestBox" ? true : fail( 'not TestBox' ) );
          },
          isNotAwesome = function( required expected ){
               return ( arguments.expected == "TestBox" ? fail( 'TestBox is always awesome' ) : true );
          }
     });

}

After it is registered, then you can just use it out of the $assert object it got mixed into.

function testAwesomenewss(){
  $assert.isAwesome( 'TestBox' );
}

CFC Assertions You can also store a plethora of assertions (Yes, I said plethora), in a CFC and register that as the assertions via its instantiation path. This provides much more flexibility and reusability for your projects.

addAssertions( "model.util.MyAssertions" );

You can also register more than 1 CFC by using a list or an array:

addAssertions( "model.util.MyAssertions, model.util.RegexAssertions" );
addAssertions( [ "model.util.MyAssertions" , "model.util.RegexAssertions" ] );

Here is the custom assertions CFC source:

component{

     function assertIsAwesome( expected, actual ){
          return ( expected eq actual ? true : false );
     }

     function assertIsFunky( actual ){
          return ( actual gte 100 ? true : false );
     }

}

Expectations

TestBox allows you to create BDD expectations with our expectations and matcher API DSL. You start by calling our expect() method, usually with an actual value you would like to test. You then concatenate the expectation of that actual value/function to a result or what we call a matcher. Please visit our BDD Primer for an in-depth view of BDD expectations. You can also concatenate matchers (as of v2.1.0) so you can provide multiple matching expectations to a single value.

expect( 43 ).toBe( 42 );
expect( function(){ calculator.add(2,2); } ).toThrow();

Matchers

The toBe() matcher represents an equality matcher much how an $assert.isEqual() behaves. Below are many other matchers available to you.

toBeTrue() :  value to true
toBeFalse() : value to be false
toBe( expected ) : Assert something is equal to each other, no case is required
toBeWithCase( expected ) : Expects with case
toBeNull() : Expects the value to be null
toBeInstanceOf( class ) : To be the class instance passed
toMatch( regex ) : Matches a string with case-sensitivity
toMatchNoCase( regex ) : Matches with no case-sensitivity
toBeTypeOf( type, message ) : Assert the type of the incoming actual data, it uses the internal ColdFusion isValid() function behind the scenes, type can be array, binary, boolean, component, date, time, float, numeric, integer, query, string, struct, url, uuid plus all the ones from isValid()
toBe{type}( [message] ) : Same as above but more readable method name. Example: .toBeStruct(), .toBeArray()
toBeEmpty() : Tests if an array or struct or string or query is empty
toHaveKey( key ) : Tests the existence of one key in a structure or hash map
toHaveDeepKey( key ) : Assert that a given key exists in the passed in struct by searching the entire nested structure
toHaveLength( length ) : Assert the size of a given string, array, structure or query
toThrow( [type], [message] );
toBeCloseTo( expected, delta, [datepart] ) : Can be used to approximate numbers or dates according to the expected and delta arguments.  For date ranges use the datepart values.
toBeBetween( min, max ) : Assert that the passed in actual number or date is between the passed in min and max values
toInclude( needle ) : Assert that the given "needle" argument exists in the incoming string or array with no case-sensitivity, needle in a haystack anyone?
toIncludeWithCase( needle ) : Assert that the given "needle" argument exists in the incoming string or array with case-sensitivity, needle in a haystack anyone?
toBeGT( target ) : Assert that the actual value is greater than the target value
toBeGTE( target ) : Assert that the actual value is greater than or equal the target value
toBeLT( target ) : Assert that the actual value is less than the target value
toBeLTE( target ) : Assert that the actual value is less than or equal the target value

Not operator

You can prefix your expectation with the not operator to easily cause negative expectations for any matcher. When you read the API Docs or the source, you will find nowhere the not methods. This is because we do this dynamically by convention.

expect( actual )
     .notToBe( 4 )
     .notToBeTrue();
     .notToBeFalse();

Custom Matchers

TestBox comes with a decent amount of matchers that cover what we believe are common scenarios. However, we recommend that you create custom matchers that meet your needs and criteria so that you can avoid duplication and have reusability. Every custom matcher must have the following signature, with MyMatcher being the name of your custom matcher function:

boolean function MyMatcher( required expectation, args={} )

The matcher function receives the expectation object and a second argument which is a structure of all the arguments with which the matcher function was called with. It must then return a true or a false depending if it passes your criteria. It will most likely use the expectation object to retrieve the actual and isNot values. It can also set a custom failure message on the expectation object itself by using the message property.

boolean function reallyFalse( expectation, args={} ){
     expectation.message = ( structKeyExists( args, "message" ) ? args.message : "[#expectation.actual#] is not really false" );
     if( expectation.isNot )
          return ( expectation.actual eq true );
     else
          return ( expectation.actual eq false );
     }
}

The next step is to tell TestBox about your matcher.

Matcher Registration

You can register matcher functions in several ways within TestBox, but we always recommend that you register them inside of the beforeAll() or beforeEach() life-cycle method blocks for performance considerations and global availability.

Inline matchers

You can pass a structure of key/value pairs of the matchers you would like to register. The key is the name of the matcher function and the value is the closure function representation.

function beforeAll(){

  addMatchers( { 
       toBeAwesome : function( expectation, args={} ){ return expectation.actual gte 100; },
       toBeGreat : function( expectation, args={} ){ return expectation.actual gte 1000; },
       // please note I use positional values here, you can also use name-value arguements.
       toBeGreaterThan : function( expectation, args={} ){ return ( expectation.actual gt args[ 1 ]  ); }
  } );

}

After it is registered, then you can use it.

it("A custom matcher", function(){
  expect( 100 ).toBeAwesome();
  expect( 5000 ).toBeGreat();
  expect( 10 ).toBeGreaterThan( 5 );
});
CFC matchers

You can also store a plethora of matchers (Yes, I said plethora), in a CFC and register that as the matchers via its instantiation path. This provides much more flexibility and reusability for your projects.

addMatchers( "model.util.MyMatchers" );

You can also register more than 1 CFC by using a list or an array:

addMatchers( "model.util.MyMatchers, model.util.RegexMatchers" );
addMatchers( [ "model.util.MyMatchers" , "model.util.RegexMatchers" ] );

Expecting Exceptions

Our default syntax for expecting exceptions is to use our closure approach (Note: CF10+, Railo 4+) concatenated with our toThrow() method in our expectations or our throws() method in our assertions object.

Info: Please always remember to pass in a closure to these methods and not the actual test call: function(){ myObj.method(); }

expect( function(){ myObj.method(); } ).toThrow( [type], [regex], [message] );
$assert.throws( function(){ myObj.method; }, [type], [regex], [message] )

This will execute the closure in a nested try/catch block and make sure that it either threw an exception, threw with a type, threw with a type and a regex match of the exception message. If you are in an environment that does not support closures then you will need to create a spec testing function that either uses the expectedException annotation or function call:

function testMyObj(){
     expectedException( [type], [regex], [message] );
}

function testMyObj() expectedException="[type]:[regex]"{
     // this function should produce an exception
}

Important: Please note that the usage of the expectedException() method can ONLY be used while in synchronous mode. If you are running your tests in asynchronous mode, this will not work. We would recommend the closure or annotation approach instead.

Output Utilities

Sometimes you will need to produce output from your tests and you can do so elegantly via some functions we have provided that are available in your test bundles:

  • console(var, top=[9999]) - Send data to the console via writeDump( output="console" )
  • debug(var, label="", deepCopy=[false], top=999) - Send data to the TestBox debug stream, will be saved in the test results and outputted in reports
  • print( message ) - Write some output to the ColdFusion output buffer
  • println( message ) - Write some output to the ColdFusion output buffer and appends a

These are great little utilities that are needed to send output to several locations. Please note that the debug() method does NOT do deep copies by default.

console( myResults );

debug( myData );
debug( myData, true );

debug( var=myData, top=5 );

print( "Running This Test with #params.toString()#" );
println( "Running This Test with #params.toString()#" );

Spies & Mocking

Please refer to our MockBox guide to take advantage of all the mocking and stubbing you can do. However, every BDD TestBundle has the following functions available to you for mocking and stubbing purposes:

  • createStub( [callLogging=true], [extends], [implements]') - Create stub objects with call logging and optional inheritance trees and implementation methods
  • createEmptyMock( [className], [object], [callLogging=true]') - Create an empty mock from a class or object
  • createMock( [className], [object], [clearMethods=false], [callLogging=true]') - Create a spy from an instance or class with call logging
  • getMockBox( [generationPath] ') - Get a reference to MockBox
  • makePublic( target, method, newName ') - Exposes private methods from objects as public methods
  • querySim( queryData ') - Simulate a query
  • prepareMock( object, [callLogging=true]') - Prepare an instance of an object for method spies with call logging

Running Tests

TestBox ships with several test runners internally but we have tried to simplify and abstract it with our TestBox object which can be found in the testbox.system package. The TestBox object allows you to execute tests from a CFC, CFM, HTTP, SOAP or REST. You can also make your CFC's extend from our BaseSpec class so you can execute it directly via the URL. The main execution methods are:

// Run tests and produce reporter results
testbox.run()

// Run tests and get raw testbox.system.TestResults object
testbox.runRaw()

// Run tests and produce reporter results from SOAP, REST, HTTP
testbox.runRemote()

// Run via Spec URL
http://localhost/tests/spec.cfc?method=runRemote

We encourage you to read the API docs included in the distribution for the complete parameters for each method.

Here are the arguments you can use for initializing TestBox or executing the run() method

Argument Required Default Type Description
bundles true --- string/array The path, list of paths or array of paths of the spec bundle CFCs to run and test
directory false --- struct The directory mapping path or a struct: [ mapping = the path to the directory using dot notation (myapp.testing.specs), recurse = boolean, filter = closure that receives the path of the CFC found, it must return true to process or false to continue process ]
reporter false simple string/struct/instance The type of reporter to use for the results, by default is uses our 'simple' report. You can pass in a core reporter string type or an instance of a coldbox.system.reports.IReporter. You can also pass a struct with [type="string or classpath", options={}] if a reporter expects options.
labels false false string/array The string or array of labels the runner will use to execute suites and specs with.
options false {} struct A structure of property name-value pairs that each runner can implement and use at its discretion.
testBundles false --- string/array A list or array of bundle names that are the ones that will be executed ONLY!
testSuites false --- string/array A list or array of suite names that are the ones that will be executed ONLY!
testSpecs false --- string/array A list or array of test names that are the ones that will be executed ONLY

Here are the arguments you can use for executing the runRemote() method of the TestBox object:

Argument Required Default Type Description
bundles true --- string The path, list of paths or array of paths of the spec bundle CFCs to run and test
directory false --- string The directory mapping to test: directory = the path to the directory using dot notation (myapp.testing.specs)
recurse false true boolean Recurse the directory mapping or not, by default it does
reporter false simple string/path The type of reporter to use for the results, by default is uses our 'simple' report. You can pass in a core reporter string type or a class path to the reporter to use.
reporterOptions false {} JSON A JSON struct literal of options to pass into the reporter
labels false false string The string array of labels the runner will use to execute suites and specs with.
options false {} JSON A JSON struct literal of configuration options that are optionally used to configure a runner.
testBundles false --- string/array A list or array of bundle names that are the ones that will be executed ONLY!
testSuites false --- string A list of suite names that are the ones that will be executed ONLY!
testSpecs false --- string A list of test names that are the ones that will be executed ONLY

The bundles argument which can be a single CFC path or an array of CFC paths or a directory argument so it can go and discover the test bundles from that directory. The reporter argument can be a core reporter name like: json,xml,junit,raw,simple,dots,tap,min,etc or it can be an instance of a reporter CFC. You can execute the runners from any cfm template or any CFC or any URL, that is up to you.

Global Runner

TestBox ships with a global runner that can be used to run pretty much anything. You can customize it or place it wherever you need it:

Test Browser

TestBox ships with a test browser that is highly configurable to whatever URL accessible path you want. It will then show you a test browser where you can navigate and execute not only individual tests, but also directory suites as well.

Bundle(s) Runner

<cfset r = new coldbox.system.TestBox( "coldbox.testing.cases.testing.specs.BDDTest" ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( bundles="coldbox.testing.cases.testing.specs.BDDTest", testSpecs="OnlyThis,AndThis,AndThis" ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( bundles="coldbox.testing.cases.testing.specs.BDDTest", testSuites="Custom Matchers,A Spec" ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( [ "coldbox.testing.cases.testing.specs.BDDTest", "coldbox.testing.cases.testing.specs.BDD2Test" ] ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( bundles="coldbox.testing.cases.testing.specs.BDDTest", labels="railo" ) >
<cfoutput>#r.run(reporter="json")#</cfoutput>

Test Runner

If you make your test bundle CFC inherit from our testbox.system.BaseSpec class, you will be able to execute the CFC directly via the URL:

http://localhost/tests/MyTest.cfc?method=runRemote

You can also pass the following arguments to the method via the URL:

  • testSuites : A list or array of suite names that are the ones that will be executed ONLY!
  • testSpecs : A list or array of test names that are the ones that will be executed ONLY!
  • reporter : The type of reporter to run the test with
http://localhost/tests/MyTest.cfc?method=runRemote&reporter=json

Directory Runner

<cfset r = new coldbox.system.TestBox( directory="coldbox.testing.cases.testing.specs" ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( directory={ mapping="coldbox.testing.cases.testing.specs", recurse=false } ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( 
      directory={ mapping="coldbox.testing.cases.testing.specs", 
      recurse=true, 
      filter=function(path){
            return ( findNoCase( "test", arguments.path ) ? true : false );
      }}) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( 
      directory={ mapping="coldbox.testing.cases.testing.specs", 
      recurse=true, 
      filter=function(path){
            return ( findNoCase( "test", arguments.path ) ? true : false );
      }}) >
<cfset fileWrite( 'testreports.json', r.run() )>

SOAP Runner

You can run tests via SOAP by leveraging the runRemote() method. The WSDL URL will be

http://localhost/testbox/system/TestBox.cfc?wsdl

HTTP/REST Runner

You can run tests via HTTP/REST by leveraging the runRemote() endpoint. The URL will be

http://localhost/testbox/system/TestBox.cfc?method=runRemote

ANT Runner

In our test samples and templates we include an ANT runner that will be able to execute your tests via ANT. It can also leverage our ANTJunit reporter to use the junitreport task to produce JUnit compliant reports as well. You can find this runner in the test samples and runner template directory.

Runner

This is the runner used for any reporter and ANT Junit style reporting

<cfsetting showDebugOutput="false">
<cfparam name="url.reporter"       default="simple">
<cfparam name="url.directory"      default="test.specs">
<cfparam name="url.recurse"        default="true" type="boolean">
<cfparam name="url.bundles"        default="">
<cfparam name="url.labels"         default="">
<cfparam name="url.reportpath"     default="#expandPath( "/tests/results" )#">
<cfscript>
// prepare for tests for bundles or directories
if( len( url.bundles ) ){
     testbox = new testbox.system.TestBox( bundles=url.bundles, labels=url.labels );
}
else{
     testbox = new testbox.system.TestBox( directory={ mapping=url.directory, recurse=url.recurse}, labels=url.labels );
}
// Run Tests using correct reporter
results = testbox.run( reporter=url.reporter );
// do stupid JUnitReport task processing, if the report is ANTJunit
if( url.reporter eq "ANTJunit" ){
     xmlReport = xmlParse( results );
     for( thisSuite in xmlReport.testsuites.XMLChildren ){
          fileWrite( url.reportpath & "/TEST-" & thisSuite.XMLAttributes.name & ".xml", toString( thisSuite ) );
     }
}
// Writeout Results
writeoutput( results );
</cfscript>

ANT

This build is our global build for running TestBox via ANT.

<?xml version="1.0"?>
<--<
This ANT build can be used to execute your tests with automation using our included runner.cfm.  
You can executes directories, bundles and so much more.  It can also produce JUnit reports using the
ANT junitreport tag.  This is meant to be a template for you to spice up.

There are two targets you can use: run and run-junit

Execute the default 'run' target
ant -f test.xml
OR
ant -f test.xml run

Execute the 'run-junit' target
ant -f test.xml run-junit

PLEASE NOTE THAT YOU MUST ALTER THE RUNNER'S URL ACCORDING TO YOUR ENVIRONMENT.
-->
<project name="testbox-ant-runner" default="run" basedir=".">
     
     <-- <THE URL TO THE RUNNER, PLEASE CHANGE ACCORDINGLY-->
     <property name="url.runner"             value="http://cf10cboxdev.jfetmac/coldbox/ApplicationTemplates/Advanced/tests/runner.cfm?"/>
     <-- <FILL OUT THE BUNDLES TO TEST, CAN BE A LIST OF CFC PATHS -->
     <property name="test.bundles"      value="" />
     <-- <FILL OUT THE DIRECTORY MAPPING TO TEST -->
     <property name="test.directory"         value="test.specs" />
     <-- <FILL OUT IF YOU WANT THE DIRECTORY RUNNER TO RECURSE OR NOT -->
     <property name="test.recurse"      value="true" />
     <-- <FILL OUT THE LABELS YOU WANT TO APPLY TO THE TESTS -->
     <property name="test.labels"       value="" />
     <-- <FILL OUT THE TEST REPORTER YOU WANT, AVAILABLE REPORTERS ARE: ANTJunit, Codexwiki, console, dot, doc, json, junit, min, raw, simple, tap, text, xml -->
     <property name="test.reporter"          value="simple" />
     <-- <FILL OUT WHERE REPORTING RESULTS ARE STORED --> 
     <property name="report.dir"        value="${basedir}/results" />
     <property name="junitreport.dir"   value="${report.dir}/junitreport" />

     <target name="init" description="Init the tests">
          <mkdir dir="${junitreport.dir}" />
          <tstamp prefix="start">
               <format property="TODAY" pattern="MM-dd-YYYY hh:mm:ss aa"/>
          </tstamp>
          <concat destfile="${report.dir}/latestrun.log">Tests ran at ${start.TODAY}</concat>
     </target>
     
     <target name="run" depends="init" description="Run our tests and produce awesome results">  

          <-- <Directory Runner 
               Executes recursively all tests in the passed directory and stores the results in the 
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter
                - Directory
                - Recurse
                - Labels
          -->
          <get dest="${report.dir}/results.html" 
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=${test.reporter}&labels=${test.labels}" 
                verbose="true"/>
          
          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html" 
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}" 
                verbose="true"/>
           -->
          
     </target>

     <target name="run-junit" depends="init" description="Run our tests and produce ANT JUnit reports">  

          <-- <Directory Runner 
               Executes recursively all tests in the passed directory and stores the results in the 
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter = ANTJunit fixed
                - Directory
                - Recurse
                - Labels
                - ReportPath : The path where reports will be stored, by default it is the ${report.dir} directory.
          -->
          <get dest="${report.dir}/results.xml" 
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=ANTJunit&labels=${test.labels}&reportPath=${report.dir}" 
                verbose="true"/>
          
          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html" 
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}" 
                verbose="true"/>
           -->

           <-- <Create fancy junit reports -->
          <junitreport todir="${junitreport.dir}">  
               <fileset dir="${report.dir}">  
                    <include name="TEST-*.xml"/>  
               </fileset>
               <report format="frames" todir="${junitreport.dir}">
                    <param name="TITLE" expression="My Awesome TestBox Results"/>
               </report>
          </junitreport> 
          
     </target>

</project>

Reporters

TestBox comes also with a nice plethora of reporters:

  • ANTJunit : A specific variant of JUnit XML that works with the ANT junitreport task
  • Codexwiki : Produces MediaWiki syntax for usage in Codex Wiki
  • Console : Sends report to console
  • Doc : Builds semantic HTML to produce nice documentation
  • Dot : Builds an awesome dot report
  • JSON : Builds a report into JSON
  • JUnit : Builds a JUnit compliant report
  • Raw : Returns the raw structure representation of the testing results
  • Simple : A basic HTML reporter
  • Text : Back to the 80's with an awesome text report
  • XML : Builds yet another XML testing report
  • Tap : A test anything protocol reporter
  • Min : A minimalistic view of your test reports

However, you can also build custom reporters very easily.

Custom Reporters

You can also build your own reporters by implementing our core class: coldbox.system.reporters.IReport

interface{

  /**
  * Get the name of the reporter
  */
  function getName();

  /**
  * Do the reporting thing here using the incoming test results
  * The report should return back in whatever format they desire and should set any
  * Specifc browser types if needed.
  * @results.hint The instance of the TestBox TestResult object to build a report on
  * @testbox.hint The TestBox core object
  * @options.hint A structure of options this reporter needs to build the report with
  */
  any function runReport( 
    required coldbox.system.TestResult results,
    required coldbox.system.TestBox testbox,
    struct options={} );
}

Once you implement your own report you just need to pass the class path or the instance of your reporter to the TestBox runner methods using the reporter argument.

r = new TestBox( reporter="my.path.toCustomReporter" );
r = new TestBox( reporter= new my.path.CustomReporter() );

Here is a sample reporter for you:

component{

  function init(){ return this; }

  /**
  * Get the name of the reporter
  */
  function getName(){
    return "JSON";
  }

  /**
  * Do the reporting thing here using the incoming test results
  * The report should return back in whatever format they desire and should set any
  * Specifc browser types if needed.
  * @results.hint The instance of the TestBox TestResult object to build a report on
  * @testbox.hint The TestBox core object
  * @options.hint A structure of options this reporter needs to build the report with
  */
  any function runReport( 
    required coldbox.system.TestResult results,
    required coldbox.system.TestBox testbox,
    struct options={}
  ){
    getPageContext().getResponse().setContentType( "application/json" );
    return serializeJSON( arguments.results.getMemento() );
  }
  
}

 
Download in other Formats:
markup Markup | html HTML | word Word

comments Comments (2)

kev@inner-rhythm.co.uk's Gravatar

url Big Mad Kev said

at 12:05:24 PM 23-Sep-2014

Expecting Exceptions it should be expectedException for the annotation and function not expectException
lmajano@gmail.com's Gravatar

Luis Majano said

at 01:46:47 PM 23-Sep-2014

Done, thanks

ColdBox Books