TFSBuild 2010 & NUnit Integration

Running and publishing NUnit test results into TFS 2010

Wednesday 04 January 2012 at 12:30 pm. Used tags: , , , , ,

Always keen that our adopted environment shouldn't dictate tooling it was important that we could achieve a good level integration between with testing tools and Team Foundation Server; however as TFS 2010 moved to using web services and XAML build workflow it can look rather challenging.

Interesting note

The snippets below are based around NUnit testing tool but it could be employed just as easily for others that are able to output their results XML in the same format, xUnit being a fine example.

Looking at integration we identified two essential requirements;

  • NUnit-Console test runner executes as part of the build workflow
  • Test results are published back into TFS 2010 and statistics

Runner

Executing nunit-console.exe using "InvokeProcess" is pretty straight forward, you will find many examples on the Internet although I used Peter Gafder's as my base to work from. Here is a slightly modified version:

  <mtbwa:InvokeProcess Arguments="[String.Format("""{0}"" /xml:""{1}"" /nologo /nodots", item, testResultXmlPath)]" DisplayName="Execute NUnit" FileName="C:\Program Files (x86)\NUnit 2.5.10\bin\net-2.0\nunit-console.exe" sad:VirtualizedContainerService.HintSize="256,403">
    <mtbwa:InvokeProcess.ErrorDataReceived>
      <ActivityAction x:TypeArguments="x:String">
        <ActivityAction.Argument>
          <DelegateInArgument x:TypeArguments="x:String" Name="errOutput" />
        </ActivityAction.Argument>
        <mtbwa:WriteBuildError sad:VirtualizedContainerService.HintSize="222,22" Message="[errOutput]" />
      </ActivityAction>
    </mtbwa:InvokeProcess.ErrorDataReceived>
    <mtbwa:InvokeProcess.OutputDataReceived>
      <ActivityAction x:TypeArguments="x:String">
        <ActivityAction.Argument>
          <DelegateInArgument x:TypeArguments="x:String" Name="stdOutput" />
        </ActivityAction.Argument>
        <Sequence sad:VirtualizedContainerService.HintSize="222,235">
          <sad:WorkflowViewStateService.ViewState>
            <scg:Dictionary x:TypeArguments="x:String, x:Object">
              <x:Boolean x:Key="IsExpanded">True</x:Boolean>
              <x:Boolean x:Key="IsPinned">True</x:Boolean>
            </scg:Dictionary>
          </sad:WorkflowViewStateService.ViewState>
          <mtbwa:WriteBuildMessage sad:VirtualizedContainerService.HintSize="200,22" Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="[	  stdOutput]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />

          <!-- if test output contains errors or failures -->

          <If Condition="[(Not stdOutput.Contains("Failures: 0") And stdOutput.Contains("Failures:")) Or
(Not stdOutput.Contains("Errors: 0") And stdOutput.Contains("Errors:"))]" DisplayName="If there were failed tests" sad:VirtualizedContainerService.HintSize="464,203">
            <If.Then>
              <Sequence sad:VirtualizedContainerService.HintSize="264,270">
                <sad:WorkflowViewStateService.ViewState>
                  <scg:Dictionary x:TypeArguments="x:String, x:Object">
                    <x:Boolean x:Key="IsExpanded">True</x:Boolean>
                  </scg:Dictionary>
                </sad:WorkflowViewStateService.ViewState>

                <!-- put build test status in failed state -->

                <Assign sad:VirtualizedContainerService.HintSize="242,57">
                  <Assign.To>
                    <OutArgument x:TypeArguments="mtbc:BuildPhaseStatus">[BuildDetail.TestStatus]</OutArgument>
                  </Assign.To>
                  <Assign.Value>
                    <InArgument x:TypeArguments="mtbc:BuildPhaseStatus">[Microsoft.TeamFoundation.Build.Client.BuildPhaseStatus.Failed]</InArgument>
                  </Assign.Value>
                </Assign>

                <!-- if treatTestFailureAsBuildFailure create build error, else create build warning -->

                <If Condition="[treatTestFailureAsBuildFailure]" sad:VirtualizedContainerService.HintSize="242,49">
                  <If.Then>
                    <mtbwa:WriteBuildError sad:VirtualizedContainerService.HintSize="219,100" Message="[assemblyName & " -> " & stdOutput]" />
                  </If.Then>
                  <If.Else>
                    <mtbwa:WriteBuildWarning sad:VirtualizedContainerService.HintSize="220,100" Message="[assemblyName & " -> " & stdOutput]" />
                  </If.Else>
                </If>
              </Sequence>
            </If.Then>
          </If>
        </Sequence>
      </ActivityAction>
    </mtbwa:InvokeProcess.OutputDataReceived>
  </mtbwa:InvokeProcess>

When executing the build workflow NUnit-Console.exe will now execute and the XML output placed into the build logs directory, the build status is also updated with a success/partial-success state. Our problem here is that not only do we not know how many were executed, how many passed/failed but importantly the associated stack trace of failing tests.

Publishing

TFS 2010 uses a proprietary results format (no suprise there then) with a ".trx" extension to publish results so we need to transpose the NUnit results before sending back through web services.

NUnitTfs

I'd been aware of NUnitTFS (NUnit Team Build) for some time but hadn't realised from the CodePlex welcome page their was a version for TFS 2010 that included publishing back through web-services; be sure to click on 'Downloads' and look for the latest release of version 2.0 (currently alpha). 

There are some gotcha's with NUnitTFS, namely that you ensure you have an active build configuration:

TFS Build Properties

.. and that if targetting a Team Project inside a collection, you update the URLs in "NUnitTfs.exe.config" to include the collection name;

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
	<system.serviceModel>

		<bindings>
			(removed for brevity)
		</bindings>

		<client>
			<!-- TFS 2010 services. -->
			<endpoint address="http://tfs.acme.co.uk:8080/tfs/MyCollection/TestManagement/v1.0/TestResults.asmx"
			 binding="basicHttpBinding" bindingConfiguration="TestResultsServiceSoap"
			 contract="Tfs2010.TestResultsServiceV1.TestResultsServiceSoap" name="TestResultsServiceSoap" />
			<endpoint address="http://tfs.acme.co.uk:8080/tfs/MyCollection/Services/v3.0/IdentityManagementService.asmx"
			 binding="basicHttpBinding" bindingConfiguration="IdentityManagementWebServiceSoap"
			 contract="Tfs2010.IdentityManagementServiceV3.IdentityManagementWebServiceSoap"
			 name="IdentityManagementWebServiceSoap" />
			<endpoint address="http://tfs.acme.co.uk:8080/tfs/MyCollection/Build/v3.0/BuildService.asmx"
			 binding="basicHttpBinding" bindingConfiguration="BuildWebServiceSoap"
			 contract="Tfs2010.BuildServiceV3.BuildWebServiceSoap" name="BuildWebServiceSoap" />
		</client>
	</system.serviceModel>
</configuration>

Workflow

Thanks to Karsten Strøbæk's blog post discussing using NUnitTFS for xUnit I had the final piece to the puzzle; all I needed to do now was to piece the two workflows together, add some configuration variables for paths, and a couple of minor workflow tweaks to ensure smooth operation.

Build Configuration

TFS Build Properties

XAML

<mtbwa:FindMatchingFiles DisplayName="Find Test Assemblies --> testAssemblies" sap:VirtualizedContainerService.HintSize="464,22" MatchPattern="[testResultXmlPath]" Result="[testResults]" />
<If Condition="[testResults.Count() > 0]" DisplayName="If Test Results Found">
<If.Then>
  <Sequence>
    <mtbwa:InvokeProcess Arguments="[String.Format("-n {0} -t {1} -p ""{2}"" -f {3} -b ""{4}"" -v 2010", testResultXmlPath, BuildDetail.TeamProject, BuildSettings.PlatformConfigurations(0).Platform, BuildSettings.PlatformConfigurations(0).Configuration, BuildDetail.BuildNumber)]" DisplayName="Publish NUnit results" FileName="[TFSPublish]" sap:VirtualizedContainerService.HintSize="234,198" WorkingDirectory="[outputDirectory]">
      <mtbwa:InvokeProcess.ErrorDataReceived>
        <ActivityAction x:TypeArguments="x:String">
          <ActivityAction.Argument>
            <DelegateInArgument x:TypeArguments="x:String" Name="errOutput" />
          </ActivityAction.Argument>
          <mtbwa:WriteBuildError DisplayName="Write NUnit Publish Failure" sap:VirtualizedContainerService.HintSize="200,22" Message="[errOutput]" />
        </ActivityAction>
      </mtbwa:InvokeProcess.ErrorDataReceived>
      <mtbwa:InvokeProcess.OutputDataReceived>
        <ActivityAction x:TypeArguments="x:String">
          <ActivityAction.Argument>
            <DelegateInArgument x:TypeArguments="x:String" Name="stdOutput" />
          </ActivityAction.Argument>
          <mtbwa:WriteBuildMessage DisplayName="Write NUnit results publish output" sap:VirtualizedContainerService.HintSize="200,22" Message="[stdOutput]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
        </ActivityAction>
      </mtbwa:InvokeProcess.OutputDataReceived>
    </mtbwa:InvokeProcess>
  </Sequence>
</If.Then>
</If>

Final solution

You can find the completed XAML build workflow in a Gist here; I don't pretend to be any kind of workflow expert so feedback and improvements very welcome :-)

No comments

Leave a comment

(optional field)
(optional field)
'cause I hate spambots, you gotta do this I'm afraid :\
Remember personal info?
Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.

My Stuff

Reading list

  • Josh Arnold - Web developer, author & contributor of FubuMVC
  • Jeremy D Miller - Web developer, author & contributor of StructureMap & FubuMVC
  • Roy Osherove - Unit Testing, Agile Development, Leadership & .NET
  • Rob Ashton - Technical Lead & author of MvcEx and AutoPoco

Feeds

Jon:

Wow! Can I work for you?! ** Breath of fresh air in management! **

Ian:

There will have to be very strict guidelines, the last thing you want is dispute over “.. but you gave him 5 credits and her only 3?!” which will undoubtedly lead to the scheme being rescinded. That re…

Gary Ewan Park:

This is a very interesting idea! It does leave some questions about how it is “policed”, and who decides whether a particular blog post, or SO answer warrants the credits, but I really do think that i…

Chris Marisic:

Being both a developer and a leader of developers, I have found little to be as integral to the speed and success of software development as intellisense. This is also why I leverage R# on top of Vis…

Franc:

I have to admit that Spark won me over very quickly, yes it is not perfect and yes I am not an expert in the subject but it has provided me with new skills. I have been a victim of its lack of intellis…

Randolph Burt:

Intellisense helps me code faster and make less mistakes. However, from a personal point of view, it can be the difference between enjoying development and not enjoying development – and if you don’t …