Get Started With Continuous Integration For Your .NET (C#) Projects - Integrate Sonar

This is the 3rd post of this series on Getting Started With Continuous Integration For Your .NET (C#) Projects.

In the 1st post, we investigated on how to build your .Net project in Jenkins using the MSBuild on every code checkin.

In the 2nd post, we investigated on how to integrate running unit test in the build using the command line tool - mstest.exe.

In this post, we will explore how to integrate Sonar, the open platform to manage code quality, with Jenkins.

Why Sonar?

If you ask a software engineering manager to list the items that he/she cares most about a software project?  One of the items must be software quality.  But, the question is how to measure the software quality, what are the metrics to measure software quality?  The obvious answer would be customer satisfaction, the number of bugs found by QA team.  However, those metrics are only available after the software is delivered to customer or QA team, which will be very late for development team to improve.  If development team could get software quality metrics feedback at the early phase, continuous improvements can be made before the software is delivered to customers.  That's the goal Sonar is trying to achieve.

What does Sonar do?

Sonar analyzes the source code and presents 7 axes of code quality for your software projects as the software is being developed.  It gives development team early feedback on the code quality. 

Note that, Sonar focuses on source code, so it doesn't do anything about whether the software meets customers requirements.  It helps you to write code right, but not helps you write right code.

The axes of code quality sonar analyze includes:

• Architecture & Design
• Duplications
• Unit Tests and Coverage
  
• Complexity
• Potential Bugs
• Coding Rules
• Comments

Sonar High Level Architecture

Below is the high-level Sonar architecture from the Sonar official website.

Sonar involves 3 major parts:

1. Sonar DB - the database to store the analysis results.
2. Sonar Runner - the application analyzes the source code and pushes the results into the sonar database.
3. Sonar Web Interface - the web interface for users to browse the analysis results.

According to the architecture, sonar analysis doesn't provide real time feedback as it takes time to analyze.  In general, making the sonar analysis as a nightly job is a good practice.

Sonar employs the plugins-based architecture which has built-in support for Java.  The C# support is done by the C# Plugins Ecosystem.  Let's take a look at the overview of the C# plugins ecosystem.

Introducing the C# Plugins Ecosystem

The C# Plugin Ecosystem (see: http://docs.codehaus.org/display/SONAR/C-Sharp+Plugins+Ecosystem) doesn't analyze the code by itself, but it invokes external tools to do the job.  Therefore, the C# plugins works as a bridge between sonar and the external tools.  It has the following major components. Descriptions are copied from the official website with some further explanations.

• [Required] C# Core : the core plugin that defines API used by every other plugin in the ecosystem
• [Required] C# Squid : the plugin that parses C# code source, computes metrics and detects copy-paste (distributed by SonarSource as a freeware)
• C# StyleCop : the plugin launches StyleCop analyses to detect violations primarily relating to coding style (formatting, comments, ...)
  - [Optional], the StyleCop is embedded in the plugin.
  
• C# Gendarme : the plugin launches Gendarme analyses to detect violations primarily relating to coding bad practices, smell detection, performance, ...
  - [Optional], the Gendarme is embedded in the plugin.
  
• C# FxCop : the plugin launches FxCop analyses to detect violations relating to potential bugs, tight coupling, globalization, ...
  - [Optional], requires to install FxCop which is shipped with windows SDK, see more details at: http://www.infoq.com/news/2010/06/FXCop-10
  
• C# Gallio : the plugin launches Gallio to run tests and gets execution report as well as coverage report (using PartCover or NCover)
  - [Optional], requires to install PartCover and Gallio to calculate the test code coverage.
  

The StyleCop, FxCop and Gendrame should be easy to setup without any tricks.  I'd like to explain the relationship between Gallio and PartCover a little bit. They work together to run the test cases and calculate the code coverage. Gallio is the tool to launch the real test runners such as mstest.exe to run the test cases. PartCover is the cover engine which calculates the code coverage details while the test cases are running.  Refer to the C# Plugin Ecosystem page on how to install the plugins and the required tools.  Be sure to install the tools version that are supported.

Note: the code coverage is supported in visual studio 2010.  You could enable it while running the test cases.  However, there is no sonar plugin to marry with the coverage results generated by VS2010.

Enable Sonar for the C# projects

To analyze C# projects using the sonar C# plugins, a sonar-project.properties must be created in folder where the .Net solution file is located. The file defines the attributes for the sonar analysis and the C# plugins.  Here are the things that you should know before creating your own sonar-project properties.

1. Do not manually write the key name, copy it from the official page.  This is because there is a typo in one of the keys (sonar.donet.visualstudio.testProjectPattern).  Some guys attempted to corrected it, and it causes failure.
2. Some keys can be configured with multiple values, but the the separators are different. Some uses comma(,) as the separator, while some uses semicolon(;) as the separator. Read carefully on the documentation while configuring such properties.
3. To comment a line, put # at the beginning of the line.  It seems that # in the middle doesn't comment the rest characters of the line.
   

Below is example sonar-project.properties file for my C# project (sensitive information removed).

##################################################################################### The properties below are for the sonar project, not pertain to the C# plugins##################################################################################### Project identification# This defines a unique key for the project that will be created in the sonar. Use the java package naming convention to define it.sonar.projectKey=com.mycompany.myproject# The projectVersion can be ovrriden in Jenkins with the build numbersonar.projectVersion= Manual Analysis# The project name that will be displayed in the Sonar web interface, make it meaningful.sonar.projectName= My Project 1# Info required for Sonarsources=.# Tell the source code is C#.sonar.language=cs# Source Encodingsonar.sourceEncoding=UTF-8##################################################################################### The properties below are for the C# plugins##################################################################################### Specify the assembly to analyze.  It could point to the build output folder.# Note: the absolute path is used here as relative path doesn't work for me.sonar.dotnet.assemblies = $(SolutionDir)/MyProject/bin/x86/Release/MyProject.exe# Specify the tests projects so that the code in the test project will not counted as the production code.# Note the separator is ';', and there is a typo in the key but do not attempt to correct it.sonar.donet.visualstudio.testProjectPattern = *MyProject_UnitTest;*MyProject_UAT# Sepcify the test assumblies which include the test cases, i.e. the build output of the test projects.  # Note the separator is ','.  This allows us to have 2 test projects.sonar.dotnet.test.assemblies = $(SolutionDir)/MyProject_UnitTest/bin/Release/MyProject_UnitTest.dll,$(SolutionDir)/MyProject_UAT/bin/ReleaseMyProject.dllsonar.gallio.runner = IsolatedAppDomain# Ignore the generated files to avoid bias on the result# Note the separator is ','sonar.exclusions = **/Generated.cs,**/Constants_Dynamic*.cs

Create your sonar-project.properties based on the example above, then you could test your configuration by using the Java runner.  The Java runner triggers the analysis, and the C# plugins invoke the external tools to do the real job.  After that the results will be pushed to the Sonar DB if everything goes well, and you will see the results via the sonar web interface which http://localhost:9000 by default.  Refer to here on how to use the Java runner:http://docs.codehaus.org/display/SONAR/3.+Run

Trouble Shooting

Possible Issues

1. The coverage report is not generated.
   Possible root cause: the pdb file must be generated during the build process.  This may be disabled for a release build.  Enable it in the C# project property in the Visual Studio. Property -> Build -> Advanced, in the Advanced Built Setting dialog, Select full or pdb-only option for the debug info.

Integrate with Jenkins

If the sonar analysis can be successfully run with the Java runner, it proves that the configurations are correct.  We can create a job in Jenkins to run the sonar analysis nightly.  To integrate Sonar with Jenkins, the sonar plugin for Jenkins must be installed.  After that, a new step in the job can be created to start the sonar analysis.  See figure below:

A few things to note:

1. The projects must be built before running sonar analysis.
2. As the figure shown, the project version can be overridden by using the build number.
3. Specify the JVM options in case of out of memory issue.

Summary

We've gone thru 3 major topics about continuous integration for .Net project, building the project, running unit tests and running the sonar analysis. With those setups, we could get early feedback on each code checkins.

The 3 topics represents different level of feedback, the lowest level is the project should be built without any compilation error, the 2nd level is the basic unit test should pass, and the 3rd level the code coverage and other code quality metric measured by sonar.  We could get some confidence about the code quality and the changes made, but we need more.  We need to know whether the software meets the customer requirements. That will be my future focus and interests.

Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning (By Winston Churchill).