Archive for the ‘CI/Buiilds’ Category

jBPM 5 database testing mechanisms

22 July 2011 Leave a comment

I recently finished making a mechanism for enabling the use of different databases with a set of unit tests.

A thorough explanation of the mechanism in its entirety can be found here:

The reason I’m also posting this here is because the mechanism that I describe can actually be built in to most maven projects that contain unit tests that use persistence.

In my case, the specifics are as follows:

  • The mechanisms are built into the jBPM framework (the code can be found here).
  • The unit tests are spefically using JTA (the Java Transaction API).
  • The Transaction layer is handled by the Bitronix transaction manager.

But none of these details mean that the same mechanism can’t be applied to any other project, including those that don’t do anything with JTA. Using JPA is to some degree intrinsic to the mechanism, although other java persistence frameworks that also use XML configuration could also be used, I think.

In short, it comes down to maven filtering + the Properties mechanism in Java + using XML configuration for JPA (persistence.xml, in this case).

Categories: CI/Buiilds, jBPM, Persistence Tags:

Jenkins, Groovy System scripts (and Maven)

23 June 2011 3 comments

So, I spent two+ weeks doing the following:

  • Digging through the Hudson/Jenkins code
  • Learning the architecture of Hudson/Jenkins plugins
    • In particular SCM plugins, like the git plugins
  • Learning groovy
  • Writing a groovy script for use by the Jenkins groovy script plugin
    • Which retrieved variables from Hudson/Jenkins
    • And instantiated a maven build, from the script

The script/job I had to make finally works — although at the end of it all, I learned/realized that I couldn’t use the groovy system script option, but had to run the job as a normal groovy script.

Given that I had put enough time into it, I figured I’d post some of what I’ve done
in order to help others.

Jenkins Groovy build step options

Groovy build steps in Jenkins

This is a screenshot of the (Groovy) build step options in a job configuration in Jenkins (1.409.1). This functionality is added when you install the Groovy plugin for Jenkins (or Hudson).

  • The “system Groovy script” option executes a groovy script within the same JVM that Jenkins is running in.
  • The (normal) groovy script build step forks a new JVM for the script to run in.

[See the Groovy plugin link above for more info]

A system Groovy script thus has access to the heap of the Jenkins process: this allows us to access lots of information that is available within your Jenkins server, including the following:

  • Information over the current build
  • Information over other builds
  • Settings for your Jenkins server

In fact, because you not only have access to the information but can also change it (or delete information!), giving developers the option to run system Groovy scripts as part of their build is actually a little dangerous — and probably not recommended. But that’s another story.


The following script includes a couple of examples of how to retrieve information via the available instances and classes of the Jenkins infrastructure. It does the following:

  1. It first retrieves the ‘out’ binding.
    • I can’t find where this is documented, but when the Groovy Plugin starts it passes two things via Bindings to your script. One of these is the out variable, which you can use to print to the log saved by Jenkins. Normal println’s will NOT work here: use out.println
  2. It retrieves all build variables and other environmental variables and stores them in config
  3. It reaches into the Hudson guts and retrieves the path for all maven 3.x installations.
  4. It modifies the variables saved in configso that we can easily run maven
    • the M2_HOME has been known to cause problems for maven 3.
    • One of the parametrized variables for the build is the MAVEN_TARGET variable. This variables contains the maven targets; for example: “clean test”.
  5. In order to tell where the repo is that I want to build (in this case buildRepo, I search for the directory that this repo has been installed in.
    • This particular job has multiple git repositories, which is possible using the Multiple SCM plugin.
  6. Lastly, I run maven using a ProcessBuilder.
    • I pass all of the variables in configto the maven process.
    • I also pass through all of the output from the maven process to Jenkins.
    • Lastly, if the maven process does not end correctly, I tell Jenkins that, using build.setResult( Result.FAILURE )


// build and Result objects
import hudson.model.*

out.println "-" * 80


def config = new HashMap()
def bindings = getBinding()

def out = config['out']

// specified variables (in Jenkins job)

def thr = Thread.currentThread()
def build = thr?.executable
def buildMap = build.getBuildVariables() 


// build/environmental variables
def envVarsMap = build.parent.builds[0].properties.get("envVars")


config.keySet().each { key -> 
  out.println key + ": " + config.get(key) 

out.println "-" * 80

// Set maven variables 
// - we can reach hudson information through static methods available to us
//   This is possible is because this is run as a sytem groovy script and has
//   access to the heap that hudson/jenkins is running on. 

def mvnBuilder = null
buildersList = hudson.tasks.Builder.all()
buildersList.each { builder -> 
  out.println "Builder: " +
  if("hudson.tasks.Maven") ) { 
    mvnBuilder = builder

def mvn3Paths = [ : ]
mvnBuilder.installations.each { installation -> 
  out.println "Maven installation: " +
  if("maven-3") ) { 
    mvn3Paths.put(, installation.home )

def mvn3Ver = null
def mvn3Path = null
if( mvn3Paths.size() > 0 ) { 
  mvn3Ver = mvn3Paths.keySet().sort().reverse()[0]
  mvn3Path = mvn3Paths.get(mvn3Ver)
else { 
  build.setResult( Result.FAILURE )
  return -1

config.put('MAVEN_HOME', mvn3Path )
config.put('M3_HOME', mvn3Path )
def mavenTarget = config['MAVEN_TARGET']

// Get the relative target dir

// closure to identify and retrieve the jbpm git repo info
def retrieveJbpmGitScm = { scm ->
  def gitScmFound = null
  if( scm.type.endsWith("GitSCM") ) { 
    scm.repositories.each { repo -> 
      repo.getURIs().each { uri -> 
        if( uri.path.endsWith("jbpm.git") ) { 
           gitScmFound = scm
  return gitScmFound
def buildScm = build.parent.scm
def jbpmGitScm = null;
if( buildScm.type.endsWith("MultiSCM") ) { 
  def scms = buildScm.configuredSCMs
  scms.each { scm ->
    if( jbpmGitScm == null ) { 
      jbpmGitScm = retrieveJbpmGitScm(scm)
else if( buildScm.type.endsWith("GitSCM") ) {  
  jbpmGitScm = retrieveJbpmGitScm(buildScm)

String relativeTargetDir = ""
if( jbpmGitScm != null ) { 
  relativeTargetDir = jbpmGitScm.relativeTargetDir
else { 
  build.setResult( Result.FAILURE )


def workspace = config['WORKSPACE']

def targets = []
(mavenTarget =~ /\w+/).each { match ->
int m = 0;
String [] mvn3Cmd  = new String[2+targets.size()]
mvn3Cmd[m++] = config.get('MAVEN_HOME').toString() + File.separator + "bin" + File.separator  + "mvn" 
mvn3Cmd[m++] = "--fail-at-end"
targets.each { targetWord ->
  mvn3Cmd[m++] = targetWord

String processDir = workspace
if( relativeTargetDir != null && relativeTargetDir.length() > 0 ) {
  processDir += File.separator + relativeTargetDir 

File workspaceDir = new File(processDir)

ProcessBuilder mvnProcBuilder = new ProcessBuilder(mvn3Cmd);

Map env = mvnProcBuilder.environment()

config.keySet().each { key -> 
  try {  
    env.put(key, (String) config.get(key))
  catch(ClassCastException cce) { 
    print "Unable to convert '" + key + "' to String: " 
    println cce.message
    // do nothing else..

out.println "Starting maven build: " 
out.println "[cmd]: " + mvn3Cmd
out.println "[dir]: " + workspace
out.println "-" * 80

def mvnProc = mvnProcBuilder.start();

InputStream procOutput = mvnProc.getInputStream();
BufferedReader outputReader = new BufferedReader(new InputStreamReader(procOutput))

String line = null;
while((line = outputReader.readLine()) != null ) { 
  out.println( line )

// Just to make sure

if( mvnProc.exitValue() != 0 ) { 
  String message = "Maven build failed: ["
  mvn3Cmd.each { arg -> message += arg + " " } 
  message += "]"
  build.setResult( Result.FAILURE )


And here’s an (edited) example of the output.

Note the environmental variables that I’ve been able to retrieve as well as
some of the Jenkins information that I show.

Started by upstream project "SystemScript" build number 10

Checkout:axis-value-1 /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1 - hudson.remoting.LocalChannel@6e70c242
Using strategy: Default
Last Built Revision: Revision 93c2c662ccc478a5eb9e98ca36437458525d1231 (origin/5.1.x)
Checkout:scripts / /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1/scripts - hudson.remoting.LocalChannel@6e70c242
Fetching changes from the remote Git repository
Fetching upstream changes from git://
Commencing build of Revision 93c2c662ccc478a5eb9e98ca36437458525d1231 (origin/5.1.x)
Checking out Revision 93c2c662ccc478a5eb9e98ca36437458525d1231 (origin/5.1.x)
TERM: vt100
M3_HOME: /opt/bin/apache-maven-3.0.3
JENKINS_HOME: /opt/bin/jenkins-1.409.1
JAVA_HOME: /usr/lib/jvm/jre
EDITOR: gvim
M2_HOME: /opt/bin/apache-maven-2.2.1
PATH: /usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/libexec:/usr/ccs/bin
LESS: -gS -j.25
QTLIB: /usr/lib64/qt-3.3/lib
PAGER: less
GZIP: -9v --name
GDM_LANG: en_US.utf8
HUDSON_HOME: /opt/bin/jenkins-1.409.1
COMMON_TOOLS: /opt/bin
OSTYPE: linux
MAVEN_HOME: /opt/bin/apache-maven-2.2.1
XFILESEARCHPATH: /usr/dt/app-defaults/%L/Dt
MANPATH: /usr/local/share/man:/usr/local/man:/usr/share/man:/usr/share/man/en:/usr/share/man/overrides:/usr/share/locale/man:/usr/java/jdk1.5.0_22/man:/usr/share/doc/man-pages-overrides-1.0/man
BUILD_ID: 2011-07-20_11-19-51
BUILD_TAG: jenkins-axis=axis-value-1-10
axis: axis-value-1
SHELL: /bin/tcsh
WORKSPACE: /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1
ANT_HOME: /usr/share/ant
JOB_NAME: SystemScript/axis=axis-value-1
Builder: hudson.tasks.BatchFile
Builder: hudson.tasks.Shell
Builder: hudson.tasks.Ant
Builder: hudson.tasks.Maven
Builder: com.nirima.AdaptivePluginBuilder
Builder: hudson.plugins.groovy.SystemGroovy
Builder: hudson.plugins.groovy.Groovy
Maven installation: maven-2.2.1
Maven installation: maven-3.0.3
Starting maven build: 
[cmd]: [/opt/bin/apache-maven-3.0.3/bin/mvn, --fail-at-end, clean]
[dir]: /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.673s
[INFO] Finished at: Wed Jul 20 11:19:57 CEST 2011
[INFO] Final Memory: 6M/102M
[INFO] ------------------------------------------------------------------------
Finished: SUCCESS
Categories: CI/Buiilds Tags: ,