Archive

Posts Tagged ‘Groovy’

Jenkins, Groovy System scripts (and Maven)

23 June 2011 3 comments

So, I spent two+ weeks doing the following:

  • Digging through the Hudson/Jenkins code
  • Learning the architecture of Hudson/Jenkins plugins
    • In particular SCM plugins, like the git plugins
  • Learning groovy
  • Writing a groovy script for use by the Jenkins groovy script plugin
    • Which retrieved variables from Hudson/Jenkins
    • And instantiated a maven build, from the script

The script/job I had to make finally works — although at the end of it all, I learned/realized that I couldn’t use the groovy system script option, but had to run the job as a normal groovy script.

Given that I had put enough time into it, I figured I’d post some of what I’ve done
in order to help others.

Jenkins Groovy build step options

Groovy build steps in Jenkins

This is a screenshot of the (Groovy) build step options in a job configuration in Jenkins (1.409.1). This functionality is added when you install the Groovy plugin for Jenkins (or Hudson).

  • The “system Groovy script” option executes a groovy script within the same JVM that Jenkins is running in.
  • The (normal) groovy script build step forks a new JVM for the script to run in.

[See the Groovy plugin link above for more info]

A system Groovy script thus has access to the heap of the Jenkins process: this allows us to access lots of information that is available within your Jenkins server, including the following:

  • Information over the current build
  • Information over other builds
  • Settings for your Jenkins server

In fact, because you not only have access to the information but can also change it (or delete information!), giving developers the option to run system Groovy scripts as part of their build is actually a little dangerous — and probably not recommended. But that’s another story.


 

The following script includes a couple of examples of how to retrieve information via the available instances and classes of the Jenkins infrastructure. It does the following:

  1. It first retrieves the ‘out’ binding.
    • I can’t find where this is documented, but when the Groovy Plugin starts it passes two things via Bindings to your script. One of these is the out variable, which you can use to print to the log saved by Jenkins. Normal println’s will NOT work here: use out.println
  2. It retrieves all build variables and other environmental variables and stores them in config
  3. It reaches into the Hudson guts and retrieves the path for all maven 3.x installations.
  4. It modifies the variables saved in configso that we can easily run maven
    • the M2_HOME has been known to cause problems for maven 3.
    • One of the parametrized variables for the build is the MAVEN_TARGET variable. This variables contains the maven targets; for example: “clean test”.
  5. In order to tell where the repo is that I want to build (in this case buildRepo, I search for the directory that this repo has been installed in.
    • This particular job has multiple git repositories, which is possible using the Multiple SCM plugin.
  6. Lastly, I run maven using a ProcessBuilder.
    • I pass all of the variables in configto the maven process.
    • I also pass through all of the output from the maven process to Jenkins.
    • Lastly, if the maven process does not end correctly, I tell Jenkins that, using build.setResult( Result.FAILURE )

 

// build and Result objects
import hudson.model.*

out.println "-" * 80

/**
 * VARIABLES
 */

def config = new HashMap()
def bindings = getBinding()
config.putAll(bindings.getVariables())

def out = config['out']

// specified variables (in Jenkins job)

def thr = Thread.currentThread()
def build = thr?.executable
def buildMap = build.getBuildVariables() 

config.putAll(buildMap)

// build/environmental variables
def envVarsMap = build.parent.builds[0].properties.get("envVars")

config.putAll(envVarsMap)

config.keySet().each { key -> 
  out.println key + ": " + config.get(key) 
}

out.println "-" * 80

// Set maven variables 
// - we can reach hudson information through static methods available to us
//   This is possible is because this is run as a sytem groovy script and has
//   access to the heap that hudson/jenkins is running on. 

def mvnBuilder = null
buildersList = hudson.tasks.Builder.all()
buildersList.each { builder -> 
  out.println "Builder: " + builder.id
  if( builder.id.equals("hudson.tasks.Maven") ) { 
    mvnBuilder = builder
  }
}

def mvn3Paths = [ : ]
mvnBuilder.installations.each { installation -> 
  out.println "Maven installation: " + installation.name
  if( installation.name.contains("maven-3") ) { 
    mvn3Paths.put( installation.name, installation.home )
  }
}

def mvn3Ver = null
def mvn3Path = null
if( mvn3Paths.size() > 0 ) { 
  mvn3Ver = mvn3Paths.keySet().sort().reverse()[0]
  mvn3Path = mvn3Paths.get(mvn3Ver)
  config
}
else { 
  out.println "UNABLE TO RETRIEVE MAVEN INSTALLATIONS FROM HUDSON!\n"
  build.setResult( Result.FAILURE )
  return -1
}

config.remove('M2_HOME')
config.put('MAVEN_HOME', mvn3Path )
config.put('M3_HOME', mvn3Path )
def mavenTarget = config['MAVEN_TARGET']

// Get the relative target dir

// closure to identify and retrieve the jbpm git repo info
def retrieveJbpmGitScm = { scm ->
  def gitScmFound = null
  if( scm.type.endsWith("GitSCM") ) { 
    scm.repositories.each { repo -> 
      repo.getURIs().each { uri -> 
        if( uri.path.endsWith("jbpm.git") ) { 
           gitScmFound = scm
        }
      }
    }
  }
  return gitScmFound
}
  
def buildScm = build.parent.scm
def jbpmGitScm = null;
if( buildScm.type.endsWith("MultiSCM") ) { 
  def scms = buildScm.configuredSCMs
  scms.each { scm ->
    if( jbpmGitScm == null ) { 
      jbpmGitScm = retrieveJbpmGitScm(scm)
    }
  }
}
else if( buildScm.type.endsWith("GitSCM") ) {  
  jbpmGitScm = retrieveJbpmGitScm(buildScm)
}

String relativeTargetDir = ""
if( jbpmGitScm != null ) { 
  relativeTargetDir = jbpmGitScm.relativeTargetDir
}
else { 
  out.println "UNABLE TO RETRIEVE JBPM GIT SCM INFORMTION FROM HUDSON!\n"
  build.setResult( Result.FAILURE )
  return 
}

/**
 * MAVEN BUILD
 */

def workspace = config['WORKSPACE']

def targets = []
(mavenTarget =~ /\w+/).each { match ->
  targets.add(match)
}
int m = 0;
String [] mvn3Cmd  = new String[2+targets.size()]
mvn3Cmd[m++] = config.get('MAVEN_HOME').toString() + File.separator + "bin" + File.separator  + "mvn" 
mvn3Cmd[m++] = "--fail-at-end"
targets.each { targetWord ->
  mvn3Cmd[m++] = targetWord
}

String processDir = workspace
if( relativeTargetDir != null && relativeTargetDir.length() > 0 ) {
  processDir += File.separator + relativeTargetDir 
}

File workspaceDir = new File(processDir)

ProcessBuilder mvnProcBuilder = new ProcessBuilder(mvn3Cmd);
mvnProcBuilder.redirectErrorStream(true);

Map env = mvnProcBuilder.environment()
env.clear()

config.keySet().each { key -> 
  try {  
    env.put(key, (String) config.get(key))
  }
  catch(ClassCastException cce) { 
    print "Unable to convert '" + key + "' to String: " 
    println cce.message
    // do nothing else..
  }
} 

mvnProcBuilder.directory(workspaceDir)

out.println "Starting maven build: " 
out.println "[cmd]: " + mvn3Cmd
out.println "[dir]: " + workspace
out.println "-" * 80

def mvnProc = mvnProcBuilder.start();

InputStream procOutput = mvnProc.getInputStream();
BufferedReader outputReader = new BufferedReader(new InputStreamReader(procOutput))

String line = null;
while((line = outputReader.readLine()) != null ) { 
  out.println( line )
}

// Just to make sure
mvnProc.waitFor()

if( mvnProc.exitValue() != 0 ) { 
  String message = "Maven build failed: ["
  mvn3Cmd.each { arg -> message += arg + " " } 
  message += "]"
  build.setResult( Result.FAILURE )
}

 

And here’s an (edited) example of the output.

Note the environmental variables that I’ve been able to retrieve as well as
some of the Jenkins information that I show.

Started by upstream project "SystemScript" build number 10

Checkout:axis-value-1 /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1 - hudson.remoting.LocalChannel@6e70c242
Using strategy: Default
Last Built Revision: Revision 93c2c662ccc478a5eb9e98ca36437458525d1231 (origin/5.1.x)
Checkout:scripts / /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1/scripts - hudson.remoting.LocalChannel@6e70c242
Fetching changes from the remote Git repository
Fetching upstream changes from git://github.com/mrietveld/myRepo.git
Commencing build of Revision 93c2c662ccc478a5eb9e98ca36437458525d1231 (origin/5.1.x)
Checking out Revision 93c2c662ccc478a5eb9e98ca36437458525d1231 (origin/5.1.x)
--------------------------------------------------------------------------------
TERM: vt100
M3_HOME: /opt/bin/apache-maven-3.0.3
JENKINS_HOME: /opt/bin/jenkins-1.409.1
JAVA_HOME: /usr/lib/jvm/jre
IMSETTINGS_MODULE: none
...
HUDSON_URL: http://127.0.0.1:8080/
...
JOB_URL: http://127.0.0.1:8080/job/SystemScript/./axis=axis-value-1/
...
BUILD_NUMBER: 10
...
EDITOR: gvim
M2_HOME: /opt/bin/apache-maven-2.2.1
PATH: /usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/libexec:/usr/ccs/bin
JENKINS_URL: http://127.0.0.1:8080/
LESS: -gS -j.25
QTLIB: /usr/lib64/qt-3.3/lib
PAGER: less
GZIP: -9v --name
KDE_IS_PRELINKED: 1
GDM_LANG: en_US.utf8
WINDOWPATH: 1
HUDSON_HOME: /opt/bin/jenkins-1.409.1
COMMON_TOOLS: /opt/bin
OSTYPE: linux
MAVEN_HOME: /opt/bin/apache-maven-2.2.1
NODE_LABELS: master
XFILESEARCHPATH: /usr/dt/app-defaults/%L/Dt
MANPATH: /usr/local/share/man:/usr/local/man:/usr/share/man:/usr/share/man/en:/usr/share/man/overrides:/usr/share/locale/man:/usr/java/jdk1.5.0_22/man:/usr/share/doc/man-pages-overrides-1.0/man
BUILD_ID: 2011-07-20_11-19-51
BUILD_TAG: jenkins-axis=axis-value-1-10
BUILD_URL: http://127.0.0.1:8080/job/SystemScript/./axis=axis-value-1/10/
axis: axis-value-1
SHELL: /bin/tcsh
WORKSPACE: /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1
MAVEN_TARGET: clean
ANT_HOME: /usr/share/ant
JOB_NAME: SystemScript/axis=axis-value-1
out: java.io.PrintStream@641ef158
--------------------------------------------------------------------------------
Builder: hudson.tasks.BatchFile
Builder: hudson.tasks.Shell
Builder: hudson.tasks.Ant
Builder: hudson.tasks.Maven
Builder: com.nirima.AdaptivePluginBuilder
Builder: hudson.plugins.groovy.SystemGroovy
Builder: hudson.plugins.groovy.Groovy
Maven installation: maven-2.2.1
Maven installation: maven-3.0.3
Starting maven build: 
[cmd]: [/opt/bin/apache-maven-3.0.3/bin/mvn, --fail-at-end, clean]
[dir]: /opt/bin/jenkins-1.409.1/jobs/SystemScript/workspace/axis/axis-value-1
--------------------------------------------------------------------------------
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.673s
[INFO] Finished at: Wed Jul 20 11:19:57 CEST 2011
[INFO] Final Memory: 6M/102M
[INFO] ------------------------------------------------------------------------
Finished: SUCCESS
Categories: CI/Buiilds Tags: ,