Verification bot

From Dogtag
Revision as of 17:53, 10 November 2017 by Mharmsen (talk | contribs) (Architecture)

Jump to: navigation, search

Overview

Whenever there is a patch submitted by a developer on Gerrithub, there might be a chance that a bug is introduced and the build might fail. These kind of logical/code quality errors might go unnoticed by the reviewer. These bugs can only be identified after it has been pushed to official repo and Travis CI catches the bugs.

This bot keeps monitoring for patches and tries to run a Travis CI test to identify these kind of bugs. The bot then updates the patch with Verified +1/-1 based on whether the patch is stable/unstable.

Architecture

The overall architecture of the bot can be summarized as below:

Pki-jenkins-bot-arch.jpg

1 - Developer submits patch to GerritHub

2 - Jenkins listens for patches and triggers a job

3 - Jenkins job creates a throw away branch and triggers Travis CI job

4 - Jenkins monitors for build status

5 - Jenkins updates the patch's Verified column with success (+1) / failure (-1) [return to step 1 on failures]

6 - Reviewer(s) review the code and mark success (Code-Review +1 or +2) or failure (return to step 1)

7 - Once patch receives Code-Review +2, the developer adds Workflow +1 and clicks on Submit

8 - Gerrithub pushes the patch to the official repo

Manual Steps to configure the bot

Installing Jenkins and Required Plugins

1. Install jenkins from official repo using the following command

   sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
   sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
   sudo dnf install -y jenkins

2. Install java (open-jdk):

   sudo dnf install -y java

3. Make sure to start the jenkins server:

   sudo systemctl start jenkins
4. Setup jenkins admin user password by visiting http://localhost:8080/. The initial password is available at
/var/lib/jenkins/secrets/initialAdminPassword

5. Click on Select plugins to install and select the following plugins and click Next:

  • github
  • ws-cleanup
  • credentials
  • gerrit-trigger
  • credentials-binding
  • plain-credentials
  • postbuild-task
  • envinject"

6. Provide the admin username and password and then click on Save and Finish

7. At this point you should have the following screen:

Jenkins-home.png

Creating SSH key for the bot

1. Run ssh-keygen and generate a public-private key pair.

2. Have this handy as you'll be using it in couple of next steps.

3. Add the public key to the pki-jenkins-bot user's SSH keys in Github

4. Copy the generated keys to /var/lib/jenkins/.ssh

Configuring Gerrit server for PKI

1. From Jenkins dashboard, click on Manage Jenkins from the left pane.

2. Select Gerrit Trigger.

3. Click on Add new server from the left pane.

4. Enter a name for the server and select Gerrit Server with Default Configurations

5. Enter the following configuration. Make sure to enter the password of the ssh key that you used in the previous step.

Gerrit-server-config.png

6. Click on Test connection to ensure the connection is successful.

7. Leave the default values

8. Click on Save.

9. Ensure that the small circle stays blue. If not, click on the small circle to bring it up.

Gerrit-server-final.png

Adding Github Token to the Jenkins

1. Go to pki-jenkins-bot's github profile.

2. Go to Settings > Developer Settings > Personal Access Tokens

3. Click on Generate new token and choose appropriate checkboxes and Click OK

4. Copy the generated token and keep it safe. This token once lost cannot be retrieved and has to be regenerated.

5. Go to Jenkins Dashboard. Click on Credentials > Jenkins > Global credentials (unrestricted) > Add Credentials (from left pane)

6. Select Secret Text from Kind and then enter the copied token in Secret. Enter github_token in the ID field.

7. Click OK.

Creating the pki-jenkins-bot job

1. Create a new job by clicking on New Item on the left or Create new jobs link

2. Enter the name of the project and select Freestyle project. Click OK

3. Configure the project as the following:

  • Check Execute concurrent builds if necessary
  • Select Git under Source Code Management and enter the following URL. Here, pki-jenkins-bot is the username of the Github Account.
$GERRIT_SCHEME://pki-jenkins-bot@$GERRIT_HOST:$GERRIT_PORT/$GERRIT_PROJECT
  • Click on Add credentials -> Jenkins. In the window that pops up, choose SSH username with private key. Enter pki-jenkins-bot in the username and the private key that you generated in the previous instruction
  • Click on Advanced under the credentials and Enter $GERRIT_REFSPEC under Refspec
  • Enter $GERRIT_BRANCH under Branch Specifier
  • Under Build Triggers check Gerrit Event and select the Gerrit server name you created in the previous step. Under Trigger on click Add and add Patchset created and Change Merged.
  • Under the Gerrit project, enter the project name as Type: Plain and Pattern: dogtagpki/pki and Branch Type: Path and Pattern: **
  • Under Build Environment, check Delete workspace before build, Inject environment variables to the build process - Under the groovy script, enter the following:
TimeZone.setDefault(TimeZone.getTimeZone('UTC'))
def now = new Date()
def map = [BRANCH_NAME: "bot_" + now.format("yyyyMMdd-HHmmss") + "_" + currentBuild['number']]
return map
  • Check "Use secret text(s) or file(s)" and add Secret Text. Enter the name as github_token and select the github_token created in the previous step.
  • Under Build, click on Add -> Execute Shell and enter the following script:
#create a temp branch and push it to official repo
ssh-keyscan github.com >> $JENKINS_HOME/.ssh/known_hosts
git remote add bot git@github.com:pki-jenkins-bot/pki.git
git checkout -b $BRANCH_NAME
git push --set-upstream bot $BRANCH_NAME
python $JENKINS_HOME/scripts/travisStatus.py -t $github_token -r pki-jenkins-bot/pki -b $BRANCH_NAME
  • Under Post-build action, click on add and select Post build task. Enter the following script:
git push -d bot $BRANCH_NAME
  • Click on Save to save this job

Copy the Travis Py script

In the Build step above, you would have added a line to execute Python script. You need to copy the file to the required location.

1. Obtain the copy of the Python script to monitor Travis from Github.

2.Copy to the Jenkins installations dir with the name travisStatus.py:

$JENKINS_HOME/scripts/travisStatus.py

Note: by default the $JENKINS_HOME points to /var/lib/jenkins

Ansible Playbook to automate

Ansible playbook that automates all the above steps has been developed and is available in Github repo. Even though all these manual step are automated, certain steps that involve security have to be done manually. The repository includes README instructions on how to execute the Ansible Playbook.

Demonstration Video

A general idea of how the bot works and a quick demo of the Bot setup using Ansible playbook is available on youtube.

References

Gerrit trigger on patch submission