Whenever there is a patch submitted by a developer on Gerrithub, there might be a chance that a bug is introduced and the build might fail. These kind of logical/code quality errors might go unnoticed by the reviewer. These bugs can only be identified after it has been pushed to official repo and Travis CI catches the bugs.
This bot keeps monitoring for patches and tries to run a Travis CI test to identify these kind of bugs. The bot then updates the patch with Verified +1/-1 based on whether the patch is stable/unstable.
The overall architecture of the bot can be summarized as below:
1 - Developer submits patch to GerritHub
2 - Jenkins listens for patches and triggers a job
3 - Jenkins job creates a throwaway branch and triggers Travis CI job
4 - Travis updates the label of the Gerrit patch. Adds comment with logs in case of Job failure
5 - Reviewer(s) review the code and mark success (Code-Review +1 or +2) or failure (return to step 1)
6 - Once patch receives Code-Review +2, the developer adds Workflow +1 and clicks on Submit
7 - Gerrithub pushes the patch to the official repo
Manual Steps to configure the bot#
Installing Jenkins and Required Plugins#
Install jenkins from official repo using the following command
Install java (open-jdk):
`` sudo dnf install -y java``
Make sure to start the jenkins server:
`` sudo systemctl start jenkins``
4. Setup jenkins admin user password by visiting http://localhost:8080/. The initial password is available at
5. Click on Select plugins to install and select the following plugins and click Next:
6. Provide the admin username and password and then click on Save and Finish
At this point you should have the following screen:
Creating SSH key for the bot#
Run ssh-keygen and generate a public-private key pair.
Have this handy as you’ll be using it in couple of next steps.
3. Add the public key to the pki-jenkins-bot user’s SSH keys in Github
Copy the generated keys to /var/lib/jenkins/.ssh
Configuring Gerrit server for PKI#
1. From Jenkins dashboard, click on Manage Jenkins from the left pane.
Select Gerrit Trigger.
Click on Add new server from the left pane.
4. Enter a name for the server and select Gerrit Server with Default Configurations
5. Enter the following configuration. Make sure to enter the password of the ssh key that you used in the previous step.
Click on Test connection to ensure the connection is successful.
Leave the default values
Click on Save.
9. Ensure that the small circle stays blue. If not, click on the small circle to bring it up.
Adding Github Token to the Jenkins#
Go to pki-jenkins-bot’s github profile.
Go to Settings > Developer Settings > Personal Access Tokens
3. Click on Generate new token and choose appropriate checkboxes and Click OK
4. Copy the generated token and keep it safe. This token once lost cannot be retrieved and has to be regenerated.
5. Go to Jenkins Dashboard. Click on Credentials > Jenkins > Global credentials (unrestricted) > Add Credentials (from left pane)
6. Select Secret Text from Kind and then enter the copied token in Secret. Enter github_token in the ID field.
Creating the pki-jenkins-bot job#
1. Create a new job by clicking on New Item on the left or Create new jobs link
2. Enter the name of the project and select Freestyle project. Click OK
Configure the project as the following:
Check Execute concurrent builds if necessary
Select Git under Source Code Management and enter the following URL. Here, pki-jenkins-bot is the username of the Github Account.
Click on Add credentials -> Jenkins. In the window that pops up, choose SSH username with private key. Enter pki-jenkins-bot in the username and the private key that you generated in the previous instruction
Click on Advanced under the credentials and Enter $GERRIT_REFSPEC:$GERRIT_REFSPEC under Refspec
Enter $GERRIT_REFSPEC under Branch Specifier
Under Build Triggers check Gerrit Event and select the Gerrit server name you created in the previous step. Under Trigger on click Add and add Patchset created, Change Merged and Comment Added Contains Regular Expression and enter the value (?i)^(Patch Set [0-9]+:)?(nn)?s*brechecks*$ to trigger bot when comment “recheck” is added to the patch.
Under the Gerrit project, enter the project name as Type: Plain and Pattern: dogtagpki/pki and Branch Type: Path and Pattern: **
Under Build Environment, check Delete workspace before build, Inject environment variables to the build process - Under the groovy script, enter the following:
def now = new Date()
def map = [BRANCH_NAME: "bot_" + now.format("yyyyMMdd-HHmmss") + "_" + currentBuild['number']]
Check “Use secret text(s) or file(s)” and add Secret Text. Enter the name as github_token and select the github_token created in the previous step.
Under Build, click on Add -> Execute Shell and enter the following script:
#create a temp branch and push it to official repo
ssh-keyscan github.com >> $JENKINS_HOME/.ssh/known_hosts
git remote add bot firstname.lastname@example.org:pki-jenkins-bot/pki.git
git checkout -b $BRANCH_NAME
git push --set-upstream bot $BRANCH_NAME
python $JENKINS_HOME/scripts/travisStatus.py -t $github_token -r pki-jenkins-bot/pki -b $BRANCH_NAME
Under Post-build action, click on add and select Post build task. Enter the following script:
git push -d bot $BRANCH_NAME
Click on Save to save this job
Copy the Travis Py script#
In the Build step above, you would have added a line to execute Python script. You need to copy the file to the required location.
1. Obtain the copy of the Python script to monitor Travis from Github.
2.Copy to the Jenkins installations dir with the name travisStatus.py:
Note: by default the $JENKINS_HOME points to /var/lib/jenkins
Retrigger build job for a patch#
To retrigger build on a patch that has already been build, add keyword recheck as a comment and publish it. The bot will be triggered automatically.
Patch build details and Build log analysis#
The jenkins build log can be accessed from the URL posted in the comments of the patch or directly from the Jenkins server link (You must be connected to the RedHat VPN)
The travis build can be accessed from bot’s Travis build https://travis-ci.org/pki-jenkins-bot/pki/builds
The temporary Github repo used to push patches for build is available at pki-jenkins-bot/pki
Note: This repo shouldn’t be cloned/used anywhere.
In case the build fails, the corresponding log will be uploaded to transfer.sh whose link can be obtained from the Travis log.
Ansible Playbook to automate#
Ansible playbook that automates all the above steps has been developed and is available in Github repo. Even though all these manual step are automated, certain steps that involve security have to be done manually. The repository includes README instructions on how to execute the Ansible Playbook.
A general idea of how the bot works and a quick demo of the Bot setup using Ansible playbook is available on youtube.