On Automatically Generating Commit Messages via Summarization of Source Code Changes- SCAM 2014 Online Appendix

This web page is a companion to our SCAM 2014 paper entitled "On Automatically Generating Commit Messages via Summarization of Source Code Changes".


1. What is ChangeScribe?


ChangeScribe is an approach for generating commit messages (a.k.a., commit logs, commit notes) automatically. ChangeScribe uses as input two versions of a Java system, and generates a message with a general description of the changes (e.g., commit stereotype, rationale) and a detailed description of each change in natural language. The length of the message can be controlled by the developer by setting an impact-value threshold, which filter out from the message all the classes in the change set above that threshold. With impact value we mean the ratio between the methods impacted by changes to the class, over the total number of methods in the change set.

Click here to see a video of ChangeScribe in action.

The following diagram depicts the ChangeScribe components:


Click to zoom in
ChangeScribe is also available as an Open Source Eclipse Plugin ( under Eclipse Public License) at Github:

https://github.com/SEMERU-WM/ChangeScribe


Installation steps

  1. In Eclipse IDE select Help > Install New Software ... (An installation dialog will be displayed).
  2. Click to Add ... button. Define a name for this update site and copy the following link: http://www.cs.wm.edu/semeru/data/ICSME14-ChangeScribe/update
  3. Click Ok button.
  4. Open summarizer group.
  5. Check the box next to ChangeScribe plugin.
  6. Click Next > button.
  7. Select ChangeScribe plugin to view installation details.
  8. Accept the terms of the license agreement. Click Next > button again.
  9. Click Finish button.
  10. Restart Eclipse IDE.
  11. Use ChangeScribe!


2. User Study


Commits and Messages

The commits and messages (original developer's and ChangeScribe's) list is available here in html version or as a CSV file. According to the categories by Hattori and Lanza [ASE 2008] the 50 commits are distributed as in the following:

System

Commit ID

Link to the commit

Link to the commit messages

Elastic Search

b81e3de85e7ee2285e61ce2fa3bb5286cd65ffbd

http://goo.gl/z8qwgj

files/commitmessage1.html

Elastic Search

9282ae4ffd14545d13e7273573ab5010b7545e23

http://goo.gl/BSiHWf

files/commitmessage2.html

Elastic Search

93a8e80affc855f156d9295568409aab3972ffa1

http://goo.gl/kcixqq

files/commitmessage3.html

Elastic Search

bdced9c7723816d2a1c4f777701493d242ad6c2a

http://goo.gl/zbrRuP

files/commitmessage4.html

Elastic Search

fc6bc4c4776a2f710f57616e3495aaf6a230c4d3

http://goo.gl/7uaWM9

files/commitmessage5.html

Retrofit

44e798e2834d2c2d42b8310f1838169b6310411f

http://goo.gl/ixwfXW

files/commitmessage6.html

Retrofit

ab26b8b06e5ded58306496e44d315622246f1ae7

http://goo.gl/WTqEcx

files/commitmessage7.html

Retrofit

654f3f0eb47ec33ec5a5a7240c5c12ec2fa58c35

http://goo.gl/60hRGz

files/commitmessage8.html

Retrofit

c6776aa9d6f1ef7708e285b785921d1d69d7dbe6

http://goo.gl/mmbxzC

files/commitmessage9.html

Retrofit

ed2e8a5b52e933034b74e5aa66bc42862bfd30de

http://goo.gl/ErsNOZ

files/commitmessage10.html

Spring Social

000721a5285807b147e8937f53af3667a79b131e

http://goo.gl/K5ABFa

files/commitmessage11.html

Spring Social

852c62509660b4ea9f64eedc483da7c3c3a6957f

http://goo.gl/dXFlGd

files/commitmessage12.html

Spring Social

b190311e40eb7ac48c95da814f94afc55105e282

http://goo.gl/NK6TRd

files/commitmessage13.html

Spring Social

fac93fc12d8027381fb3aacfdb7d5b1840dccb43

http://goo.gl/QFrlrN

files/commitmessage14.html

Spring Social

d2ec33111fb5d12e79480a34bd001971c18f739b

http://goo.gl/fO57VP

files/commitmessage15.html

Spring Social

5b1c92e9bd5b2854379803d49d3d246f7ac480c1

http://goo.gl/6MFLHr

files/commitmessage16.html

Spring Social

03b308d622200f77dab9827500e40a50bb3491d7

http://goo.gl/zj2iZc

files/commitmessage17.html

Spring Social

8b1f8d8d6e4589926d0bab4cce5ef71e5f32e352

http://goo.gl/64dI1z

files/commitmessage18.html

Spring Social

0f2f72fb3e8cfb2cbbef67fbc958a89532250b92

http://goo.gl/i4PpCs

files/commitmessage19.html

Spring Social

7b029f2b765e9a789b5671f0e768e758f716cd1a

http://goo.gl/RXu8IH

files/commitmessage20.html

JFreeChart

01e24fcb3a8c0bc77570cc8e04604b92c3cf0768

http://goo.gl/8mWEvp

files/commitmessage21.html

JFreeChart

73e9a423c96a6275df2571fa901efbef3011ed2f

http://goo.gl/YD61GZ

files/commitmessage22.html

JFreeChart

12b3d349a597ad09df0f324c4ab7e0d3ab65172e

http://goo.gl/ynoYhN

files/commitmessage23.html

JFreeChart

74e804eb896bb7fc70fb1bb3071322ff74cd9fc5

http://goo.gl/VXRlRO

files/commitmessage24.html

JFreeChart

eebe22d75077c33ec126a29a2573f307c3e5e2b8

http://goo.gl/CIgUOr

files/commitmessage25.html

JFreeChart

2f6c23b2dbcc5dab755ed2e78bafcc217fa33456

http://goo.gl/E6Qnhr

files/commitmessage26.html

JFreeChart

6e9c0bdfe2b2957a0f4d06757ad6ec5e5d2560fa

http://goo.gl/cx7V4y

files/commitmessage27.html

JFreeChart

8c4514dec8254d530e311be36075b6290a240ad9

http://goo.gl/C5WHZa

files/commitmessage28.html

JFreeChart

7eb6e6f248dfd272d1e8c95300f9945c41aa05d7

http://goo.gl/FbXePs

files/commitmessage29.html

JFreeChart

ecbd702c2c81a04a0f2e75162b650d700920b490

http://goo.gl/LykuD0

files/commitmessage30.html

Apache Solr

aafff21e67ae628085d6b1c53e7526bbf6424d2c

http://goo.gl/Io9ik9

files/commitmessage31.html

Apache Solr

6adadf76461893a30fee9b3be461e9b6d69c690b

http://goo.gl/fj2bXC

files/commitmessage32.html

Apache Solr

d4e14735a500c18a91e72a0fe6be4b09b995ed8d

http://goo.gl/7r0IHD

files/commitmessage33.html

Apache Solr

783d774562343190ea543efe452f128809645456

http://goo.gl/HeZEAN

files/commitmessage34.html

Apache Solr

78950416c289dc72ab7bd88349603ec7fed9ccfe

http://goo.gl/JGQfx5

files/commitmessage35.html

Apache Solr

088f1973021abdc83d2207fb2543182ec26d2f38

http://goo.gl/PQuFfz

files/commitmessage36.html

Apache Solr

7702de0a6ca22f9aad670f5952bd85618d105c9d

http://goo.gl/UH2jwX

files/commitmessage37.html

Apache Solr

761ff1a1433ba27be3e64387f484dcf339368dd8

http://goo.gl/2V5Ysn

files/commitmessage38.html

Apache Solr

fa452cec34e242610bd41ff0c2a40c9f19738ebd

http://goo.gl/Z6lQUz

files/commitmessage39.html

Apache Solr

c9b036ce2461a956964f7649cadd90c5e4f1a5a1

http://goo.gl/Pm4Evl

files/commitmessage40.html

Apache Felix

d98e08a5c8eb9548b32c0fed96c0e3108095acd8

http://goo.gl/awUIdG

files/commitmessage41.html

Apache Felix

52b15d67f3a85f97f1771aa50b0e2e24a0fc3dd5

http://goo.gl/s7RyUD

files/commitmessage42.html

Apache Felix

7a9a3cece7e0eef1d6bfa2f882a8f3e108871e03

http://goo.gl/DQkj87

files/commitmessage43.html

Apache Felix

41ff672929c81d805313b4aec902b47963733c69

http://goo.gl/L5MCOC

files/commitmessage44.html

Apache Felix

5ae51c582e168d0d11475ceed0bcde5dcf750f0c

http://goo.gl/p0TZwg

files/commitmessage45.html

Apache Felix

5d22673a314129bcf5aa575513f31d228d65085e

http://goo.gl/5aNUcW

files/commitmessage46.html

Apache Felix

5bef4a83258249f4204d2c33a15cc9888d33bf5a

http://goo.gl/MiPSBr

files/commitmessage47.html

Apache Felix

993435b880210933c1cd1933abf36cdd32cbade9

http://goo.gl/lfQbLh

files/commitmessage48.html

Apache Felix

d64965620accd9e91c308399a03cd32ec31276a9

http://goo.gl/FVDNbX

files/commitmessage49.html

Apache Felix

e26ed202103200c03ac5b413be3410d628458ff2

http://goo.gl/LRCN7R

files/commitmessage50.html

3. Results

The spreadsheet with all the data collected with the user study can be downloaded here. The sheets in the file have the following data:
  • RawData from Qualtrics: the dataset as it was collected by Qualtrics
  • FilteredData: the dataset after eliminating the commit messages evaluated as "bad"
  • CommitStatsA: statistics using the raw data set
  • CommitStatsB: statistics using the filtered data set
  • Preference: Stats related to participants' preferences
  • Participants: Stats related to participants' background
  • Data for Mann-Whitney test: tables used for Mann-Whitney test and effect size

Answers to Demographic Questions(PDF)


*Authors

  • Luis Fernando Cortés-Coy - Universidad Nacional de Colombia, Bogotá, Colombia
    Email: lfcortesco at unal dot edu dot co
  • Mario Linares-Vásquez - The College of William and Mary, VA, USA.
    E-mail: mlinarev at cs dot wm dot edu
  • Jairo Aponte - Universidad Nacional de Colombia, Bogotá, Colombia
    Email: jhapontem at unal dot edu dot co
  • Denys Poshyvanyk - The College of William and Mary.
    E-mail: denys at cs dot wm dot edu