Test Smell Research

Tool Comparison

As part of our study, we also performed a comparative exercise of tsDetects ability of smell detection against state-of-art detection strategies. We were successful in acquiring Test Smell Detector (TSD), a tool used in a few recent test smell studies [1][2]. Ideally, the comparison has to be performed with a dataset of manually detected test smells. Since there are no publicly available test smell datasets, we decided to run both tools against a set of manually validated smells, identified in the previous experiment. However, the comparison was limited to smell types recognized by tsDetect and TSD. Our findings indicated that tsDetect is better equipped to detect smells in specific scenarios than TSD. Listed below are some scenarios that we encountered where tsDetect performed better than TSD.

Assertion Roulette

In Example #1, TSD falsely indicates that the method exhibits the smell. The detection is not valid as this method only contains one assertion statement. In Example #2, TSD falsely indicates that the test method does not exhibit the smell. The detection is not valid since there are multiple non-documented assertion statements.


//Example #1
@Test
public void testEncrypt() throws Exception {
    String xml = readFileAsString(DECRYPTED_DATA_FILE_4_14);
    byte[] encrypted = Cryptographer.encrypt(xml, "test");
    String decrypt = Cryptographer.decrypt(encrypted, "test");
    assertEquals(xml, decrypt);
}
                    
//Example #2:
@Test
public void testTimestamp() {
    Assert.assertEquals("201205292002",ExporterFileNameUtils.getTimeStamp(new Date(1338314522376L), Locale.GERMANY));   
    Assert.assertEquals("201205292010", ExporterFileNameUtils.getTimeStamp(new Date(1338315008925L), Locale.GERMANY));
}
                
Eager Test

Example #1, contains multiple calls to the production class, but is not indicated as a smell by TSD. However, it should be noted that the methods are defined in the parent class of the production class. Example #2 is reported as having an Eager Test smell, by TSD since the same production class method `isArea()' is called twice in the same method. However, it can be argued, that this not constitute to an Eager Test smell since it is the same method and the Eager Test rule definition does not indicate if the called methods have to be different.


//Example #1
public void testOnUpgrade() {
    DbHelper helper = createHelperVersion(1);
    assertTablesExist(helper.getReadableDatabase(), TABLES_V1);
    helper.close();                    
    helper = createHelperVersion(2);
    assertTablesExist(helper.getReadableDatabase(), TABLES_V2);
    helper.close();                    
    helper = createHelperVersion(3);
    assertTablesExist(helper.getReadableDatabase(), TABLES_V3);
    helper.close();                    
    helper = createHelperVersion(4);
    assertTablesExist(helper.getReadableDatabase(), TABLES_V4);
    helper.close();
}
                      
//Example #2:
public void testRelation(){
    assertFalse(OsmAreas.isArea(new OsmRelation(0,0, null, null)));
    Map tags = new HashMap<>();
    tags.put("type","multipolygon");
    assertTrue(OsmAreas.isArea(new OsmRelation(0, 0, null, tags)));
}
                
Lazy Test

The example is not reported as having a Lazy Test smell, by TSD. However, it should be noted that the methods are defined in the parent class of the production class. The existence of the smell in this scenario can be debated.


public void testOnCreate_v2() {
    DbHelper helper = createHelperVersion(2);
    assertTablesExist(helper.getReadableDatabase(), TABLES_V2);
    helper.close();
}
                    
public void testOnCreate_v3() {
    DbHelper helper = createHelperVersion(3);
    assertTablesExist(helper.getReadableDatabase(), TABLES_V3);
    helper.close();
}
                
Mystery Guest

TSD checks for the Mystery Guest smell by performing a text-based search for `File', `db' and `File('. In the sample code provided, there is a string variable that contains the characters 'db'; TSD wrongly identified this test file as containing the smell.


@Test
public void testParseRadicalName() {
    String kanjidicStr = "巛 565F U5ddb B47 S3 V1527 H9 MN8669 MP4.0326 P1-1-2 I0a3.2 Q2233.7 Ychuan1 Wcheon セン かわ T2 まがりがわ {curving river radical (no.47)}";
        KanjiEntry entry = KanjiEntry.parseKanjidic(kanjidicStr);
    .....      
}
                
 
X

References

[1] Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. Aretestsmellsreallyharmful? anempiricalstudy. EmpiricalSoftwareEngineering, 20(4):1052–1094, 2015.

[2] Fabio Palomba and Andy Zaidman. Does refactoring of test smells induce fixing flakytests? InProceedingsoftheInternationalConferenceonSoftwareMaintenance (ICSME). IEEE, 2017.