Jan 28, 2013

Using Fitnesse Test Tool for Acceptance Test in Mobile Applications


There are few tools to make acceptance tests for mobile applications, eg J2ME platform.
The tool J2MEUnit seems a good solution for unit testing tools but compared to Fit frameworks, the tests become more cumbersome. 
So I decided to use the test tool Fitnesse to validate business rules in a J2ME application and the tests implemented were simpler and faster. See the instructions  to do Test-First approach.


Configuration

To use the tool Fitnesse into eclipse, it is necessary to install Fitnesse plugin for eclipse.
1) Start up Eclipse and select the HELP menu -> Install new software
2) Click the Add button and type in the Name field: Fitnesse and field Location: http://www.bandxi.com/fitnesse/ or usu this Link.
3) Select Ok
4) Select Fitnesse Plugin for Eclipse and then click Next.
5) The Eclipse Plugin will be installed and will restart.
6) When eclipse restarts, the two icons of Fitnesse appear on the toolbar.

Create a Project

1) In eclipse, you click the File menu and select New-> Project.
2) You can select a JavaME Middlet Project.
3) Give a name, for example: FitnesseProject.
4) Select your new  project in the left of the eclipse package explorer tab.
5) Select the project created with right click and select Fitnesse -> Add to project classpath libraries fitnesse. 
6) Then click again with the right click over the project and select File -> New -> Folder to create a folder.
7) Give the name of the folder: fitnesse
8) Right-click the folder and select Fitnesse-fitnesse> Lunch Fitnesse runtime on this folder.
9) In the name of the project click the right click and select Configure Buildpath
10) In the window to configure Buildpath, add the following jars: 
  • collections.jar
  • ee.minimum.jar
It is necessary because of fitnesse dependences.

11) In your test project, select the src folder with the right click and select: New -> package, give the name of the fixtures this new package.
12) Build a fixture to test for example an calc method.
13) In your test project FitnesseProject go to fixtures package and right click and choose New-> Class
14) In the window to create the new class type the name CalcFixture and in superclass field, click Browse and enter in the window ColumnFixture.
15) Select the class from the list and select Ok to create the class.
16) Write a test fixture with the following code:

package fixtures;


import fit.ColumnFixture;

public class CalcFixture extends ColumnFixture {

public int a, b;
  
      
    public  int result() {
   
        return calc.add(a, b);
    }

}

The line  return calc.add(a, b); is gonna be red because the method add does not exist.

Lets create the system code:

17) In your project, create a new package called app (src right click and select: New -> package,).
18) Create a new Class in this package called Calc.
19) Insert a code just to fix the error in your fixture.

package app;

public class Calc {

public int add(int a, int b){
return 0;
}
}

Now the error in your fixture will disappear.

Executing Tests 

1) In the eclipse toolbar, click the icon to launch the Fitnesse server.
2) the eclipse browser will appear in the eclipse fitnesse page.
3) the page is similar to a Fitnesse wiki, click the Edit menu on the left top of the page.
4) Enter the word anywhere ProjectTest and save the change.
5) You will see the word followed by a question ProjectTest?
6) Click the interrogation ? mark.
7) You will see an edit page of a wiki.
8) Enter the following code in the wiki:

!path /C :/workspace/FitnesseProject/bin

! 3 Column Fixture
|!-fixtures.CalcFixture-! |
| a | b | result () |
| 1 | 2 | 3 |
| 13 | 5 | 18|

It is important to point to the bin folder of the the project using !path.
To indicate the package and fixture class to be used, we use the command |!-packagename.classname-! |
Then the decision table is implemented with the parameters to be passed, text separator and the expected return result class ().
9) Save the page in the wiki.
10) In the page saved, select the blue menu on the left Properties.
11) On page properties select Test and then click Save Properties.
12) When you return to your test wiki the button Test will appear in the menu.
13) Click the Test to run your test.
14) If a message in yellow saying there was not found the test class or fields, make sure the project generated classes in the bin folder or you are pointing to the correct address in the bin folder.
15) Another important check is if you put the exact same name as the parameter of the fixture in the table.
16) If the test run the errors in red will appear as the picture bellow:

The errors appear because the method add in the class Calc wasn't implemented.

17) Back to project and open the class: src -> app -> Calc.java.
18)Insert the code:

package app;

public class Calc {

public int add(int a, int b){
return a+b;
}
}

19) Save and back to Fitnesse page in the eclipse.
20) Don't forget to build the project.
21) Execute the fitnesse tests again clicking in Test.
22) The test table will be green, your test is passed as the picture bellow.

23) When your class will be totally tested, you can create a middlet and call your class Calc.
24) In src, right click and create the package with the name mobile in New-> package.
25) In the package created click the right mouse button and create a midlet. New-> midlet
26) Put the name of the HelloWorld midlet for example.
27) In midlet created add a private variable private Display display;.
28) In the startApp () put the following code:


Display display = Display.getDisplay (this);
String message = String.valueOf(new Calc().add(3, 4));
TextBox textBox = new TextBox ("Hello World", message, 50, 0);
display.setCurrent (textBox);

30) Run the midlet selecting the project and clicking the right click and selecting: Run As -> Emulated Java ME Midlet
31) The midlet will run Hello World and will appear the result of  add:  7.

For the next functionalities, repeat all the process :)

Fitnesse pros:

- A fixture class tests one class of the system.
- Parameters are passed only in wiki and it is not necessary one test method for each parameter (as JUnit).
- It is friendly and developers and Testers can use the eclipse itself.

Fitnesse cons:

- Tests are run only in eclipse and it is not possible to embed the tests in the emulator or in the cell phone as J2MEUnit do.

My Conclusion is:

If you want to validate business rules and acceptance criteria, the Fitnesse works well and fast. But if the developer wants to analyze the memory limits and the cell phone performance, so J2MEUnit and Manual tests are required. 

Jan 22, 2013

Starting to use Selenium Webdriver


For a long time I used the Selenium IDE and RC for web system testing. Using java platform, it was possible to build robust test suites and generate automatic reports using Junit.
Like last year I have been worked in systems for mobile devices, I had not yet experienced the successor of Selenium RC, the Selenium Webdriver.

Selenium webdriver is a tool designed to provide a simple interface and a concise programming to better support dynamics web pages.

In my research about Selenium Webdriver, I found a interesting comparison table that I show bellow. It can help to understand the differences between Selenium RC and Webdriver.


Selenium IDESelenium RCWebdriver
Works only on mozillaWorks on almost all browsers.Does not work on latest version of firefox/IEWorks on latest versions of almost all browsers - Firefox, IE(6,7,8), OPera, Chrome
Record and run toolNo Record and runNo Record and run
No server required to startServer is required to startNo server required to start
Core engine is Javascript basedCore engine is Javascript basedInteracts natively with browser application
Very simple to use. If using User extentoins, you require knowledge on javascript which makes the work a little bit tough.Its a simple and small APIComplex and a bit large API as compared to RC
Not at all object orientedLess Object orinted APIPurely Object oriented API
Cannot move mouse with itCannot move mouse with itCan move mouse cursor
Full xpaths have to be appended with 'xapth=\\' syntaxFull xpaths have to be appended with 'xapth=\\' syntaxNo need to append 'xpath=\\'
No ListenersNo ListenersImplementation of Listeners is provided
Cannot test iphone/Android applicationsCannot test iphone/Android applicationsCan test iphone/Android applications


source: http://qtpselenium.com/selenium-tutorial/difference-between-ide-rc-webdriver/


I found very interesting the following points:

  1. In Selenium Webdriver is not necessary to initialize the RC server.
  2. It is not necessary to use XPath.
  3. Fully object oriented.
  4. You can move the cursor.
  5. Can test on iphone and android applicat

For these reasons I decided to try the Selenium Webdriver.

I will describe my first example:

Installation and Configuration

  1. In Selenium page http://code.google.com/p/selenium/downloads/list  download the following  files: selenium-server-standalone.jar, selenium-java-2.29.0.zip , and the browser drivers : IEDriverServer_x64_2.29.0.zip and chromedriver_win_23.0.1240.0.zip .
  1. Open the Eclipse newest java version.
  2. Create a Java Project . File -> New -> Java Project
  3. Choose a  name for your  project,  SeleniumTest for example.
  4. The new project will appears in the package explorer.
  5. Select the project created in the package explorer and click using right mouse button.
  6. Create a new folder : New -> Folder
  7. The name of new folder is lib.
  8. Extract the selenium-java-2.29.0.zip and copy selenium-server-standalone.jar to inside the new folder lib.
  9. Select the jars and click using right mouse button and select Buildpath -> Add to Build Path.
  10. The IEDriver and Chromerdriver must be stored in a folder in the C:\BrowserDrivers.

Fast Example


Let's try to test the login page of testlink demo.

  1. Open the firefox and Install the Selenium IDE (http://seleniumhq.org/download/). 
  2. After Install the Selenium IDE, start it in Firefox -> Web Developer -> Selenium IDE.
  3. The Selenium IDE will open and start to record the actions in the browser.
  4. In firefox start open the page  of testlink demo  http://demo.testlink.org/tl194demo/login.php.
  5. In the Login Name enter: admin 
  6. Password enter: admin
  7. The testlink home page will appears.
  8. Stop the Selenium IDE record.
  9. In the Selenium IDE window, save the test in File-> Export As -> Java /JUnit 4/ webdriver.
  10. save the file with the name Login.java in a folder.
  11. Go to Eclipse IDE
  12. Select the src folder in your java project created .
  13. Create a class using right click and select New -> Class
  14. The name of the Class is: TestLogin
  15. Open your test recorded Login.java, copy the content and paste in the eclipse TestLogin class. Bellow the class example.

package example;

import java.util.regex.Pattern;
import java.util.concurrent.TimeUnit;
import org.junit.*;
import static org.junit.Assert.*;
import static org.hamcrest.CoreMatchers.*;
import org.openqa.selenium.*;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.Select;

public class TestLogin {
  private WebDriver driver;
  private String baseUrl;
  private boolean acceptNextAlert = true;
  private StringBuffer verificationErrors = new StringBuffer();

  @Before
  public void setUp() throws Exception {
    driver = new FirefoxDriver();
    baseUrl = "http://demo.testlink.org/";
    driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
  }

  @Test
  public void testEWebdriver() throws Exception {
    driver.get(baseUrl + "/tl194demo/login.php");
    driver.findElement(By.id("login")).clear();
    driver.findElement(By.id("login")).sendKeys("admin");
    driver.findElement(By.name("tl_password")).clear();
    driver.findElement(By.name("tl_password")).sendKeys("admin");
    driver.findElement(By.name("login_submit")).click();
  }

  @After
  public void tearDown() throws Exception {
    driver.quit();
    String verificationErrorString = verificationErrors.toString();
    if (!"".equals(verificationErrorString)) {
      fail(verificationErrorString);
    }
  }

  private boolean isElementPresent(By by) {
    try {
      driver.findElement(by);
      return true;
    } catch (NoSuchElementException e) {
      return false;
    }
  }

  private String closeAlertAndGetItsText() {
    try {
      Alert alert = driver.switchTo().alert();
      if (acceptNextAlert) {
        alert.accept();
      } else {
        alert.dismiss();
      }
      return alert.getText();
    } finally {
      acceptNextAlert = true;
    }
  }
}

  1. Fix the package name errors and the name of the class.
  2. In eclipse package explorer select TestLogin.java with mouse right click and Run As -> JUnit
  3. The Login recorded will run.

Adjusts

One of the resources of Selenium webdriver is run the same test for more than one web browsers. Some adjusts in our first example test is required.


  1. First create more two drivers variables :  private WebDriver driver1, driver2, driver3;
  2. In the setUp() method replace for the following code:
   driver1 = new FirefoxDriver();
  driver1.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
File file1 = new File("C:/BrowserDrivers/IEDriverServer.exe");
System.setProperty("webdriver.ie.driver", file1.getAbsolutePath());
driver2 = new InternetExplorerDriver();
driver2.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
File file2 = new File("C:/BrowserDrivers/chromedriver.exe");
System.setProperty("webdriver.chrome.driver", file2.getAbsolutePath());
driver3 = new ChromeDriver();
driver3.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
                
  baseUrl = "http://demo.testlink.org/";


To run my test was necessary to include the browser driver address in the computer (File file1 = new File("C:/BrowserDrivers/IEDriverServer.exe");) to start the IE browser and Chrome browser.

Change your test code adding the tests for each browser. See bellow:

 @Test
 public void testLoginFirefox() throws Exception {
 
 driver1.get(baseUrl + "/tl194demo/login.php");
     login(driver1);
 }
 
 @Test
 public void testLoginIE() throws Exception {
 
 driver2.get(baseUrl + "/tl194demo/login.php");
 login(driver2);
 }
 
 @Test
 public void testLoginChrome() throws Exception {
 
 driver3.get(baseUrl + "/tl194demo/login.php");
 login(driver3);
 }
 public void login(WebDriver driver){
 
 driver.findElement(By.id("login")).clear();
 driver.findElement(By.id("login")).sendKeys("admin");
 driver.findElement(By.name("tl_password")).clear();
 driver.findElement(By.name("tl_password")).sendKeys("admin");
 driver.findElement(By.name("login_submit")).click();
}
@After
 public void tearDown() throws Exception {
   driver1.quit();
   driver2.quit();
   driver3.quit();
   String verificationErrorString = verificationErrors.toString();
   if (!"".equals(verificationErrorString)) {
     fail(verificationErrorString);
   }
 }

All drivers must be closed using driver.quit() function.

Save the changes and Run the TestLogin.java again.
Run As -> JUnit.

As result the Login test will be executed for 3 browsers: Firefox, IE and Chrome browsers.

There's no need to start the Selenium server before the tests and without effort it was possible to run tests for  different kind of browsers.

I will try to test using resources for mouse cursor moving and I will post here the results.

Bye.

Jan 16, 2013

How I Automate Tests...


To automate tests, first I think the requirements that I need to start.
The figure 1 bellow shows the inputs and the outputs that I consider in test automation.

Figure 1 Test Automation Inputs and Outputs
To start the test automation activities I need the inputs:

  1. Test Cases selected to be automated, they must be repeatable.
  2. Test Data.
  3. Test Tools environment configured and preferably Continuous Integration.
  4. Code, even using TDD (test driven development) the code (can be fake)  is needed to run the tests.

As desirable outputs of test automation, basically I consider the following artifacts:

  1. Test Scripts.
  2. Execution Log.
  3. Results Report.
  4. Traceability (tests and requirements or tests and failures) and code coverage.
Depending of  test tool, other metrics could be extracted from test automation.

During the project a lot of external events can happen to disrupt the activities of automation as changes in requirements, extra requests from customer, change of deliverable dates and etc. For this reason I adopt some procedures listed bellow to automate tests and execute the software test activities.

  1.  Plan the test cases for the priority functionalities developed using a test tool (for example Testlink). 
  2. Select the test cases to be automated (selection can be according to acceptance criteria).
  3.  Automated test scripts creation according to test cases selected to be automated. These test scripts must be possible to be edited and gathered in a test suite, and reused to others test cases;
  4. Update the test scripts is important to avoid wrong results.
  5. Then, the testers executed the automated tests using a test tool and the exploratory manual tests also. The defects found were registered in Testlink and in the defect tracking tool.
  6. The automatic regression tests for all functionalities  must be executed also when defects were validated to ensuring that another part of system was not affected by code changes.
  7. The automated TestLink reports can be generated. The development team released another system version and the same activities process began again. 
I call these procedures Test Iteration because it will be repeated again for next version of software until the end of project.
I executed it in Scrum projects and Waterfall projects as well.My experience is reported in my paper "Iterative Software Testing Process for Scrum and Waterfall Projects using Open Source Testing Tools Experience". It was presented in the event ICTSS 2010 (International Conference of Testing Software and Systems).
Executing these procedures is possible to ensure the test automation activities in the project and avoid problems as outdated test scripts.

According to project platform is possible to use many test automation tools, for example:

Web projects: Selenium Test tool and Jmeter
For desktop projects (Java AWT or Swing) : Marathon test tool
For C# or Java application is possible to use Fitnesse test tool
For mobile J2ME applications : J2MEUnit


My favorite tools to use are Selenium, Fitnesse and Jmeter. In next posts I will detail how to use them.