Agile Testing with Python Test Frameworks

Grig Gheorghiu

Avamar

What is agile testing?

Resources:
James Bach's site (exploratory testing and much more): http://www.satisfice.com
Brian Marick's Agile Testing site: http://testing.com/agile
agile-testing Yahoo group: http://groups.yahoo.com/group/agile-testing
Elizabeth Hendrickson article on Agile Testing: http://www.qualitytree.com/ruminate/011905.htm
TDD portal: http://www.testdriven.com

Python as an agile language

Python as an agile language (cont.)

Ron Jeffries:
"""
Extreme Programming is a discipline of software development based on values of simplicity, communication, feedback, and courage. It works by bringing the whole team together in the presence of simple practices, with enough feedback to enable the team to see where they are and to tune the practices to their unique situation.
"""

Unit testing with unittest

Resources:
pyUnit home page: http://pyunit.sourceforge.net/pyunit.html

import unittest

class TestSort(unittest.TestCase):
    def setUp(self):
        self.alist = [5, 2, 3, 1, 4]

    def test_ascending_sort(self):
        self.alist.sort()
        self.assertEqual(self.alist, [1, 2, 3, 4, 5])
        
def suite():
    suite = unittest.TestSuite()
    suite.addTest(unittest.makeSuite(TestSort))
    return suite

if __name__ == "__main__":
    unittest.main()
    #unittest.TextTestRunner(verbosity=2).run(suite())

Unit testing with doctest

Resources:
doctest documentation:http://docs.python.org/lib/module-doctest.html
epydoc home page: http://epydoc.sourceforge.net/

def test_ascending_sort():
    """
    >>> a = [5, 2, 3, 1, 4]
    >>> a.sort()
    >>> a
    [1, 2, 3, 4, 5]
    """
if __name__ == "__main__":
    import doctest
    doctest.testmod()

Unit testing with py.test

Resources:
py.test home page: http://codespeak.net/py/current/doc/test.html
py lib home page: http://codespeak.net/py/current/doc/why_py.html

class TestSort:
    def setup_method(self, method):
        self.alist = [5, 2, 3, 1, 4]

    def test_ascending_sort(self):
        self.alist.sort()
        assert self.alist == [1, 2, 3, 4, 5]

# py.test -v test_sort.py
inserting into sys.path: /usr/local/dist-py
============================= test process starts =============================
testing-mode: inprocess
executable  : /usr/local/bin/python  (2.4.0-final-0)
using py lib: /usr/local/dist-py/py
initial testconfig 0: /usr/local/dist-py/py/test/defaultconfig.py/.
===============================================================================
0.000 ok test_sort.py:5       TestSort.test_ascending_sort()
0.000 ok test_sort.py:9       TestSort.test_custom_sort()
0.000 ok test_sort.py:23      TestSort.test_sort_reverse()
0.007 ok test_sort.py:28      TestSort.test_sort_exception()

Agile development tools

Resources:
PyLint: http://www.logilab.org/projects/pylint
coverage: http://www.nedbatchelder.com/code/modules/coverage.html
buildbot: http://buildbot.sourceforge.net
trac: http://edgewall.com/trac/

Homegrown test automation framework


    def main_loop(self):
        self.test = Test.Test('EXECUTABLE.' + self.__class__._name_)
        try:
            # start timer
            self.timer.start("total")
          
            # generate data
            if self.test.config.isset("generate"):
                self.generate_data()
                if self.test.config.isset("norun"):
                    self.timer.stop("total")
                    return
             
            # backup
            self.backup()

            # restore
            self.restore()

            # validate
            self.restore_validate()

            # end timer
            self.timer.stop("total")

            self.test.add_memo("test took times of: %s" % self.timer.timestr())
        except:
            # If we have an uncaught exception here we should log this test as INVALID
(exceptiontype, exceptionvalue, exceptiontrace) = sys.exc_info()
            msg = "Uncaught exception %s in basic::main_loop" % str(exceptiontype)
            self.test.test({"message": msg})
            self.test.set_passfail('INVALID')
            self.test.log_result()

Homegrown test automation framework

STAF/STAX

Sample command sent to STAX Service machine:

STAF mgmt1 STAX 
EXECUTE FILE /QA/STAF/stax_jobs/client_test_harness.xml 
MACHINE mgmt1 
SCRIPTFILE /QA/STAF/stax_jobs/global_vars.py 
JOBNAME "CLIENT_TEST_HARNESS" 
SCRIPT "VERSION='1.0.2'" 
CLEARLOGS Enabled
Sample STAX job file:

<script>
  HFSADDR = 'dpe03'
  HARNESS_TIMER_DURATION = '60m'
  
  clients_os = {'neon':'unix', 
  				'sunv2401':'unix',
  				'ibmp6151':'unix',
  				'copper':'win',
  				'dellwin2ks2':'win'
  				}
  				
  pylts_path = {'unix': '/data01/qa/pylts',
  			    'win' : 'C:/qa/pylts'
  			    }
  			    
  tests_unix = [ 
  [ 'unix_perms', 'avtar/snapup_unix_perms.py' ],
  [ 'commonality', 'avtar/snapup_commonality.py' ],
  [ 'weird_names', 'avtar/snapup_weird_names.py' ],
  ]
  
  tests_win = [ 
  [ 'srv_unicode_names', 'avtar/srv_unicode_names.py' ],
  ]
</script>

<defaultcall function="Main"/>

<function name="Main">
  <sequence>
    <import machine="'neon'" file="'/vortex/pub/QA/STAF/stax_jobs/log_result.xml'"/>
    <call function="'ClientTestHarness'">
      [clients_os, pylts_path, tests_unix, tests_win]
    </call>
  </sequence>
</function>

<function name="ClientTestHarness">
  <function-list-args>
    <function-required-arg name='clients_os'/>
    <function-required-arg name='pylts_path'/>
    <function-required-arg name='tests_unix'/>
    <function-required-arg name='tests_win'/>
    <function-other-args name='args'/>
  </function-list-args>
  <paralleliterate var="machine" in="clients_os.keys()">
    <sequence>
    <script>
    os_type = clients_os[machine]
    tests = {}
    if os_type == 'unix':
    	tests = tests_unix
    if os_type == 'win':
    	tests = tests_win
    </script>
    <iterate var="test" in="tests">
      <sequence>
        <script>
        test_name = machine + "_" + test[0]
		</script>
        <testcase name="test_name">
          <sequence>
            <script>
            cmdline = pylts_path[os_type] + "/" + test[1] + " --hfsaddr=%s" % HFSADDR
            </script>
			<timer duration = "HARNESS_TIMER_DURATION">
				<process>
				  <location>machine</location>
				  <command>'python'</command>
				  <parms>cmdline</parms>
				  <stderr mode="'stdout'" />
				  <returnstdout />
				</process>
			</timer>
            <call function="'LogResult'">machine</call>
          </sequence>
        </testcase>
      </sequence>
    </iterate>
    </sequence>
  </paralleliterate>
</function>
</stax>

Automated regression test scenario

Acceptance testing with FitNesse/PyFIT

Resources:
FitNesse home page: http://www.fitnesse.org
FitNesse fixtures: http://fitnesse.org/FitNesse.WritingAcceptanceTests
PyFIT available in the files section of http://groups.yahoo.com/group/fitnesse

FitNesse/PyFIT: ColumnFixture tables

Resources:
FitNesse ColumnFixture: http://fitnesse.org/FitNesse.ColumnFixture

FitNesse/PyFIT: RowFixture tables

Resources:
FitNesse RowFixture: http://fitnesse.org/FitNesse.RowFixture

FitNesse/PyFIT: Sample test run

FitNesse/PyFIT: Other types of fixtures

Web application testing

Resources:
Twill: http://darcs.idyll.org/~t/projects/twill/README.html
mechanize: http://wwwsearch.sourceforge.net/mechanize
webunit: http://mechanicalcat.net/tech/webunit
Selenium: http://selenium.thoughtworks.com/index.html
Pamie: http://pamie.sourceforge.net
Jssh: http://www.croczilla.com/jssh
Watir: http://rubyforge.org/projects/wtr

Browser simulator: twill

Resources:
Titus Brown's twill home page: http://www.idyll.org/~t/www-tools/twill/
Wikipedia RSS feed based on twill: http://www.deheus.net/petrik/blog/post/68/

Browser automation: Selenium

Resources:
Selenium home page: http://openqa.org/selenium/
Selenium Recorder: http://seleniumrecorder.mozdev.org/

Python GUI test tools

Resources:
pyGUIUnit: http://pyguiunit.sourceforge.net
dogtail: http://people.redhat.com/zcerza/dogtail/
Marathon: http://marathonman.sourceforge.net

Jython in the testing world

Resources:
Tim Bray post on Dynamic Java: http://www.tbray.org/ongoing/When/200x/2004/12/08/DynamicJava
The Grinder v3: http://grinder.sourceforge.net/g3/whats-new.html
TestMaker: http://pushtotest.com/Downloads/downloadtmdoc.html
HttpUnit: http://httpunit.sourceforge.net/index.html

Conclusions

Resources:
Bret Pettichord's "Homebrew test automation" class notes:
http://www.io.com/%7Ewazmo/papers/homebrew_test_automation_200409.pdf
Danny Faught and James Bach: "Not your father's test automation"
(go to http://www.stickyminds.com in the Articles section)

Conclusions (cont.)