Distributed Unit Testing: supporting multiple platforms accurately and efficiently

Publicado por

Colecciones : DIA. Artículos del Departamento de Informática y AutomáticaIUCE. Artículos del Instituto Universitario de Ciencias de la Educación GRIAL. Artículos del Grupo de Investigación en Interacción y E-learning
Fecha de publicación : nov-2006
Testing is the most basic way to check the evolution of a software project. A strong test suite helps development team assuring that a certain minimum is always kept. Release after release software must pass a test group, large enough to check, at least, the most common use cases. Results will numerically show project s progress. Depending on the project testing could be done on different platforms under various operating systems. Forcing race conditions and heavy load is also a desired feature for the test suite.
Publicado el : miércoles, 01 de noviembre de 2006
Lectura(s) : 47
Fuente : Gredos de la universidad de salamenca
Licencia: Más información
Atribución, No Comercial, Compartir bajo la misma forma idéntica
Número de páginas: 5
Ver más Ver menos
Core Technology
by Pablo Santos and Francisco J. Garcia
DistributedUnitTestingSupporting multiple platforms accuratelyand efficientlyAt Codice Software we design and develop software configuration management tools that runon various combinations of operating systems and hardware platforms. For instance, both serv-er and clients run on Windows XP/2000/2003/Vista, Linux, and Mac OS X. And since our soft-ware uses .NET, it runs on native Microsoft implementation for Windows and Mono for UNIX-like operating systems.There’s only one way to accurately the testing process. The key to our distributed testing solution isPCaobdlicoe  iSs oaf tSwoaftrew. aHree  cEangibneeer atand efficiently support platform com- PNUnit, short for “Parallel NUnit.” PNUnit is a modified version ofreached at nbinations such as these—testing, and the familiar NUnit (www.nunit.org) testing framework that waspsantosl@codicesoftware.com.lots of it. However, running a software originally ported from JUnit to .NET. The source code and relatedFranciscois a Professor ofpackage on 36 platform combina- files for PNUnit are available electronically; see “Resource Center,”CUonimvperustietry  Socf iSeanlcae mata tnhcea. Youtions, release after release, is a page 5.can reach him atHerculean effort. The solution we We were already using NUnit to develop unit tests because ourfgarcia@usal.esadopted was to distribute testing tasks development is .NET based. NUnit lets you write unit tests with allby running them in parallel. In addi- .NET languages, and even adhere to test-driven development princi-tion to uncovering bugs, this ples. Still, our primary concern was testing on multiple platformsapproach significantly speeded up using distributed test scenarios. Unfortunately, stock NUnit doesn’tsupport this, hence our decision to extend NUnit to support distrib-uted testing.One of the reasons we wanted to stick with the NUnit framework isthat we were familiar with its environment. Usually when you move toa new testing platform, the first thing you have to do is learn a newscripting language. By extending NUnit, we could use the same pro-gramming language and constructions (test suites, fixtures, and thelike) that we were used to with regular unit testing.Developing Automated Tests with PNUnitFigure 1 shows the basic structure and main components of thePNUnit system. Launcher is the program responsible for launchingtest suites on test machines. It is called by a test configuration fileas an argument, reads the file, and sends instructions to the testingmachines. It then gathers test results and prints them on thescreen. The test configuration (testconf) file is written in XML andcontains a definition of the test scenario being created. It defines
Agent.exeFigure 1:PNUnit high-level structure.28Dr. Dobb’s Journallwww.ddj.comlNovember 2006
<TestGroup> <ParallelTests>    <ParallelTest>Each ParallelTest defines <Name>BasicLabelling</Name>tests to run in parallel. <Tests>Different ParallelTests <TestConf>groups run sequentially. <Name>Server</Name> <Assembly>cmtest.dll</Assembly> <TestToRun>cmtest.server.Run</TestToRun> <Machine>linuxbox00:8080</Machine> <TestParams> <string>../server</string> <!-- server path -->  </TestParams> </TestConf>     <TestConf> <Name>Client00</Name> <Assembly>cmtest.dll</Assembly>aTso sdeemfibnley  aw theesrte  wthe es tpeesct iifys the <TestToRun>cmtest.LoadTest.SimpleFileGet</TestToRun>defined, the test fixture and the <Machine>winbox00:8080</Machine>machine in which it will be run. <TestParams>Then each test can have params, <string>.\wkspaces</string> <!-- client path -->specific of each kind of test. <string>linuxbox00:8084</string> <!-- server name --> </TestParams> </TestConf> <TestConf> <Name>Client01</Name> <Assembly>cmtest.dll</Assembly> <TestToRun>cmtest.LoadTest.SimpleFileGet</TestToRun> <Machine>winbox01:8080</Machine> <TestParams> <string>.\wkspaces</string> <!-- workspace dir -->  <string>linuxbox00:8084</string> <!-- server name --> </TestParams> </TestConf> </Tests> </ParallelTest> </ParallelTests></TestGroup>Figure 2:Test script.the appropriate assemblies, specifies where they have to run, and Each time PNUnit receives a call in theIPNUnitAgent::RunTestidentifies the data to execute. Figure 2, a typical testconf file, con- method (meaning that a new test needs to be launched), it creates atains at least one parallel test section (test group). Each test group new instance ofPNUnitTestRunner. In NUnit parlance,runs sequentially. Several tests are defined inside the test groupPNUnitTestRunneris a real test runner. It creates aTestDomainand identified with TestConf labels. Each test inside the group instance, runs the test specified by the remote Launcher, collects theruns in parallel, so each test group defines a parallel scenario. results, and notifies the remote Runner. Each test resides on regularThe TestConf section tells Launcher in which assembly the assemblies, just as if they were normal NUnit tests.test resides, which method to invoke, and where the machine is Where does the NUnit framework fit in? It is the key layer into launch it. Launcher uses this information to communicate whichPNUnitTestRunnerresides. We take advantage of all ofwith the Agent. To handle communications, Launcher creates aTestConf eldMeaning      Runner for each test defined in a test group. The Runner is theSection Ficomponent that calls the Agent methods. More importantly,Runner handles synchronization via theIPNUnitServicesinter-Name Test name you want displayed on the resultsface that it exports to the tests. Runner also gathers test resultsscreen.and informs Launcher about them.Assembly .NET Assembly in which the test resides.Agents are located on test machines. Each test machine needsto have at least one Agent waiting to run tests. Agent is a smallTestToRun Method name that defines the test to be run..NET application that, once started, registers a remote interfaceMachine Machine where you want the test executed.calledIPNUnitAgent. This is the interface that Runner uses tolaunch the tests.PNUnitAgentis the actual component inside agentsTestParams Test-specific parameters.that handles the various processes.Table 1:TestConf members.November 2006lwww.ddj.comlDr. Dobb’s Journal29
Dr. Dobb’s Journallwww.ddj.comlNovember 2006
NUnit’s functionality to load/run tests andcollect results.
Synchronization FacilitiesRunner is responsible for providing syn-chronization mechanisms for the tests.Example 1 presents the methods in theIPNUnitServicesinterface. For now, it sim-ply provides a barrier-based synchroniza-tion mechanism.You can initialize a barrier using Testby defining the maximum number of ele-ments that pass through the barrier, orTest can just let Runner handle it. In thelatter case, Runner assumes that all testspass through the barrier, so it onlydepends on the definition made in theXML file. However, the former approachis useful when you need to define syn-chronization between a few participantsin the test.As Figure 3 illustrates, the barrier mecha-nism is a basic synchronization primitive.All tests must pass through a barrier beforeany of them are allowed to pass.
test00 test01 test02
barrierStep 1:Only test00 is hitting the barrier
test00 test01 test02
Step 2:All clients hit the barrier,they can pass
barriertest00 test01
Step 3:Tests passing through the barrier
Figure 3:Barrier mechanism in action.
Listing One, the barrier implementa- our system. Still, it would be easy totion, shows that there are two important include some other primitives—sema-methods plus a constructor for the phores, for instance.Barrierclass. The constructor initializesthe number of elements that must passWriting the Teststhrough the barrier to release it. The At first glance, a PNUnit test doesn’t appearEnter()method is invoked when a test that different from standard NUnit tests.enters a barrier. If it doesn’t enter a bar- All the facilities available for a normalrier and all of the other involved tests NUnit test (assertions and the like) are alsohave gone through it, then the thread available.goes to sleep using a Monitor. The sec- Listing Two presents a couple of PNUnitond method,Abandon(),is used to tests.release barriers under fail conditions. The differences between a PNUnit andFor instance, if a test fails before hitting regular NUnit test involve the functionalitiesa barrier,Abandon()is called, so that an present in thePNUnit.Frameworkpackage.erroneous test won’t freeze the rest of It includes a class calledPNUnitServicesthe tests. through which the extended functionalityWith theBarriermethod, we have can be accessed.been able to implement large test suites, ThePNUnitServicesclass providescovering all the basic core functionality of methods for initializing a barrier,public interface IPNUnitServices{void NotifyResult(string TestName,PNUnitTestResult result);void InitBarrier(string TestName, string barrier);void InitBarrier(string TestName, string barrier, int Max);void EnterBarrier(string barrier);}Example 1:IPNUnitServicesinterface.Listing Onepublic class Barrier{private int mCount;private int mMaxCount;private Object mLock = new Object();public Barrier(int maxCount){mCount = 0;mMaxCount = maxCount;}public void Enter(){lock( mLock ){++mCount;if( mCount >= mMaxCount ){mCount = 0;Monitor.PulseAll(mLock);}elseMonitor.Wait(mLock);}}public void Abandon(){lock( mLock ){mMaxCount;if( mCount >= mMaxCount ){mCount = 0;Monitor.PulseAll(mLock);}}}}
November 2006lwww.ddj.comlDr. Dobb’s Journal31
entering it as previously described, and the interface received from the remote tialization, making the second one wait for itretrieving the test name and test parame- Runner must be made available to the in theEnterBarrierstatement. With onlyters. Why doesn’t the test access the running test. This is done with theCreate-these primitives, you can implement com-IPNUnitServicesinterface directly? Be-InstanceAndUnwrapAPI. plex scenarios and distribute the tests overcause of an implementation problem: So the tests listed in Listing Two simply the network.Since the test runs on a different applica- create a barrier, then use it to synchronize Listing Three is a typical configurationtion domain than thePNUnitTestRunner, each other. The first barrier waits after ini- file. The script specifies both the methodswhere tests are implemented and thePNUnit is a modified version of the familiar NUnitmachines to execute them. Of course, thetesting frameworktest machines must have an Agent startedand listening in the correct port number—8080 by default—and the assemblies mustbe deployed, too.L TwouissitinnggSystem;Conclusionusing NUnit.Framework;Over the past few months, we’ve been usingusing PNUnit.Framework;PNUnit for three types of tests—smoke, load,namespace SimpleTestand merging tests. Smoke tests are run every{    [TestFixture]time a developer finishes a task, togetherp{ublic class Testwith regular NUnit test suites. Smoke tests[Test]basically cover core product functionality,{public void FirstTest()and have been extended to include morestrinPgN[U]n itteSsetrPvaircaemss. G=e t().GetTestParams();and more test cases.PNUnitServices.Get().InitBarrier(BARRIER);Each time we create a new release (typi-S/y/s tweami.tT htrweoa dsiencgo.nTdhsread.Sleep(2000);cally once a week), the “integrator” (one ofPNUnisttSreirnvgi.cFeosr.mGaett(().WriteLine(our team members) merges all the tasks toteFsitrPsatrTaemsst[ 0s]t)a)r;ted with param {0},create a new baseline, then executes all thePNUnitServices.Get().EnterBarrier(BARRIER);               smoke tests. However, he does so by launch-}public void SecondTest()ing them on different platform combinations,{   PNUnitServices.Get().WriteLine(covering all our supported platforms. Merge“Second test will wait for first”);tests (a large test suite covering lots of branch-/P/N UwniiltlS erwvaiicte sf.oGre tt(h)e. IfniirtsBta rtreisetr(BARRIER);                merging scenarios) are also executed duringPPNNUUnniittSSeerrvviicceess..GGeett(())..WErnitteerLBianrer(ier(BARRIER);integration on different platforms. Finally,“First test should be started now”);PNUnit tests are specifically designed to be}scalable. Each test implements simple opera-} }tions so they can be combined on the testconfscript, and easily define load scenariosListing Threethrough the available test machines.<Test<GPraoruapl>lelTests>We’ve also started to use PNUnit on a<Para<lNlaemleT>eSsitm>pleTest</Name>cluster (about 50 Xeon machines), letting us<Tests>test software under heavy load conditions.<Test<CNoanmfe>>FirstTest</Name>The framework has proved to be very useful<T<eAsstsTeomRbulny>>Stiemsptl0e0T.edsltl.<T/eAssts.eFmibrlsyt>Test</TestToRun>on our project, having detected many bugs<Machine>localhost:8080</Machine>over the months.<TestParams><string>Option1</string> There are still several aspects of the frame-</Tes<t/CToensft>Params>                                    work that need improvement. For instance,<Test<CNoanmfe>>SecondTest</Name>tests are currently only run from the command<Assembly>test00.dll</Assembly>line, and results are gathered this way. A huge<<MTaescthTionReu>nt>eSsitmbpolxe:8T0e8s0t<./TMeascth.iSneec>ondTest</TestToRun> step forward would be to include GUI testing</Test<s/>TestConf>         support in the framework. Also, it would be</ParallelTest>useful to be able to record user events and play</ParallelTests></TestGroup>them back as with other tools, but with theadded power of multiplatform support.DDJ32Dr. Dobb’s Journallwww.ddj.comlNovember 2006
¡Sé el primero en escribir un comentario!

13/1000 caracteres como máximo.