In my last entry I gave an overview of how we test the core component for the Visual COBOL product. In this entry I will cover how we test the Visual Studio and Eclipse integration. Testing of the Visual Studio IDE is kicked off using an in house tool called ‘TestCat’. This tool simply monitors for the availability of a batch script, which is created during the overnight build of the product. The TestCat tool runs the script, which has a number of commands contained – it starts Hyper-V machine instances, copies the build onto the virtual machine and kicks off the IDE tests after installing the test build. Tests cover the basic functions of the IDE, but also more advanced testing such as various project and debugging scenarios. The results are then assembled in HTML reports and published. The Eclipse IDE is tested in a very similar manner. The tests are kicked off overnight, and test failures are reported through a web page and via email notifications.  This process runs several thousand tests on the IDEs and confirms Visual Studio and Eclipse integration is working correctly.

There are also other components that are extensions to the core and IDE. Some are also a part of the IDE, such as SQL support. These components are updated and tested in a similar way by specialists in those areas. Once the development teams have finished their testing the product is delivered to our build and release infrastructure where further checks are made before releasing the product.