CNews - November 28, 2011
Outsourcing of software development services is small wonder nowadays. This practice is widely employed in organizations of various business dimensions. On the contrary, software testing is a service that companies prefer to leave in-house and often do not trust to third parties the quality assurance of their information systems. What prevents organizations from outsourcing software testing more extensively? How to build effective relationships with a software vendor and assess its quality? Will the new model of long-term outsourcing – establishing of a Dedicated Testing Center – be able to shake traditional patterns? On these and other acute issues dwells Artak Oganesyan, Business Development Director at EPAM Systems, in an interview to CNews.
CNews: How in-demand is outsourcing of software testing services nowadays? What is a typical profile of a customer of such services?
Artak Oganesyan: High-quality testing is a kind of service that requires significant investments. You need to find on the market or nurture yourself good professionals, constantly invest in their education and training, into the deepening of their expertise, buy expensive hardware and software to create test environment.
Considerable costs are also required for the organization of workflows, integration of corresponding methodologies. So the typical customer would be a mature company, which pays careful attention to the quality and reliability of its information systems, but at the same time doesn’t want to bear the costs of building a comprehensive IT organization within its enterprise. The amount of such customers is constantly growing and the demand for outsourcing software testing services is on the rise. This service is largely popular in the financial, insurance, and telecom industries, where there exists a critical dependence of business on a range of information systems and where there’s a need in constant development and deployment of new applications.
CNews: In this or that way many Russian software companies fall under this description: there’s a range of software products and a continuous flow of tasks on their development, high quality and reliability are also of high priority. Do these companies show much interest in outsourcing software testing?
Artak Oganesyan: In terms of outsourced testing, software Russia is very conservative - Russian vendors are still largely cautious of outsourcing software services (unlike their Western counterparts). Firstly, many still think that it’s better to hire and keep in-house personnel than to engage an external team. Why is such approach not always appropriate? Mainly because you have to learn on your own mistakes, you do not have a possibility to use experience of other companies. For example, EPAM Systems accumulates all its testing expertise (methods, technologies, case studies) in a special organizational department – EPAM QA & Testing Competency Center.
Each project team has access to its resources. At the stage of development of a testing strategy and program or during the QA process itself, our specialists rely on the knowledge and experience that’s been accumulated on previous projects for our clients in various industry domains, including global industry leaders. Engaging such experts, who have experience of dozens of completed similar projects, for targeted consulting or building of a test infrastructure allows the company to mitigate risks, optimize costs and release dates. Moreover, apart from access to additional expertise, an outsourcer ensures independent audit and quality assurance of in-house development projects – for many companies it’s a weighty argument.
Secondly, for quite a long time there’s been thriving piracy in Russia and software companies righteously tried to protect their know-hows. Up to now this largely affects our attitude to outsourcing, although leading software vendors provide complete protection of commercial information, starting from undertaking legal confidentiality obligations to organizing for clients closed secure perimeters. By the way, our customer base includes such companies as Oracle, SAP, and Microsoft – three grandees which are in reality direct competitors. During all the years of our cooperation, none of them has ever voiced a doubt in EPAM’s ability to guarantee secrecy of their newest products.
None the less, despite all doubts, the number of companies which outsource part of their testing effort to third party vendors is gradually rising. We judge from our own experience. A couple of years ago, projects for Russian companies were rather scarce, but now EPAM closely works with a number of such software vendors. These companies have own development centers in Russia, but they expand them though our testing labs. I’ll give an example: for one of our such customers we’ve performed functional testing of a product line, performed compatibility testing with hardware and software platforms of various configurations, implemented localization testing for more than thirty countries and tested special versions for a dozen of large end clients.
CNews: Are there any limitations to outsourcing software testing? Which services can be outsourced to third party vendors and which not?
Artak Oganesyan: You can outsource anything. Although certain limitations do exist. It depends. The most complicated areas are large-scale load testing and stress testing. In this case you often need to build a so called test bed, which is almost identical real production environment, because simulation of work on a smaller model won’t bring the required results. Building a test infrastructure, which corresponds to a production one, involves considerable financial and labor costs. Just imagine a system of a mobile operator where hundreds of millions of records from dozens of millions of subscribers have to be processed daily, or a range of systems of a retail bank servicing millions of transactions of thousands of its branches and dozens of thousands of terminals. The work of such sockdolagers is ensured by powerful data centers, for the creation of which you’d have to suffer hefty expenses. Plus you need to build a reserve center. And when you touch upon similar investments for the creation of the test environment, financial directors and business owners lose heart and do not understand why do this for a third party vendor. Some our customers provide their own testing facilities. Alternatively, we can use virtual environments and partner test beds, for example IBM and HP give access to their laboratories. Never the less, inability to create a deep copy of production environment within an in-house test bed appears as a serious challenge for many enterprises, which is hard to avoid on some projects.
CNews: Which outsourcing models best suit client-vendor relationships in outsourcing: fixed price, time and material, or a dedicated center?
Artak Oganesyan: It depends. Single projects are most often T&M based. But when you have a well-defined project scope (e.g. execute a number of tests within a certain period of time), you can also use a fixed price model.
But quite often a customer says: “Every three months we release new builds, and every month – small patches. So once a month you’ll provide a small team for testing the updates, and quarterly – a bigger team for testing the new build.” For such projects, fixed price or T&M is not the best choice. During such breaks between build releases and patches, the vendor may engage the team on other projects and the team might not be available any more for the next testing stage for this client. As a result, the customer won’t get the team which has been testing previous builds and perfectly understands the specifics of use.
In such cases we recommend to create a Dedicated Testing Center. In this case a steady team is built especially for this customer for a long period of time. It is the core of the testing group: their tasks include preparation and update of test cases, stable functioning of the test environment, assistance to the support team of Level 2 and 3. The most important thing here is that these specialists ensure continuous accumulation of experience and expertise, help to set up interaction processes. Their services are paid on a monthly basis. When a new build or patch is released and the volume of testing significantly increases, the team squad also expands. But since the communication processes and workflows are already well-functioning, we have full understanding of the specifics of the customer business and its certain systems, have up-to-day test scenarios, so the testing is performed within a shorter period of time and with higher quality. When the “peak season” is over, the team shrinks to its core members. In this case, testing of separate builds and patches are regarded as separate projects which are paid on a fixed price or time and material basis above the monthly payments.
CNews: Is a dedicated team usually located onsite or offsite?
Artak Oganesyan: Both variants are possible. Most customers in our country have headquarters in our capitals – Moscow and St. Petersburg, where IT salaries are pretty high, rental payment are also way higher, as well as more expensive are other things like building the infrastructure and so on. So to hire a testing team in the customer’s central office does not make much sense. In order to optimize costs, it is a lot better to create a dedicated testing center in some regional area, where direct and indirect expenses are lower than that in the capital. With the same level of qualification and experience, the employees there have 20-30% lower salary than in Moscow, but are more loyal and are not so prone to change employers quickly.
In our practice we have examples when our dedicated testing centers could fit well into the infrastructure of a customer’s regional office. One of them is our testing center for Citi in Russia and the CIS, built on the basis of its branch office in Ryazan. Now there’re engaged approximately 40-50 testing engineers who continuously perform functional, load, and integration testing of the bank’s systems, as well as are involved on other projects like acceptance testing of other IT solutions developed for the customer by third party IT companies.
But often we perform testing services remotely. In such cases we agree on the legal, organizational and process conditions which suit both us and the customer. Such cooperation is safe and mutually beneficial, which means successful.
Sometimes we come across funny situations: part of the team is located at the customer’s central office, and part – at EPAM’s regional office.
CNews: What risks do customers see in outsourcing testing services?
Artak Oganesyan: The major risk for the customer is to become dependent on the outsourcer. By delegating all or part of the testing effort to an external vendor, the company also entrust to it the safely and reliability of its business. The customer may have a fear that the vendor would blackmail it by constantly raising the service fee. But this is not the case with long-term outsourcing. Such strategic cooperation is secured by firm legal and financial obligations of both parties, and the vendor is no less dependent on the customer. If the company withdraws from the cooperation, it has to promptly look for other variants where to settle the team, what to do with the created infrastructure of the dedicated center, etc. These are direct financial losses. Mutual dependency is exactly the thing that allows to build winning partner relationships and search for compromises that fit both parties.
Another potential risk is poor quality testing. With all due diligence, responsibility, experience, and availability of best testing tools, a testing engineer may overlook a bug which can later trigger errors or lead to a breakdown of some major system. Most often it happens due to lack of understanding of the customer’s business specifics and lack of necessary experience. A testing engineer checks: "Does the form open? - Yes. All necessary fields are present? – Yes. Can you fill in the field? - You can. Can you click the buttons? – You can. Excellent, all done". And the fact that the systems allows to enter in one of the fields the data that are meaningless or dangerous from the business standpoint eludes detection. Of course I exaggerate, but the point is that testing engineers should clearly understand what they test. Therefore there should be constant contact and cooperation between the testing teams and the teams of analytics and developers; maybe it’s also worth to involve them into the system maintenance and support, so that every team member could hear users crying and understand the consequences of not detected bugs. And long-term outsourcing gives such a possibility.
CNews: Most companies, in the first place banks and other financial organizations, are very concerned with data security. The risk of data leaks and lack of confidence in vendor’s ability to guarantee proper security are often named as compelling arguments against outsourcing. Are these reasonable concerns?
Artak Oganesyan: Elaboration and coordination of security policies is one of the key issues in the process of creation of a dedicated testing center. If it is based on the customer’s infrastructure, everything is pretty simple: the team must comply with all the regulations and restrictions that exist within the company. The vendor’s specialists use all safeguards that are also used by all the customer’s employees.
If a dedicated center is located on the outsourcer’s premises, a special closed perimeter is built for the sake of data protection. This process involves a range of organizational, technical, personnel, and legal efforts that fully guarantee data protection and rule out data leaks. Certification to ISAE 3000 Type 2 (SAS70 Type II) or ISO/IEC 27001:2005 security standards gives an additional guarantee to the customer.
Special emphasis is placed upon the protection of personal data of employees and the customer’s clients, as well as financial transactions that are stored or processed in the information systems. As a rule, the outsourcer doesn’t work with the client’s live data. For example, it can use a database that is automatically generated and is filled in half-manually by the outsourcer which simulates functioning of one of the customer’s business directions (e.g. crediting). Never the less, it is essential to foresee the situations when the outsourcer’s specialists do have to work with real systems and therefore build the infrastructure and processes in such a way so that it completely excludes direct access to personal data. One of the ways is to use special algorithms for data processing: in this case the developer will get the information which is modified in a random and irreversible way. For example, obfuscated names and personal data of the bank’s clients. Or mix numeric data in the transaction amount fields retaining their logic connections, so that the developer could not assess the bank’s business volume or its clients.
Moreover, the outsourcer provides a guarantee that the team won’t participate on projects for the customer’s direct competitors, sometimes even within a certain period of time after the project delivery or end of cooperation (1-3 years). It is also important to protect information confidentiality in connection with the algorithms and logic of the systems under test. For example, if the specialist tests the bank's scoring system, he/she therefore knows how it works and potentially knows how to bypass it. Such issues are solved by signing confidentiality agreements with the dedicated team.
CNews: How do you balance functions between the customer’s IT service and the dedicated testing center?
Artak Oganesyan: A lot depends on what “heritage” the customer already has by the moment when it decides to outsource. If the company’s IT service already has strong managers and QA and testing engineers on board, it can undertake such functions as the development of the overall strategy, testing plan and scenarios, and leave routine tasks to the outsourcer.
If the outsourcer bring in own expertise (e.g. the customer has no experience in automated functional testing or building development and testing processes in continuous integration), it usually takes upon the initiative and allocates tasks. There can also be such variants when third party specialists are first given only single tasks, but gradually receive more and more management and production functions.
There exist all possible capabilities for communication between the customer and the dedicated team – from video conferences, skype and other communication tools to regular on-site meetings and business trips.
I’d also like to bring to your notice another important issue. You have to clearly distinguish between the development and testing teams, because their interests often may and must contradict to each other. For a developer it is important to build the solution functionality as quickly as possible and to deliver it. A testing engineer has to thoroughly check everything and make sure that there exist no bugs, problems or inconsistencies with the initial requirements. When the deadlines are tight, project managers frequently start to put pressure on the testing teams. Sometimes the testers themselves start “taking close to heart” the situation and become tolerant to some system weaknesses. But this may lead to serious problems with the quality of the end product. So it is very important to draw the line between their tasks and give testers the power to veto the final delivery or at least to give the right to escalate some issues in order to inform the management of possible risks. This will help to split the balance more effectively in terms of the project’s “deadlines – budget – quality”.
CNews: You surely have a set of own testing processes, methods, and tools, but the customer may have individual preferences. Do you usually recommend your approach or adjust to the existing one?
Artak Oganesyan: Both is possible. EPAM often provides own testing methods and toolsets: we choose them relying on our experience and the customer’s requirements, test tasks, peculiarities of a certain solution or infrastructure. For example, for automation of load testing we can use both commercial and open source products. The decisive point here is how well this or that solution can solve the existing tasks. Commercial solutions are more easy-to-use and have better performance. Moreover, commercial vendors usually provide full user support. Free tools are usually not so comprehensive as commercial ones, but on some projects their capabilities may be enough to resolve all tasks at full breath. The choice of this or that system rests on the analysis of the customer’s tasks: from all available tools you choose the one that can solve them most effectively.
Sometimes the customer proposes own variants, if for example it has already bought some standard packets from HP, IBM or other vendors, and we have to use them. If the customer’s IT department already has rigid established testing methods, then it is reasonable to adjust our processes to them.
CNews: How do you assess the quality of work of the dedicated testing center?
Artak Oganesyan: As a rule, such assessment is based on a special system of metrics. For this we have to document all accomplished actions, the number of revealed and removed bugs, open and resolved issues, and other statistics at all testing stages – in-house testing by the development and testing teams, acceptance testing, user testing, beta and alpha testing. These data are entered into the project management system (at EPAM, for example, we’ve developed a proprietary EPAM Project Management Center – EPAM PMC) and are then analyzed. Ideally, this process involves steady reduction of the number of bugs at each testing stage and their complete absence at the production release. Then it means that it was high-quality work. If we notice, that at the stage of in-house testing there were few bugs and at the stage of acceptance testing there weren’t any at all, but at the same time users elicit defects, it means that it is high time to take measures and thoroughly analyze why it is happening. Probably you’ve taken a formal approach to acceptance, and the project team hasn’t paid enough attention to testing. Sometimes you just objectively cannot prognosticate all bugs that may become obvious in daily use – tests cannot simulate all situations that arise in real life. Then the elicited bug is included into test cases and the next time it won’t be omitted. The worst case is when a bug could have been detected but it hasn’t been done.
However, the assessment of the quality of testing according to such metrics does not always work, if the system is developed for a completely new business direction. The customer does not completely know what exactly it needs and therefore cannot clearly formulate all requirements to the IT solution. You have to be ready that bugs may emerge at final testing stages and this will be a sign of shortcomings at the stages of analytics and design, and not poor work of the testing team.
CNews: How does such system of metrics may help estimate testing costs?
Artak Oganesyan: To the full extent. You just need to trace labor costs and match them against salary rates. This will allow not only to estimate costs of each testing stage and type, but will also make it possible to understand how to optimize the whole testing process. For example, you can hire a hundred ordinary testers and manually test all possible variants and patterns of the system use, e.g. matrix of the server and client platforms, versions of all operating systems and browsers, etc. Alternatively, you can build the team of a dozen experts in automation testing whose qualification (and accordingly the cost) will be higher: they will build the test environment, select or write automation tests, and will organize the whole testing process in the most optimal way. Then in future you’ll need not a hundred people, but only 10-20 specialists who will only have to test the part of the functionality, which is not yet possible to automate or is too time-consuming. This will allow you to reduce testing costs later (e.g. when the new system build is released).
From the point of view of testing costs, there’s also another “hidden rock”. Many customers insist to tie the quality metrics to penalties. If problems are detected at the stage of acceptance testing by users, the outsourcer’s service fees are paid in full. If errors appear on production, the testing team shall be fined for a certain amount depending on how critical was the issue. Generally, such approach may be justified, but it should be applied reasonably and carefully. The outsourcer, who becomes scary of multiple sanctions, starts to play safe at some stage and will try to test all and everything. As a result, the testing costs will rise manifold. The release of a new build will have to be delayed at a period that may not suit the customer and its business. So you need to carefully weigh all factors, and probably motivate your outsourcer to keep the deadlines, even if the probability is high that some minor bugs may remain in the system. They are not so critical and their elimination on production may appear substantially cheaper for the customer.
In this respect, organization of a dedicated center is effective in a way that it allows to monitor the quality and cost of the testing services over a long period of time. It finally helps to achieve such a level of service, which will suit both the customer and the outsourcer. Lasting and continuous cooperation allows to accumulate expertise, do not start every project from scratch, and build clear and transparent software testing processes. What other outsourcing model may bring such advantages?
Original publication is here.