Foreword
If you have picked up this white paper and are about to commence, or are in the middle of, a data miigration between two or more systems and system architectures then good luck. If the success rate of application migrations as analysed by the Standish Group is at all accurate you have little more than a 25% chance of being on a successful project. The clear message from across the IT industry is that the problem of application data migration is not local to one or two companies, or even to a particular market vertical, but rather it’s a problem common to all types of enterprise. Here in telcoland, I regularly come across architects who say that they can’t remember a single successful migration.
Take Marvin Whizztech of multi-service operator Convergicoms. He’s a CIO in hot water. A while back, he commenced a major systems migration programme intent on cutting costs by reducing the application count and overhead whilst improving flexibility by installing a new convergent architecture. He got the CFO Boris Beancounter to sign off the project on the basis that it would enable rapid launch of new products and reduce the cash-to-order cycle, thereby maintaining the competitiveness of Convergicoms and fending off competition from cut-price rival Coms-U-Like. It’s now two years down the line and although Marvin has managed to turn off several small applications, all the major systems remain firmly in place, with little sign of significant progress. The first attempt at migrating from some of these systems has already failed; while other migrations are either late starting or are unable to create a ‘beach-head’ on the target applications. For most projects there is no ability to predict any real progress. The consultants are now recommending turning off the billing system for a few weeks to facilitate the migration.
Some of you may be feeling uncomfortable recognition of Marvin’s predicament. That’s not surprising. What is surprising is that, given the scale of these types of problems, so little attention has been given to solving them and to creating products specifically designed to help. Of course, many software tools have been modified or applied to the problem – data warehouse load tools most commonly – but there has been a complete lack of out-and-out application migration focused products to choose from.
Having a specifically developed tool providing technical support for application migration is an important part of the solution, but it is not the entire solution. In addition, business and IT must work together, using practised methods, in order for the tool to deliver. But what’s also clear, from work in application migration methodology (undertaken by major SIs and independent experts), is that method is not enough on its own either. To be successful you need both tool and method.
The upside to the grim statistics quoted earlier, however, is that the expectation of a successful outcome is so low that if you are able to pull it off then you’ll be a hero. So, if you’re one of the 25%, congratulations! You’ve made it. You now have a job for life as the expert on application migrations. At Celona Technologies, our business is assisting telecoms service providers to deliver successful
application migrations. We have developed a third generation data migration tool that has already helped tier 1 service providers deliver projects not on time, but ahead of time. We would like to help you become one of the 25% of winners, rather than one of the also-rans. Find out more about us and what we do from our website www.celona.com or email us at [email protected].
Paul Hollingsworth
Celona Technologies, March 2007
Abstract
The telecoms industry is transforming, with new datacentric networks, a proliferation of services and increasing competition. The need for speed and the inevitability of change are just two of the factors that telecoms service providers have to grapple with. As a result, telecoms companies themselves are transforming – from technology-centric, network operators into customer-centric service providers.
This paper considers the factors driving these changes and what this commercial transformation means for internal business processes and the IT systems that support them. New requirements and a new customer- and service-led paradigm is affecting IT investment patterns within telcos seeking to benefit from the new multi-trillion dollar industry that is currently being created. However, while many large traditional operators are currently overhauling their back-end architectures in order to align themselves to the needs of the new telecoms market, there is a major stumbling block in their quest to reinvent themselves – telecoms data.
While it is generally recognised that telecoms data used well can offer great competitive advantages, in traditional operators valuable data is all too often trapped in multiple data silos, inaccessible and unusable. Unlocking its potential is, however, far from risk free using traditional methods. But the entry into the market of increasing numbers of competitors, who are unhampered by legacy data issues and adept at using data to their advantage, mean that large operators have little choice but to act now.
This paper looks at the three generations of data migration methods, analysing the risks presented by traditional methods and showing how new next-generation data migration approaches can unlock the benefits of data while also mitigating the risks.
The change drivers: why transformation is necessary
The telecoms industry is barely recognisable from the single product, monopolistic, stateowned utility it once was. Today’s dynamic, competitive, multi-product, multi-trillion dollar industry is increasingly mobile and continually changing in response to consumer demand and innovation. As liberalisation continues to spread across the globe and competition hots up, rapid technological change has meant telecoms services are ever more firmly embedded in our lives. These services now underpin both our business and personal interactions, with changes in the telecoms business both mirroring and driving changes in the way we live and in the way we conduct business.
What is becoming increasingly apparent, however, is that we are not simply witnessing an industry transformation – which suggests a change from one state to another – but rather a process of evolution comprising a series of small and large changes of various kinds that stretch from the 1980s, through the exciting state of flux that is the present and on into an unknown future. Telecoms is a rapidly changing, growing and vibrant industry. Amidst all the uncertainty there is only one certainty: change is here to stay.
Change in the telecoms market is being driven by a range of factors, as shown in Figure 1. These factors do not necessarily work in isolation from one another, but are often interdependent. For example, regulators have acted to reduce prices for consumers and to introduce competition; while increased competition has, in part, stimulated innovation. Similarly, globalisation has acted to increase competition and also to stimulate customer mobility.
All of the factors outlined combine to create a market that is constantly evolving. The survivors and beneficiaries of this market will be those companies that can embrace change and respond positively to the challenges thrown up. This is what BT calls the requirement for ‘continuous innovation’. In other words, surviving change involves embracing change.
Business imperatives result in new technical strategies The change drivers that we have outlined result in six main business imperatives for telecoms service providers. These imperatives are the things that telecoms service providers need to do in order to stay in business and prosper, and include:
– need to cut costs and improve revenue management
– the need to innovate and get products to market more quickly
– the need to retain customers
– the need to increase ARPUs
– the need to comply with regulation and legislation
– the need to use business information more effectively (data transparency).
The initial reaction of the industry to change drivers such as competition and investor pressure was to embark on a programme of cost cutting in response to price competition.
However, once the first round of obvious costs were trimmed away, subsequent efficiency gains became more challenging to achieve, so service providers looked, for example, at how they could automate more effectively and streamline systems in order to reduce cost.
However, cost-cutting alone is not enough. In order to achieve business goals such as retaining customers, increasing ARPUs and so on, service providers need to innovate. It is also vital for them to be able to react more quickly to change and competition – which is often described as becoming ‘more agile’. In the new service-based paradigm we are entering, this translates into the ability to launch new services more efficiently and at the right time – and preferably more quickly than competitors.
Although these ideas seem relatively simple on a business level, achieving them at an operational level is far more complex. Legacy systems and architectures, for example, were not designed for this environment and many service providers are therefore struggling to deliver the essential fast time-to-market for new services, and fast provisioning times to customers. At the same time, legacy IT infrastructure is often very expensive for service providers to maintain. In many companies, the vast majority of the IT budget is spent simply maintaining legacy infrastructure rather than on innovating to improve competitiveness. In addition, established service providers do not have the luxury of greenfield projects. They have to continue doing business while working on system transformation projects. They also have large sums and effort invested in back-end solutions and are often not willing to simply throw this away and start again. A further problem is that back-end systems and architectures are often so complex, and so poorly understood, that making significant changes to them is highly risky. One of the thorniest pieces of this complex puzzle, and a significant barrier to change, is the myriad of problems surrounding the data silos that sit beneath existing solutions.
Using business information more effectively is also a vital issue. This is about making data the company holds more accessible and therefore more usable. It is important for compliance issues, but also aids better business analytics to help make the business function better and compete better. However, a consistent and complete view of data is essential to achieve this goal.
A lot of major systems overhauls are already underway
The business imperatives that service providers are seeking to deliver against, have stimulated them to undertake strategic overhauls of their back-end systems and architectures to reduce costs and align them better to new commercial requirements.
Three types of consolidation can be identified:
– New architecture – moving legacy to a new B/OSS stack
– Removing replica applications – reducing the number of instances of the same applications
– Re-platforming – replacing existing platform infrastructure or technologies.
Only the former of these requires both the business and IT department to work together to transform systems, data and process. Examples of rationalisation projects that are currently being undertaken include:
– BT has an ongoing project aimed at reducing supporting systems from several thousand to less than 100, and realising cost savings of euro 150 million by October 2007
– Deutsche Telekom announced further cuts in admin and IT costs in September 2006, aimed at reducing OPEX by euro 5 billion over
four years
– Telstra is also undergoing a massive consolidation of its back-end systems aimed at cutting out more than 1,000 existing systems.
As can be seen from these examples, large established service providers are suffering from system proliferation resulting from years of tactical investment. This means that back-end architectures are often needlessly complicated and expensive to maintain, with key processes fragmented between best-of-breed systems, and multiple systems performing the same function for different services.
However, when service providers contemplate consolidating systems and processes, they are faced with significant data migration challenges, which are complications arising from the requirements to move information from a legacy environment and transform it to fit in the new system environment. They can’t even outsource the problem to make it go away, as this still requires them to deal with fundamental data quality issues for example.
Data may be a competitive weapon, but it’s also a twinedged sword
The dual nature of telecoms data as both an enabler and a disabler, an opportunity and a problem, was highlighted by industry analyst Teresa Cottam in a recent Analysys report.
“Data, not networks, is now the basis of a service provider’s entire business. Without accurate data, service providers cannot bill for services supplied, assure their revenues or services, design or target new services properly, or make sensible investment decisions. The advent of the VNO effectively decouples the network from the business of telecoms provision. Even in markets where this model has yet to be introduced, data is what turns a collection of network equipment into a business. And telecoms service providers are awash with valuable data that has great potential to transform their businesses…[but] while data is the key to unlocking next-generation revenues, operators possess not a single skeleton key, but a bunch of keys of varying sizes, types and ages. Merely buying another new system will not solve their problems.”
Teresa Cottam, World Telecoms BSS and OSS Markets, Analysys Research, 2006
This is the dilemma that many service providers face. They cannot deliver against their business goals unless they solve both their systems and architecture challenges, and one of the biggest barriers to effective systems renewal are the problems surrounding data conversion, management and migration.
These problems are not unique to telecoms. It is estimated that around two-thirds of Fortune 1000/Global 2000 companies are engaged in some form of data conversion project at any given time. These projects include creating new data warehouses and datamarts, migrating data from legacy systems to packaged applications, data consolidation exercises and data quality improvement projects.
So in theory, at least, you would think that with all this practice companies would be highly proficient at data conversion. Well unfortunately that’s all it is – theory and wishful thinking. In practice, the stats on failed projects make dire reading. For example, according to the Standish Group, in 2006 more than half of all technology projects in the US were considered failures, costing businesses between $80-100 billion annually. That figure goes even higher if you add in ‘shelfware’ – that is, projects that are completed but nobody actually needed them or uses them.
Garbage in, garbage out
Analysing failed IT projects reveals that one of the main causes of failure are problems due to source data. Traditional approaches to data migration create nearly as many problems as they resolve. Common problems encountered include data not loading properly, poor data quality and compounded inaccuracies – resulting in ineffective projects, time and cost overruns, and project cancellations.
It is all too common to woefully underestimate at the start of a project the complexity and risk presented by data migration projects in large companies. Business managers often have a very simplified view of IT systems and how they interact, and make misleading assumptions such as that two systems maintaining similar data must be performing similar tasks, therefore mapping data from one to another should be straightforward. This is hardly ever the case and, unfortunately, it often only becomes obvious that it is not the case when the project is already failing.
The Standish Group say in the 2004 Chaos Report that 71% of IT projects fail or overrun. Eight-yfour per cent overrun on time and fifty-six per cent overrun on cost.
Another common fatal error is to make assumptions or overestimates about the true quality of underlying data. Data may be spread across multiple datastores, with problems stemming from the fact that legacy systems might have their own data terminology, data types, metadata, codes and master data. There may be many different ‘flavours’ of the same data and many data elements of the same ‘flavour’. The data might not be in the right format, data quality might be poor or data might even be missing or incomplete.
‘Data model accuracy’ is also a problem. In other words, there might not be a clear understanding of how the business is using individual data fields. In such cases, the business might be using an attribute for something other than that documented (for example, a ‘location’ field might be actually being used for customer segmentation). This causes problems during migration because it means the data will get mapped to the wrong place. This is particularly problematic in the telecoms market because, all too often, how the fields are actually being used is undocumented and with staff turnover over a period of time this can become a substantial problem that is only discovered during the migration itself.
In addition, data profiling tools while good at capturing the technical and statistical analysis of data, are not good at capturing the semantic meaning. This means that while data may be technically and statistically correct, it is semantically wrong. For example a negative tariff might have been used by the business to flag a special case of service. To effectively translate data semantics requires a merger of both business and technical definitions. Either of which might be undocumented.
Data migration tools are becoming more sophisticated
Data migration was originally delivered using manual methods, involving expensive and skilled people. While such an approach may still be valid for very simple environments and datasets, with little impact upon the business, it is not an approach that scales in terms of volume or complexity. It is also, by its nature, prone to human error.
This led to a second generation approach (see Figure 2), which involved building automated tools to reduce manual effort and error. However, bespoke tools often lacked flexibility and were still very expensive and time consuming to build. This led to the development of a market for third-party data migration and integration tools.
This current generation (2.5G) of data migration tools use methods such as ETL (extract-transform-load), often with a data cleanse between the ‘transform’ step and the ‘load’ step. Third generation data migration products use data federation (or EII) to provide real-time integrated views across multiple datastores. The market has, until recently, been highly fragmented with different types of vendors providing different styles of data migration tool or different parts of the data integration puzzle. The market is now consolidating to create more fully functional packages. The problem is that while traditional 2G and 2.5G tools may be very effective at performing bulk data warehouse loads, or database transfers (that is, the sort of migration seen when upgrading from one version of a system to another), they are not as effective at tackling the complex application-level migrations that are characteristic of systems transformation projects currently underway in major tier 1 service providers.
Using traditional tools still creates fundamental problems such as:
– a high level of risk
– application integrity constraints
– business processes split across systems (in both source and target environments)
– lack of flexibility and ability to cope with change
– dependency between systems and processes.
The third generation of data migration tools addresses these issues. Such tools support faster data migration at lower risk, natively support service-oriented principles and data federation. Built using open principles, they help service providers buffer themselves against inevitable change and are re-usable (thereby protecting their investment). One way in which re-use is facilitated, for example, is by separating the physical data transformation (schematic transformation) from the business data semantics, from the transformation process, contrasting current generation approaches which combine all three layers into a single script/graph layer.
Although 2G and 2.5G tools can accommodate the move to service-oriented architectures, they are not built to natively support it, which means ‘workarounds’ have to be designed to accommodate this. These ‘workarounds’ (such as XML ‘wrappers’) introduce performance problems and mean, for example, that updates take weeks to implement rather than hours.
A migration success story: BT Featurenet
Despite the horrendous statistics on failed data migration projects, there are some that go well and a few that go even better than planned. An example of such a project is provided by BT’s highly successful Featurenet migration.
Featurenet is BT Retail’s fully managed service supplying centrex-based services, VPNs and value-added services to around 500 large corporates and enterprises. More than one million directly connected users are supported, who together generate between three and five million charged events daily. The service is highly tailored to the needs of individual customers, meaning that each customer effectively has its own dial plan which has to be mapped on to the public network plan and tariffs. There are a large number of complex, multi-layered tariffs, combined with multi-tiered and multilayered discount schemes. Featurenet services were supported by a bespoke legacy solution for billing, tariff and dial plan management. BT Retail wanted to migrate these customers to Convergys’s Geneva system, but wanted a “completely seamless transition” to the new system.
BT Retail achieved a successful migration in just six months and a full 13 months ahead of schedule, using the Evolve tool from Celona Technologies. Evolve’s application transformation capabilities allowed a much higher level of control, resulting in a lower risk, stepped migration that was driven by the business. BT Retail has since credited the successful project with creating more than euro 148 million in new revenues, thanks to its ability to launch innovative new services to Featurenet customers. “It’s hard to imagine how we could have achieved this highly strategic evolution without Celona’s next-generation products and highly skilled team,” reflects BT’s Billing Development Manager Carl Wilson. Celona Evolve doesn’t use ETL, but a unique synchronise-reconcile-switch methodology that delivers greater control at lower risk.
Questions they’d rather you didn’t ask
– Will your data migration project deliver on time and at budgeted cost?
– Are you comfortable with the risk profile you are assuming?
– What happens if it doesn’t work?
– How much is it costing your organisation for every day of delay?
– How flexible is your migration architecture, will it be able to respond to inevitable change?
