The Impact of Scale

The topic of scale is best illustrated through an analogy. 

If you live in a mega-city, then city planning and transportation come to mind. With so many people living in one city, responsible governments must deal with public safety and creating livable structures. Individual buildings, transportation routes, water supply systems - are part of the city. Overtime we got clever and started to architect and build vertically to deal with scarce physical space. As the number of cars on the streets zooms upwards, we study traffic patterns, congestion and invest in alternative modes of transportation. All told, all these "parts" of the city are a system.

The dimensions we use to characterize whether a "system" is large includes number of elements (people, hardware, "things"), tasks, relationships, policies, domains of interest, and enforcement points etc.

If we are investing in smart buildings, smart cities, smart transportation, smart grids, healthcare infrastructures etc. we are talking large scale. We are talking about "system of systems" that is complex and in fact ultra-large.

The theme of ultra-large scale IT systems is explored in "Ultra-Large-Scale Systems: The Software Challenge of the Future" here, a report, published in 2006.

Ask the statisticians, and they would agree, big is in: 

• Number of Cell Phones Worldwide Hits 4.6B in 2010
• 4,000: The number of lines of code in MSDOS 1.0 - Microsoft's first operating system
• 50 million: The number of lines of code estimated to be in Microsoft Vista
• Data storage requirements for smart meters will increase at a rate that resembles a natural logarithmic rate
• The 2009 movie Avatar is reported to have taken over one petabyte of local storage for the rendering of the 3D CGI effects
• The US DoD has 3 million desktops, with just one data-center housing 18 terabyte storage

Of course big things are broken down into manageable parts in an attempt to make sense of the pieces. Cars, routes, traffic control, toll plaza's etc. 

However, we are still left with menacing problems:

• Emergent properties; (or in plain English) stuff will happen that we can't cope with because it is brand new and never happened before. (e.g. the impact of mobile phones on driving behavior)
•  We cannot completely define let alone measure the properties of all components a-priori i.e. somethings are just out of our view-lens
•  It is impossible to update all elements of the system (as each changes) without leaving some window of vulnerability or ambiguity 
• There is continuous evolution of our surrounding environment with unknowable outcomes and inconsistent (changing) states (e.g. new devices, new personalities, new enemies)
• There is no clear ownership, possession and boundaries (e.g. cloud computing)

The Internet and the typical IT infrastructure of an enterprise and government agency is very complex. We think about external networks, web sites, data flows, applications and users, insiders, hardware/software, transactions and supply chains. The Internet and everything in it and connected to it, is very much an ecosystem: a dynamic community of interdependent and competing organizations in a complex and changing environments. The elements have an intrinsic adaptive behavior and we can measure the health and sense troublesome indications. 

To deal with that sort of large-scale complexity, rich research is needed (and going on in) to:

• Understand the biological metaphore and it's applicability to tech-centric views
• Use heuristics and learn how to better predict behaviors 
• Learn how to respond to external stimuli with keener intelligence gathering 
• Assure the software and hardware components that are used to build systems 
• Survive hostile intentions and operate in a continuously changing system