Cloud Shopping. Signing on the dotted line and then some.

Sometimes the blindingly obvious takes a long time to -- well -- become obvious. 

Not so long ago we were promised cloud services would be a one-click shopping experience. Simple or no contracts at all. That seemed far fetched. And it is. 

The business and legal contract is alive and well. And by all account there remains a gulf in cogent understanding about all of the compulsory commercial steps (legal/contract, risk management, solution scope) before you (as a customer) can claim victory and turn that sometimes intangible "cloud service" on. The contract is just one component of the due-diligence and commercial process.

  

A cloud computing “purchasing” guide will not differ from the run of the mill due-diligence in many IT procurement guides. The detail is a re-frame of what we know to be true, but for some reason have not been applied. Maybe humans abandon what is blindingly obvious, until we make a mistake or when we realize the short-cuts lead us astray.

Whatever the physiology, here are the activities that we've come up with: 

• Perform preliminary/legal due diligence
• Define compliance requirements
• Conduct risk assessments
• Define security controls, including security architecture components provided as standard service elements and controls that may need to be negotiated to meet your requirements. 
• Define and implement contract monitoring requirements to measure and correct deviations from the service agreement

Nothing here is rocket-science. If you are part of an "enterprise" (enter company size here) then the scale of your business means the following is mandatory:

1.    Legal due diligence with the cloud provider

• A legal contract (document) must be in place to clarify the roles and responsibilities in accordance to a sub-set of regulatory sources e.g. European Union, Spain, Healthcare. As an initial step the customer of the cloud provider would need to identify the regulatory requirements/frameworks to which they are subject. At the end of the day you will be responsible for meeting those requirements..even those you have not initially identified as relevant in the contract. 
• It's obvious that the regulated entity will be ultimately responsible for compliance. This fact will likely be high-lighted in clear contract language to reinforce each party's rights, obligations and intentions. The use of a cloud solution will not in and itself make a client organization compliant with *all* laws and regulations. 
• Depending on the risks (and complexity of the relationship) it will be necessary to enumerate all the assumptions which are related to the ability of all parties involved to deliver on their commitments be they the data owner, processor and custodian. Here a common interpretation of the law is necessary. 
• A minimum set of terms and conditions are included to raise trust and to help get both the customer and cloud vendor on common ground e.g. confidentiality, intellectual property, warranty, payments, termination, limitation of liability. The list of clauses will grow and shrink depending on whether we are dealing with a regulated entity, private company or mom-and-pop shop
• A cloud vendor will be expected to state in the contract their ability to "support" data protection acts or law. 
• Let’s get an analogy to bring all the points above together. The contracts builds trust between two parties. Trust is like a bank account. You add and remove to it. It will be clear and unambiguous clauses that add to the trust balance. If you start with a “click-through” agreement – you may-be looking at the other end of the gun barrel. The short end of the stick etc.

2.    Compliance Audit of the cloud provider 

• There will be a point where a vendor's assertions (I do this or that) simply have to be demonstrated and proven. An audit will have to be conducted against some "standard" criteria. You can decide to believe what they say – at your own risk.
• An audit can be a cross check against an industry (e.g. PCI DS) or government standard (e.g. NIST SP800-53) or against self-asserted statements (e.g. SAS 70 report).  It is important to specify which compliance or regulatory framework(s) must be met. Other common frameworks: FIPS, ISO27000 series, CoBIT, COSO, and HITRUST (Health Industry Trust Alliance, which includes elements of HIPAA, ISO, PCI, and NIST SP800-53.). The Cloud Security Alliance  has a number of fantastic check-lists and guidelines. 
• There will be compliance regimes that mandate some sort of certification (testing and evaluation) and a separate accreditation step (formal audit by an independent third-party) after the contract is signed and for specific scenario's. Regardless, the contract language will need to be very clear on which parties must be compliant, accredited, or audited, if that is a requirement. The customer or the service provider may need to be compliant/audited/accredited, or that a total solution, spanning across services and components of both the customer and service provider. All depends.
• Another practical matter: who is responsible for costs and for management of any audit or accreditation review and certification for each of the scenarios described above? A customer may want a comprehensive, automated capability for system monitoring, but are not willing to pay for it and seem to expect that the service provider will supply those capabilities without added cost.

3.    Security Due Diligence and Risk Assessment

• A customer may want to (re)confirm assumptions that could have a material effect on the cost of the solution and "true-up" the cost and the solution against a use-case or expectation. The degree of security due diligence depending on the level of trust that has been achieved and that is desired. Security due diligence can take place before the contract is signed, or after -- or (unlikely) never. 
• A list of questions maybe asked of the cloud provider, above and beyond compliance. For example: "Do you have controls in place to prevent data leakage or intentional/accidental compromise". An industry-standard assessment will help in understanding any potential vulnerabilities. The SIG shared assessment framework was developed by several bank and publish a public domain tool that is available on their site at www.sharedassessments.org. They also have a white paper on applying the SIG assessment methodology to cloud computing (available from the Resources page at www.sharedassessment.org/value/resources.html).  HITRUST also publishes an assessment tool for their Common Security Framework.
• Intellectual property and confidentially will be mentioned in the contract. However, that’s not the end of the road. A customer will still need to “design in” the right procedures for things like encryption, restrictions on data access. Key point: it is critical to identify the specific expectations of customer and service provider. As described elsewhere in this list, a customer may ask or be told that a solution will support a particular compliance regime; however, there is often a wide range of possible methods to “support” or “meet” compliance requirements. As an example, consider the HIPAA requirement to monitor system activity; that could be met several ways, each of which would differ in deployment and operational costs: logging system/application activity and manually reviewing logs; IDS; SIEM; GRC, etc. There is as yet no widely agreed “standard of practice” for minimal levels of protection, so a service provider should specify what capabilities will be provided. 
• A security subject matter expert will be expected to dig deeper into vulnerabilities depending on the scope of the solution. Adequate security controls will then be proposed.
• Security due-diligence of the cloud provider (the "host") ideally should be wrapped up before the contract is signed. 
• A security risk assessment of the custom system, application or data elements may need to be completed after the contract is signed and ahead the lights being turned on. 

4.    Security Methods and Quality of the delivered solution 

• Security controls will not be implemented by magic. The necessary resources and tools will need to be put into action by someone Indeed, for both the initial implementation and the ongoing management/maintenance; see my previous comment.
• Someone has to accept the risk decision before the lights are turned on and the “go" or "no go" decision should be based on the circumstances of that moment. In other words risk tolerance may have changed since the contract was signed. 
• This procedural step argues for a robust system security review and acceptance process, similar to the Federal “Authority to Operate” procedure (although, hopefully, not as time consuming or costly). FedRamp is another example of a repeatable, portable and certification and accreditation process.  

5.    Security Operations and Continuous Monitoring: 

• Need to consider "run-time" and operational processes in addition to the technical and procedural security controls. The goal: continual validation of the physical and logical environment and it's status. What level or reporting or visibility will the service provider make available to the client? Will the service provider track incidents and report them (or all that exceed an agreed-to threshold) to the client? Or will the service provider maintain that they will manage the system and treat it as a “black box” from the customers perspective? (This argues that reporting requirements, reporting processes and frequency, and operational responsibilities should be explicitly specified in the contract)
• You can't just audit your systems on an snap-shot basis, things may drift over-time and its best to be "on top of things". 
• Adherence to regulatory compliance is increasingly about a more near-real-time reporting of changes, the status of the operating environment and any priority milestones 

A Model for Credible and Response Security Operations

The focal point for keeping tabs with security and compliance activities has been the Security Operations Center (SOC): a physical location that is the front-line to handle incident reports, review system logs and constantly monitor the environment - 24x7. 

Take the latest Wikileaks extravaganza (details left for another blog post) that pits the security operations of some of the world’s most IT savvy companies (Visa, Mastercard, Paypal and Amazon.com) against relatively un-organized “hactivists”. 

The wall-street journal online gives a picture of what a SOC looks and feels like: “in PayPal's network operation center, charts showing total payments processed per minute and total traffic to the site, along with other data, are projected on a large, curved wall in front of around 20 workstations, each holding three to five computer monitors.” Add security events and blinking lights for threat alerts and you get a SOC.

According to the reports Paypal did not suffer a down-time, neither did Amazon.com. Mastercard and Visa didn’t fair so well. In the article it was speculated that MasterCard and Visa simply did not invest in their security operations to “gird for attacks from a more-sophisticated cyber army”.  

That sort of after-the-fact (lets investigate what happened and then make things better) approach to security operations can be very costly. A survey of 45 organizations by the Ponemon Institute found that on average, cybercrime takes 4 weeks to investigate and each cybercrime averages a cost of $3.8 million/year in financial loss and response and remediation costs.

Security operations also have another omnipresent master: regulatory compliance mandates and legal enforcements. Hundreds of laws are introduced every year at the US state level that affect the collection, use, handling, and disclosure of personal data in one way or another.  These laws may be introduced as “privacy” laws, or may be attached to financial services, health care, employment, children’s services or other laws as well.

And so the conventional model of security operations is too pick and choose from a menu of out of the box technologies and tools with no over-arching strategy or long-tail capabilities road-map. By the latter I mean a lack of investments in niche and customizable applications such as illicit insider threats or dealing with persistent threats. The standard functions that are sourced and acquired either in-house or as a managed service: Infrastructure security, device monitoring and management, Security incident and event management, Security incident response and forensics and Threat research and vulnerability management. 

If we are not getting any safer and compliance is unabated – is doing the same thing over and over again not insanity? The thought is a new working model that helps guide an organization towards a better-quality operational picture that is more responsive, rather than reactive. I am not suggesting new technology or standards. Instead a framework to orient the enterprise in sourcing, acquiring and deploying the arsenal.

The model starts with an abstract layer of requirements to help evaluate the maturity of an organization. Not unlike some common principles (integration, automation, service oriented architectures) that would apply to CRM, ERP, or SCM – a domain that is being re-framed by cloud computing.

Here are some of the competencies or master specifications that could apply to “next generation” security operations:

Process Automated Responses

• Operator activity should be automated processes that accommodate human-in-the-loop work-flows for decision making that can be optimized. The aspiration here is processes that shuttle information to policy makers, engineers, the C-Suite and application developers. If data encryption is ramping up then security operations would be prepared to deal with the ramifications of additional reporting. An anti-virus clean-up task would trigger notifications to engineers and analysts spell out countermeasures
• Strategic plans and courses-of-actions would govern pre-and post incidents and designed to avoid disruption to the mission or business. Basically some sort of “rules engine” that makes the system hum. EINSTEIN 3 is a system that will be deployed by US government agencies that “will have the ability to automatically detect and respond appropriately to cyber threats before harm is done, providing an intrusion-prevention system supporting dynamic defense”.

Enablement of Virtual Resources

• Ability to rapidly source high powered computing resources, to process sudden or unplanned volumes of traffic as well as test countermeasures. For example, a bandwidth-based attack like a Denial-of-service would be met with defense that shield parts of the network from collateral damage. The current trend is also to break apart malware and study it for intent, origin and then design of countermeasures. There are vendors that offer tools to simulate the behavior of malware in a safe and condoned test-bed environments. Support for integration and interoperability is key.
• Private and public partnerships are used to to analyze indicators of attacks and early into the planning phases of the adversary. Imagine, if you will, the National Security Agency (in the US) working with private firms to alert them of possible cyber strikes.  A slippery slope of government intervention. Google is working with NSA to help sort through the Chinese hack of its computers.
• A virtual team of experts and analysts will make-up the diverse mix of users and consumers. Decades of research and technology in IT security has shown us that absolute security is a fallacy. Now its also clear that we have not invested in nurturing the right crop of professionals. A recent study by the Center for Strategic and International Studies nails home the point for the US, that we simply don’t have the talent to stay steps ahead. So outside collaboration and the means to do so will be critical.

Analytics-Driven Security

• We know that fusion of all sorts of data is pivotal to create higher-fidelity alerts of intruders, and spot subtle yet suspicious activity. The idea is to monitor ingress and egress points and pull data from human resources, firewall logs and even law enforcement. Trouble is that today’s intrusion detection systems are short-sighted and often blind-sided. The US government’s EINSTIEN 2 system that is meant to up the ante and is installed at all connections between government computer systems and the public Internet. The system gathers and sketches out threat signatures from foreign intelligence and DoD information assurance missions. 
• Absorbing and monitoring Internet traffic; human, software and computer activity is one thing and then trying to make sense of it all is quite another. The mathematics behind statistical analysis and analogue techniques in data mining offer powerful aids to understand past and present events. Forecasting models can be used to project probability of whether a threat scenario will come to pass.
• Privacy-enhancing mechanisms will be needed to limit the collection and retention of personally identifiable information. When speaking about any surveillance recommendations the temptation will be to over-reach authority and the risk of privacy violation is all too real. Basically the principle here should be to redact or obfuscate sensitive content that is not pertinent to down-stream threat analysis or won’t help the advancement of compliance.

Do these above themes make sense? Send me a note at walid.negm@acecnture.com. I’d like to hear your thoughts on whether the above above themes are relevant to CIOs, CTOs, CISOs and business executives struggling to get their heads around where to prioritize their security investments.

Dumbstruck - Wikileaks, law, cyberware and politics (regulary updated)

Wikileaks as of 12/9/2010 has yet to be convicted of any crime by the US government.

Most reasonable folk can agree that the unauthorized release (leak) of sensitive information should be handled with care. The trouble is most folk are neither sensible, nor in agreement about what sensitive information is.

And the story continues to unfold...

Update on Legal Position
12/09/2010: "The U.S. government indicated 12/09/2010 that WikiLeaks spokesman Julian Assange could be in legal jeopardy for disclosing classified information because he is "not a journalist." When asked whether "traditional media" organizations that republish secret documents could be prosecuted, State Department spokesman P.J. Crowley said that the administration applauds "the role of journalists in your daily pursuits." "In our view, Mr. Assange is not a journalist," Crowley added". Source here.

12/10/2010: "Wikileaks founder Julian Assange, the man behind the publication of more than a 250,000 classified U.S. diplomatic cables, could soon be facing spying charges in the U.S. related to the Espionage Act, Assange's lawyer said today" Source: here

12/13/2010: The World War I Espionage Law criminalizes anyyone who possesses or transmits any "information relating to the national defense" which an individual has "reason to believe could be used to the injury of the United States or to the advantage of any foreign nation." The Espionage Act was not written to distinguish the leaker or the spy and the recipient. 

Streaming Reactions

The US government is rightfully so more than livid and is exploring the options. A slew of companies have pulled the plug on the Wikileaks organization from Visa, Mastercard, PayPal, EveryDNS.

Here is Amazon Web Services getting an earful from some of their customers for pulling the plug.

A new Wikileaks kid on the block is on tap reported by the Swedish newspaper Dagens Nyheter reported today. The new project: Openleaks is said to be online any time now. "The two organizations are similar in that aspect that both are focusing on providing means for whistleblowers to anonymously provide the public with information,” as stated by an insider.

The Politics: 
Varying positions continue to be voiced. One side of the debate foxnews and the other side... the atlantic

The Attacks and Counter Attacks
12/8/2010: Lets get this party started
Wikileaks plays it cool and diversifies -- real quick --spreading its documents on mirror sites, adding redundancy to "caller-id" DNS look-ups and a variety of things so it can take a licking and keep on ticking.

And things all of a sudden start to escalate into extremist and rash retaliation  Reuters: "More cyber attacks in retaliation for attempts to block the WikiLeaks website are likely in a "data war" to protect Internet freedom, a representative of one of the groups involved said Thursday"

12/9/2010: Operation Payback in the news.
"A collective of hackers who have set their sights on those companies that have denied service to WikiLeaks and its founder are now trying to take down Amazon.com. They announced via Twitter that they would begin their attack at 11 a.m. ET". Source: http://mashable.com/2010/12/09/operation-payback-amazo/

Espionage act 'Makes Felons of Us All' - legal experts

Quotes 
"To me, New York Times has committed at least an act of bad citizenship, and whether they've committed a crime, I think that bears very intensive inquiry by the Justice Department," IS Senator Joe Lieberman

"In a time of universal deceit, telling the truth becomes a revolutionary act." -1984, George Orwell 

I think in today's climate, telling the truth is classified as "terrorism"

 

"In our view, Mr. Assange is not a journalist" State Department spokesman P.J. Crowley

"Leaks of classified information to the press have only rarely been punished as crimes, and we are aware of no case in which a publisher of information obtained through unauthorized disclosure by a government employee has been prosecuted for publishing it," - Jennifer Elsea, a legal researcher for the US Congress

Techie Section
Its just like a video game: Operation Payback is asking its followers to download a piece of software called LOIC to fire off a distributed denial of service attack at targets. The question of course is “what IF I get caught”. Here is a snipped from from one of their FAQ’s. By the way V& stands for Van’d – as in when the FBI shows up at your house in Van:

• You probably won't. It's recommended that attack with over 9000 other anons while attacking alone pretty much means doing nothing. If you are a complete idio and LOIC a small server alone, there is a chance of getting V&. No one will bother let alone have the resources to deal with DDoS attacks that happens every minute around the world. Then theres always the botnet excuse. Just say your pc was infected by a botnet and you have since ran antivirus programs and what not to try to get rid of it. Or just say you have NFI what a DDoS is at all.

Cloud Services With Strings Attached...

Without trust, banks can't exist. President Franklin D. Roosevelt said in his first fireside chat with the citizens of the United States, March 12, 1933. That radio broadcast came only eight days into his first term.  After Roosevelt had to close down all banks because of a run on assets.

Since then a varitey of regulations and faith building steps were put into place, until of course the financial banking calamity of recent years put into question "who is running this ship". And another loss of confidence. Trust is what makes an economy flourish or flounder. Trust in companies, the relationships we build and the assurances we get that the products we buy are fresh, well built and without hazard.

Trust is given and taken each time we open our mailboxes..the physical ones.
 
(Believe it or not) 87% of 9000 Americans surveyed by the Ponemon Institute in its 2010 Privacy Trust Study of the US Government ranked the Postal Service first amongst 75 federal government agencies. Simply put we trust that the US Postal  Service is able to keep our information safe and secure. It has a 230 year
history. When you go into a USPS office, you expect reliable and safe delivery of your packages. 

OK, if you are extra cautious about those tax forms and sensitive merger documents you call in FedEx or any number of courier services. Its more expensive but it lowers risk and increases peace of mind with guaranteed delivery and signatures. 

Oh but I am blogging about online a.k.a cloud services.

A cloud provider (LLC, corporation etc.) that cannot be trusted will not exist. Sounds simple enough. I will add some other predictions for a patently untrustworthy cloud service providers:

(1) put out of business because of the natural order of things
(2) be relegated into a bazaar of low-cost and low-quality offerings
(3) be shutdown by the government after a major data breach
(4) if large enough and a "critical infrastructure" be nationalized in the event of a national security issue
(5) address a commodity market that deals with pretty much worthless customer data

Its hard to see that an enterprise (hospital, bank, law firm, pharma etc.) that is going paper-less, will disregard the weight and meaning of trust as it goes about doing its business. In particular when talking about proprietary information. And while the security of sensitive data in the cloud is being hashed out, there is not enough focus on the larger context and meaning of trust: how its created, how it's maintained and how it's destroyed. The conversation dive's into passwords, encryption, regulatory compliance and he said, she said.

OK sure, as consumers we store sensitive information (banking information, personal letters, legal documents) on any number of popular online mail and storage accounts. Truth of the matter, most of these online service providers have been in business less than 10 years. There is social shift in the expectation of low (or no) privacy in cyberspace (another topic).

With that said, the more we have to lose, the less likely we'll trust just "anybody".  And by trust, I mean our willingness to depend (or interdependence) on someone (or something) else. In my simple mind, there are four behaviors that we exhibit in the real-world which are naturally present in cyberspace:

1. Carelessness          
2. Paranoia          
3. Practicality: Most businesses and consumers would like to be in this mindset as we are bombarded with new technology and the peer pressure to move into the state of the art. We rank convenience high. We may take care of basic check boxes, but we are not finicky
4. Prudence
   
As enterprises(and consumers) ramp up the volume of outsourcing of data storage, shared application hosting and third-party data processing, we'll see more of the prudent mind-set start to change the marketplace for cloud services.

Vendors will have to respond to a demand for "assured" cloud services that offer more than one-sided technical security controls, standards and empty promises.

The service provider in this sector will demonstrate financial viability. They have a history and reputation. They see your data as a currency with a Dollar, Yen, Euro etc. value to it. There will be unambiguous obligations to provide compensation from disruption, damage or loss of data. Its nothing new in terms of old-school expectations from any service provider.

Its an understanding that there are some relationships that simply must be built on confidence, some mistakes were built to last - and you can in fact measure trust in cyberspace.

President Roosevelt told his countrymen, "there is an element in the readjustment of our financial system more important than currency, more  important than gold, and that is the confidence of the people"...replace financial system with critical cloud services.

Reminiscing

A little more than 2 years ago I put down on paper rudimentary thoughts (and borrowed some good ones of course) about the risks of cloud computing. Since then there has been no stopping the cloud tsumani. I recently got introduced to Apple's iDisk. A near perfect utility: an innocent looking icon on the desktop where you can move all your beloved stuff to the "cloud". 

Where the photo's, documents will go, no one knows.

Here are some of those risk-related properties of cloud computing that were swirling in my mind not so long ago:

• Trust and lack there of: How (or why) do you trust a cloud provider to do the right thing? The root of the matter is putting a believable trade-off in place between the risks and one's alternatives. No surprise here. 
• Ease of Reach: Anything (data, machines, applications) that will be neatly placed "out there, somewhere" will be at an elevated risk of abuse by some disgruntled employee, hacker or [insert favorite bad nation here]. The network is the hack. Like black magic an invisible hand will reach over the ether to tinker with, break-into and cause mischief.
• Dispersed Data: Personal, private, pseudo-classified and classified data ... all sitting side-by-side. It just sounds and feels so unnerving... no matter what precautions or promises are made by the trusted provider. Of course there is an answer: isolation. On the spectrum of shared everything, or shared nothing you will have to pick your position. 
• Virtual Time: A Google, salesforce.com or any cloud provider will take advantage of a secret sauce coined: virtualization. Long story short: virtual machines and storage live in a world of virtual time (and space). Without proper accounting, the space-time-continuum can get out of order. Realistically, an anti-virus scan can get tripped up.    
• Mobility: Those virtual servers (which are essentially files) will be placed and then moved around the network whether for maintenance, resiliency or due to randomness. The files will take with them whatever -- data, malicious ware, outdated policies. 
• Fate Sharing: A multi-tenant application or infrastructure that is hit by a catastrophic attack will affect all customers. Unlikely event. But those are famous last words.
• Old Foundations: The internet was not designed for a hostile setting. It is anonymous. It is about speed. There are no safeguards for privacy. It is about openness. All at odds with locks, keys and body guards.
• Emergent Properties: My favourite one of all. The "i don't know what's about to hit me, cause this is all so new". Have a house? Add a window. You have added change. Change = vulnerability. Have a cloud? Who knows whats going to be exploited...

DEFCON 18

The session titles are tinged with cloak and dagger, anarchy and freedom of expression: "We don't need no sticking badges: hacking electronic door access controllers", "Your ISP and the Government: best friends for ever", "Practical cell phone spying".

My personal impression of the some-what cult-like DEFCON security conference can be characterized as  smart people instinctively driven to share knowledge and unadulterated research for a greater good. Whether its the protection of civil liberties, revealing stupid security vulnerabilities and flaws of products or unabashedly calling out vendors on incompetent engineering.

A smattering of speaker comments offer's a peak into the topics for this year's conference:

• There is no patch for stupidity 
• 15 year device life-time == long tail for bad decisions 
• Clever does not mean secure 
• What appears secure is not
• Privacy is a subtle thing
• The warm, fleece-y Snuggie of Obscurity
• Software moves power on the grid
• Cute smart meter is cute
• The dumbest lock design ever
• Assumed to be trustworthy - 543 million devices shipped in first half of 2010
• Download games at your own risk
• An attack on any one node of an electric grid could take that entire grid down
• My life as a spyware developer and why I'm probably going to Hell 
• There is no such thing as privacy. It is dead. Get over it.
• Malware scanner's are mostly stupid
• What can you do with Twitter that is utterly evil? Lots and lots of things 
• There are 155, 693 public water systems - serving 286 million American's
• I don't think you need a sophistical exploit, there will always be a certain number of people that will click "yes" no matter what
• Social engineering has a long history and works just fine on the Internet

The Impact of Scale

The topic of scale is best illustrated through an analogy. 

If you live in a mega-city, then city planning and transportation come to mind. With so many people living in one city, responsible governments must deal with public safety and creating livable structures. Individual buildings, transportation routes, water supply systems - are part of the city. Overtime we got clever and started to architect and build vertically to deal with scarce physical space. As the number of cars on the streets zooms upwards, we study traffic patterns, congestion and invest in alternative modes of transportation. All told, all these "parts" of the city are a system.

The dimensions we use to characterize whether a "system" is large includes number of elements (people, hardware, "things"), tasks, relationships, policies, domains of interest, and enforcement points etc.

If we are investing in smart buildings, smart cities, smart transportation, smart grids, healthcare infrastructures etc. we are talking large scale. We are talking about "system of systems" that is complex and in fact ultra-large.

The theme of ultra-large scale IT systems is explored in "Ultra-Large-Scale Systems: The Software Challenge of the Future" here, a report, published in 2006.

Ask the statisticians, and they would agree, big is in: 

• Number of Cell Phones Worldwide Hits 4.6B in 2010
• 4,000: The number of lines of code in MSDOS 1.0 - Microsoft's first operating system
• 50 million: The number of lines of code estimated to be in Microsoft Vista
• Data storage requirements for smart meters will increase at a rate that resembles a natural logarithmic rate
• The 2009 movie Avatar is reported to have taken over one petabyte of local storage for the rendering of the 3D CGI effects
• The US DoD has 3 million desktops, with just one data-center housing 18 terabyte storage

Of course big things are broken down into manageable parts in an attempt to make sense of the pieces. Cars, routes, traffic control, toll plaza's etc. 

However, we are still left with menacing problems:

• Emergent properties; (or in plain English) stuff will happen that we can't cope with because it is brand new and never happened before. (e.g. the impact of mobile phones on driving behavior)
•  We cannot completely define let alone measure the properties of all components a-priori i.e. somethings are just out of our view-lens
•  It is impossible to update all elements of the system (as each changes) without leaving some window of vulnerability or ambiguity 
• There is continuous evolution of our surrounding environment with unknowable outcomes and inconsistent (changing) states (e.g. new devices, new personalities, new enemies)
• There is no clear ownership, possession and boundaries (e.g. cloud computing)

The Internet and the typical IT infrastructure of an enterprise and government agency is very complex. We think about external networks, web sites, data flows, applications and users, insiders, hardware/software, transactions and supply chains. The Internet and everything in it and connected to it, is very much an ecosystem: a dynamic community of interdependent and competing organizations in a complex and changing environments. The elements have an intrinsic adaptive behavior and we can measure the health and sense troublesome indications. 

To deal with that sort of large-scale complexity, rich research is needed (and going on in) to:

• Understand the biological metaphore and it's applicability to tech-centric views
• Use heuristics and learn how to better predict behaviors 
• Learn how to respond to external stimuli with keener intelligence gathering 
• Assure the software and hardware components that are used to build systems 
• Survive hostile intentions and operate in a continuously changing system