Hybrid Cloud as Alternative to "Pure" Cloud Computing

Using edge computing for some services improves the utilization of expensive WAN bandwidth, offer better security and cause less headaches for the users. It can be combined with the private cloud

We can define "pure cloud" approach as one in which IT services are provided with the following conditions:

1. Via public Internet or private WAN network
2. By an external company
3. Hardware maintenance is completely outsourced
4. Allocation of computer resources is performed using VMs or containers.

From the historical standpoint "pure cloud" providers represent the return to the mainframe era on a new technology level. Cloud providers are new "glass data centers" and due to overcentralization will soon will be hated as much if not more.  The bottleneck is WAN links, and they are costly both in public and private clouds.

Typically datacenters in large enterprises network provide fixed outbound traffic and as there are many players using the same outbound link. So the saturation of outbound link is a real problem which affects many things including site problems with the videoconferences held in the central office, and such. And the outbound desktop traffic is often channeled via datacenter too.

The limit usually is twice or more higher at night (off-hours) then during the day but  with backups running at night that helps only partially. Another problem is when night backup spills into the morning and "poison" critical applications like SAP/R3 clients, Oracle clients, and such

With proliferation of home offices due to COVID-19 VPN became another critical factor that create the bottlenack over WAN, as VPN consume the bandwidth twice: if you download a ISO from internet it first is send to the your VPM provider (typically corporate datacenter, or what left out of it) and then you again download the file over WAN to your desktop.

Microsoft and Google drives also work similarly if your company uses proxy. In other words, if in its infinite IT management wisdom a particular enterprise uses Microsoft Drive for desktop backup instead of local USB drives or some local NetApps, then there is a constant outbound traffic from desktops, which influence other players.

Using UDP-based software like Aspera for transfer of data to the cloud solves the problem of high latency (dramatically, often 20 times, increasing the speed of transfer) , but not the problem of limited outbound WAN bandwidth.

An interesting technology that now is developing as a part of "hybrid cloud" technologies is so called "edge computing".  Of course, now the centralization train is still running at full speed, but in two to three years people might start asking questions "Why my application freezes so often".

There a synergy between "edge computing" and "grid computing": the latter can be viewed as grid computing at the edges of the network with the central management node (the headnode in “HPC speak”) and software like Puppet, Chef or Ansible instead of HPC scheduler.

As WAN bandwidth provided at night is several times higher than during the day, one important task that can be performed by "edge computing" is the “time shift” of large transferes: the delay of some transmissions and synchronization of data to the night time. UDP based tools for night allow to transfer large amount of data by saturating WAN bandwidth, but even TCP is can be usable at night on high latency links, if you can split the data stream into several sub streams.  I reached comparable to UDP speeds when I was able to split the data stream into 8-16 sub-streams.  Those "time shift" schemes does not require large investment.  All “edge” resources should use unified images for servers or VMs and unified patching scheme.

I would define "hybrid cloud" as a mixture of private cloud, public cloud and "edge" computing with the emphasis on an optimizing consumed WAN bandwidth and offloading processes that consume WAN bandwidth at day time (One perverted example of WAN bandwidth usage is Microsoft Drive synchronization of user data)  to the edge of the network via installation of "autonomous datacenters." Which can be just one remotely controlled.  No local personnel is needed, so this is still "cloud-style organization" as all the staff is at central location.  For the specific task of “shifting synchronization of user data” you need just one server with storage oer site.

From Wikipedia("edge computing"):

Edge computing is a distributed computing paradigm in which computation is largely or completely performed on distributed device nodes known as smart devices or edge devices as opposed to primarily taking place in a centralized cloud environment. The eponymous "edge" refers to the geographic distribution of computing nodes in the network as Internet of Things devices, which are at the "edge" of an enterprise, metropolitan or other network. The motivation is to provide server resources, data analysis and artificial intelligence ("ambient intelligence") closer to data collection sources and cyber-physical systems such as smart sensors and actuators.[1] Edge computing is seen as important in the realization of physical computing, smart cities, ubiquitous computing and the Internet of Things.

Edge computing about centrally (remotely) managed autonomous mini datacenters. Like in classic cloud environment staff is centralized and all computers are centrally managed. But they are located on sites not in central datacenter and that alleviate WAN bottleneck, for certain applications eliminating it completely. Some hardware producers already have product that are designed for such use cases.

... ... ...

Edge computing and grid computing are related.

IBM acquisition of Red Hat is partially designed to capture large enterprises interest in "private cloud" space, when virtual machines that constitute cloud are running on local premises in para-virtualization mode (Solaris x86 Zones, Docker containers).  This way one has the flexibility of "classic" (offsite) cloud without prohibitive costs overruns. On Azure a minimal server (2 core CPU, 32GB of space and 8GB of RAM) with minimal, experimental usage costs about $100 a month.  You can lease $3000 Dell server with full hardware warranty and premium support for three years for $98.19 a month.  With promotion you can even get free delivery and installation for this amount of money and free removal of hardware after three years of usage.  So like with cloud you do not touch hardware, but you do not pay extra to Microsoft for this privilege either (you do have air-conditioning and electricity costs, though). And such "autonomous, remotely controlled,  mini-datacenter" can be installed anywhere where Dell (or HP) provide services.

There are now several hardware offerings for edge computing. See for example:

• Dell
  • Dell EMC PowerEdge M640 Blade Server Dell. Dell m1000e enclosure has an excellent remote management capabilities, far superior to HP offerings.
  • Dell EMC PowerEdge T440 Tower Server Dell a single tower server can run several virtual instances, capable to create full infrastructure for a small office. DRAC 9 enterprise is the leading server remote control on the market.
• HP
  • Scalable Modular Data Centers (MDCs) for Enterprise HPE
  • HPE Synergy Composable Infrastructure Platform HPE
    • moorinsightsstrategy.com
  • HPE Moonshot System family guide (Family guide-4aa4-6076enw.pdf)

Unfortunately, centralization drive now still rules. But having some "autonomous units" on the edge of network is an attractive idea that has future. First of all, because it allows to cut the required WAN bandwidth. Second, if implemented as small units with full remote management ("autonomous"), it allows to avoid many problems with "classic cloud" problems -- which are similar to the problems of any large corporate datacenters. 

Logically sooner or later the age of fully centralized cloud should come to the logical end.  Something will replace it. As soon as the top management realizes how much they are paying for WAN bandwidth there will be a backlash. I hope that hybrid cloud might become a viable technical policy in two to three years’ timeframe.

And older folks remember quite well how much IBM was hated in 60th and 70th (and this despite excellent compilers they provided innovative VM/360 OS which pioneered multitasking in the USA; base OS of VM/360 used REXX as shell) and how much energy and money people have spent trying to free themselves from this particular "central provider of services" model: "glass datacenter".  It is not unfeasible that cloud provider will repeat a similar path on a new level. Their strong point: delivering centralized services for mobile users is also under scrutiny as NSA and other intelligence agencies have free access to all user emails. 

Already now in many large enterprises WAN constitute a bottleneck, with Office 365 applications periodically frozen,  and quality of service deteriorates to the level of user discontent, if not a revolt.

People younger then 40 probably can't understand to what extent rapid adoption of IBM PC was driven by the desire to send a particular "central service provider" to hell. That historical fact raises a legitimate question of the level of hidden user resistance to the "public cloud" model. Now security consideration and widespread NSA interception of traffic, geo-tracking of mobile phones and unlimited unrestricted access to major cloud providers webmail also became hotly debated issues.

WAN bandwidth is the bottleneck; Peter Deitch "eight classic fallacies"

The key selling point of edge computing for the enterprise datacenter is that the remote control technology now reached the level on which autonomous small datacenters -- datacenters with zero local support personnel are not only feasible but can compete with the classic cloud model, because this model has one grave deficiency -- WAN bandwidth as the bottleneck.

Like any new service along with solving old problems public cloud computing creates new, often more complex and less solvable. Proponents often exaggerate positive features and underestimate problems and costs. In no way cloud services as exemplified by Azure and Amazon are cheap. On Azure a minimal server (2 core CPU, 32GB of space and 8GB of RAM with minimal, experimental usage costs about $100 a month).  You can lease $3000 Dell server for $98.19 a month.  With promotion you can  get free delivery and installation.

The vision of the IT future based on a remote centralized and outsourced datacenter that provides services via "cloud" using high speed fiber links is utopian as WAN can't scale. It is very true that the huge bandwidth of fiber optic lines made "in the cloud" computing more acceptable and some business models impossible in the past quite possible (Netflix). But that does not mean that this technology is a "universal bottle opener". Bottlenecks remain and enterprises now spend quite a bit of money fighting them. I would like to stress that this is just another example of centralized computing model (with usage of WAN connection instead of LAN connection for clients) and as such it has its limits.

According to Wikipedia "The Fallacies of Distributed Computing" there exists a set of common but flawed assumptions made by programmers in development of distributed, connected via network applications. They were originated by Peter Deitch and his "eight classic fallacies" can be summarized as following:

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

Giants of cloud computing such as Amazon, Google, Yahoo and Microsoft push the idea of "pure" centralized "cloud-based" services to the extreme. They essentially are trying to replicate mainframe environment on a new level. Inheriting all the shortcoming of such an approach. Many companies is such industries as aerospace, financial services, chemical and pharmaceutical industries are trying to avoid "pure" cloud computing model because of their trade secrets, patented formulas and processes as well as compliance concerns. Also large companies usually run multiple datacenters and it is natural for them to be integrated into private "gated" cloud. No rocket science here.

These private clouds use a dedicated private network that connect their members and that help to avoid the problem of "bandwidth communism". At the same time a private cloud can consolidate the hardware and software needs of multiple smaller sites, provide load balancing and better economies of scale.

Backup in the cloud controversies

Costs of cloud servers are high. Using a small one socket "disposable" server and automatic provisioning and remote control tools (DRAC, ILO) you can achieve the same result in remote datacenters at a fraction of costs. This new idea of disposable server (a small 1U server now costs $1000 or less) is a novel idea that gives the second life to the idea of "autonomous datacenter" (at one point promoted by IBM,  but soon completely forgotten as modern IBM is a fashion driven company ;-)  and is a part of hybrid cloud model when some services like email are highly centralized, but other like file services are not.

The problem with "pure cloud" is the bandwidth cost money.  That's why the idea of "backup to the cloud" (which in reality simply means backup over WAN) is such a questionable idea. Unless you can do it at night and did not splash to working hours  you compete for bandwidth  with other people and applications. It still can be used for private backups if you want to say good buy to your privacy.

But in an enterprise environment if such a backup spills over to morning hours you can make life of your staff really miserable, because they are depending of other applications which are also "in the cloud". This is classic Catch 22 situation with such a backup strategy. It is safer to do it "on site" and if due to regulations you need to have off site storage is probably cheaper to buy a private optical link (see, for example, AT&T Optical Private Link Service) to a suitable nearby building. BTW a large attached storage box from, say, Dell costs around $40K for 280TB, $80K for 540TB and so on, while doubling the bandwidth of your WAN connection can run you a million a year.

For this amount of money you move such a unit to a remote site each week (after full backup is done) using a mid size SUV  (the cost of SUV is included ;-). For more fancy setup you can use ideas from container based datacenters  and use two cars instead of one (one on remote site and the other at main datacenter) for weekly full backup (Modular data center - Wikipedia ). Differential backup usually are not that  large and can be handles via  wire. If not, then USPS or FedEx is your  friend. You will still money left of a a couple of lavish parties in comparison with your "in the cloud" costs.

Yes there are some crazy people who are trying WAN transferees of, say, 50-100TB of data thinking that this is a new way to do backups or synch two remote datacenters. They typically pay arm and leg for some overhyped and overpriced software from companies like IBM that uses UDP to speed transfer over lines with  high latency. Such huge site-to-site transfer can't be saved with such a software no matter what presentation IBM and other vendors will give to your top brass (which usually does not understands WAN networking, or networking at all; the deficiently on which IBM relay for a long long time ;-). But multicast feature of such software is a real breakthrough and if you need to push the same file (for example a large videofile) to multiple sites for all employees to watch,  it is really great).

But you can't change basic limitations of WAN. Yes some gains can be achieved by using UDP instead of TCP/IP, better compression, and using rsync-style protocol to avoid moving of identical blocks of data. But all those methods are perfectly applicable to local data transfers. And on WAN you typically are facing 3-5MB/sec links (a 10% fraction of 1Gbit link in best case). Now please calculate how much time it will take to transferee 50-100TB of data over such a link.  On local 10Mbit link you can probably get 500 MB/sec. So the difference is speed is 100 times, give of take.

Solving WAN bottleneck using edge computing and autonomous datacenters

There is also a complimentary trend of creating remote autonomous datacenters controlled and managed from the well equipped central command&control center. Server remote control tool now reached the stage than you can mange server hundreds miles from your office as well as if you can walk to the server room. Actually not quite, and here much depends on your ingenuity.

But having a local servers with a central control center like is common of blade enclosures instead of centralized in some disctant location serve farm is more flexible approach as you can provide computer power directly were it is needed at the cost of a typical large server hosting provider.

Distributed scheme also requires much less WAN bandwidth and is more reliable as in case of problems with WAN connectivity. for example in case of  hurricane that cut electricity supply in the central office, local services remain functional. IBM in the past promoted this approach under the name of "autonomic computing". Without much success. Probably due IBM level costs and marketing involved. 

The level of automation strongly influences how efficiently IT is running. This factor alone might have greater influence on the future shape of IT than then all this "in the cloud" buzz. Vendors propose multiple solutions know as Enterprise System Management (ESM), computer grid (Sun), etc. The vision here are autonomous centrally controlled virtual instances controlled from a common control center. This approach can be combined with several other alternatives to "in the cloud" computing, such as "datacenter in the box", virtual appliances, and application streaming.

One important advantage of hybrid approach is the possibility of using autonomous servers. This is pretty new development that became possible due to advances in hardware (DRAC 7, blades) and software (Virtual Software Appliances). By autonomous server we will understand a server installed in location where there is no IT staff. There can be local staff knowledgeable on the level of a typical PC user, but that's it. Such servers are managed centrally from the central enterprise-wide location which retains highly qualified staff. Such location can be even on a different continent, if necessary.

This type of infrastructure is pretty common for large chemical companies which might have a hundred or more factories, storage depot, research labs, etc), banks (with multiple branches) and many other companies with distributed infrastructure. Details of implementation of course differ. Here we will concentrate on common features rather then differences. The key advantage of local autonomous servers (and by extension small local autonomous datacenters) is that local services and local traffic remains local. The latter provides several important advantages. Among them:

• You are less dependent on the reliability of central infrastructure of your "cloud provider" or company "master" datacenter hosting private cloud (which can well be across Atlantic). In case of "hybrid cloud" you can provide bandwidth-intensive services over local LAN via local real or virtual servers. Only administration is centralized.
• Snooping of traffic is more difficult as processing is distributed and, for example, if user mailboxes exist at local locations, not at central location where you can pick up all the information in one swipe.
• Cost of such infrastructure is lower then central cloud infrastructure
• Dependency on third parties is lower and you can control more aspects of it.

The attraction of cutting WAN traffic is directly connected with advances of modern hardware and software:

• Modern hardware provides reliable remote administration with virtual console and good remote management capabilities build-in (especially in blade environment); there are advances in virtual machine hypervisors, which allow running virtual instances at speed close to the speed of "native" OS, especially Solaris zones and linux containers.  A typical Dell or HP server easily endure fives years of service without single minute of hardware-related downtime.
  • Services from modern manufacturers have built-in KVM (DRAC in Dell, ILO in HP) with remote control of power and many bell and whistles. Dell DRAC is the standard in the industry as for capabilities (especially in blade environment) with DRAC 7 outperforming HP ILO by a wide margin.
  • Raid configurations with additional redundant drive and/or two small/cheap (64GB or 128 GB) mirrored SSDs as a separate OS drive are essentially bullet-proof. Chances that they can cause downtime are very slim.
  • Redundant power supplies usually last five-seven years without malfunctioning.
• Modern software now provide powerful tools for managing multiple similar servers from the central location dramatically lowering the cost of "reverse cloud" administration.

Problems with "pure" cloud computing

Cloud computing in it "pure" form (100% on remote datacenter) is centralized mainframe-style solution and it suffers from all limitations inherent in centralization. All powerful remote datacenters with minimum local client functionality are actually pretty costly and savings sited if advertisements are more often then not pure fiction. They are attractive for higher management as "stealth" outsourcing solution, but that's it. As any outsourced solution it suffers from loyalty problem. Cole combination of local and centralized capabilities are much more optimal. There are several important concerns with "pure" centralized solution (single remote datacenters-based cloud provider):

• Application performance concerns (WAN bandwidth issues; capability of remote datacenters to provide adequate response time, etc)
• Bureaucratization problems, sweeping under the carpet mainframe-style problems with external providers of IT services (everything becomes contract negotiation).
• Interoperability problems. Integration issues and balkanization threat in case of multiple "in the cloud" vendors. The lock-in concerns as well as potential rip-off due to complicated pricing models
• Total cost of ownership concerns. Cloud providers are pretty greedy and the long time cost can't be assumed to be stable, it can rise substantially as soon as the vendor feels that the customer locked into solution, evaporating all potential savings.
• Security concerns. Those are simply huge, from that danger of government interception of all data to competitors stealing the crown jewels of technology.
• Capabilities of application and application performance concern. Even the most powerful remote datacenter has difficulties to compete with model 3GHz 16GB of memory and 400GB SSD laptop.
• Poor customer service and increased risk of unexpected and damaging downtime. For cloud provider you are just one of many clients and in case of downtime you are typically left in the cold. As downtime is couse by problem in mind boggling complex giant infrastructure (typiclaly this is network problems) they are often slow to resolve depite of concentration of high level specialists.

Cloud computing as a technology is already more then ten years old ;-) and now we know quite a bit about its limitations.

While an interesting form of reincarnation of mainframes, it is critically dependent on availability of high speed fiber WAN networks. But this technology can be implemented in many flavors: can be a pure play (everything done on the servers of the cloud provider, for example Amazon or Google) or mixed approach with a significant part of the processing done at the client (more common and more practical approach). In no way it presuppose simplistic variant in which the everything should be done of the server while for some strange reasons rich client considered to be not kosher.

What is more important is providing Software as a Service (SaaS) . SaaS is a model of software deployment where an application is hosted as a service provided to customers across the network (local or Internet). By eliminating the need to install and run the application on the customer's own computer, SaaS alleviates the customer's burden of software maintenance, ongoing operation, and support. Various appliances (email, DNS, etc) are one popular implementation of SaaS.

Webmail, web hosting and other new "groupware" technologies (blogs, wikis, web-conferencing, etc) can he hosted on the local network cheaper and more reliably then on Google or Amazon and they enjoy tremendously better bandwidth due to usage of local network (which now is often 10Gbit/sec) instead of much slower (and in case of private networks more expensive) WAN. Web hosting providers emerged as a strong, growing industry that essentially pioneered the commercialization of SaaS model and convincingly proved its commercial viability. As Kismore Swaminathan aptly noted in his response to Alastair McAulay’s article

.... as presently constituted, cloud computing may not be a panacea for every organization. The hardware, software and desktop clouds are mature enough for early adopters. Amazon, for example, can already point to more than 10 billion objects on its storage cloud; Salesforce.com generated more than $740 million in revenues for its 2008 fiscal year; and Google includes General Electric and Procter & Gamble among the users of its desktop cloud.

However, several issues must still be addressed and these involve three critical matters: where data resides, the security of software and data against malicious attacks, and performance requirements.

Hillary Clinton private emails server scandal and Podesta email hack provides additional interesting view of the problems of cloud computing,

Actually the fact that General Electric and Procter & Gamble use Google arises strong suspicion about quality of high IT management in those organizations. IMHO this is too risky gamble to play for any competent IT architect. For a large company IT costs already are reduced to around 1% or less, so there are no big saving in going further in cost saving direction. But there are definitely huge risks, as as some point quantity of cost cutting turns in real quality of service issue.

I would argue that "in the cloud" paradise looks more like software demo from a popular anecdote about the difference between paradise and hell ;-). It turns out that for the last five years there are several competing technologies such as usage of virtual appliances and autonomous datacenters or as they are sometimes called "datacenter in the box".

Usage of a local network eliminates the main problem of keeping all your data "in the cloud": possibility of network outages and slowdowns. In this case all local services will continue to function, while in the "pure" cloud services are dead. From the end user perspective, it doesn’t make a difference if a server glitch is caused by a web host or a software service provider. Your data is in the cloud, and not on your local PC. If the cloud evaporates you can’t get to your data. This iss well known to Google users. If the service or site goes down, you can’t get to your data and have nobody to contact. And even if you have somebody to contact as this is a different organization, they have their own priorities and challenges.

While software as a service might represent a licensing savings, there might be a better way to achieve the middle ground and lot to lose all advantages of local storage of data. For example by using some kind of P2P infrastructure automatically synchronized so that you can view/edit data locally. Data should not be held hostage until the user can get back to the cloud. In this sense Microsoft's Microsoft Live Mesh might be a step in right direction as it provides useful middle grown by synchronizing data across multiple computers belonging to a users (home/office or office1/office2, etc).

But the key problem with "in the cloud" delivery of services is that only some services, for example Web sites and email, as well as others with limited I/O (both ways) are suitable for this deployment mode. Attempting to do company wide videoconference via cloud is a very risky proposition.

That does not means that the list of services that are OK for centralization and can benefit from it is short. Among "limited I/O" services we can mention payroll and enterprise benefits services -- another area where "in the cloud" computing definitely makes sense. But most I/O intensive enterprise processing like file sharing is more efficiently done on a local level. That includes most desktop Office related tasks, ERP tasks to name just a few. They are more difficult and more expensive to move into the cloud and economic justification for such a move is open for review. So in a way it is a complementary technology to the local datacenter not an alternative one. Moreover "in the cloud" approach can be implemented on a local level over LAN instead of WAN ("mini cloud" or "cloud in the box").

Actually cloud-based services itself are not that cheap so cost savings in switching to the "in the cloud" service provider for large organizations are typically exaggerated and might even be illusory.

The top rated providers such as Amazon are mainly interesting, if you experience substantial, unpredictable peaks in your workloads and or bandwidth use. For stable, consistent workloads you usually end paying too much . Spectrum of available services is limited and outside running your own virtual servers it is difficult to replace services provided by a real datacenter. May be except small startups. The most commercially viable part is represented by Web hosting and rack hosting providers. For web hosting providers the advantages are quickly diminishing with the complexity of website, though. Among Amazon Web services only S3 storage currently can be called a successful, viable service. Microsoft Live Mesh is mostly a data synchronization service. It presuppose existence of local computers and initially provides syncing files between multiple instances of local computers belonging to the same user. This is an important service, but this not a pure "in the cloud" provider model. It is a more realistic "mixed" approach.

Again, with the current prices of servers and network equipment, most existing services are not cheap and became really expensive as you scale them up. We will discuss this issue later.

All-in-all it is not very clear what market share "in the cloud" services deserve and but it is clear that "in the cloud" service providers can't fit all enterprise needs. Currently, there are a lot more places to run software than there used to be, thanks to the proliferation of powerful laptops, mobile phones like iPhone/Android phones, tablets, etc. Actually smartphones represent another interesting development that runs in the direction opposite to the move to the "in the cloud" computing. It is a powerful local device with wireless network connectivity. The key here is substantial processing power and storage capacity available on the device which is increasing with each new model.

From social standpoint cloud provider services are also problematic. Service providers mind their own interests first. also large service provider is the same bureaucratic monster as a typical datacenter that share with the latter all the spectrum of Dilbertalization problems. Large customers experience "vendor lock-in" working with service providers as it involves significant effort to adapt on both sides. So walk-out is a less viable option that one can assume on pure theoretical backgrounds. It is also possible that outsourcing to "software service providers" like any outsourcing can be used as a negotiating tool (aka "method of blackmail"), to force wage concessions from workers, tax breaks from governments, etc., even when the actual savings would be small, or even when they are negative and moving makes no business sense whatsoever.

Bandwidth communism

Promoters of remote outsourced datacenter, such as Nicholas Carr usually ignore the availability and the cost bandwidth. Think Netflix and all conflicts it is fighting with the local cable internet providers. We can't assume that free Internet connectivity is adequate for all business purposes. such an assumption is corrected called "bandwidth communism". Yes, fiber-optic changed WAN landscape making remote services more viable and Internet tremendously more powerful. But the devil is in details. For example file sharing for a large company over WAN is still bad idea as public bandwidth is insufficient and private is costly. Also any enterprise making bet of 24x7 availability of public bandwidth for vital corporate services looks slightly suicidal because of the "tragedy of commons" problem, which already demonstrated itself in repressions against P2P enthusiasts by large ISPs. All-in-all this "grand utility computing" vision ("bandwidth communism") is problematic because somebody needs to pay for all this expensive infrastructure.

Fiber networks increased both Internet and LAN bandwidth substantially and revitalized distributed computing. But there is a big difference whether you distribute over LAN or WAN. The latter is much tougher case. With all the tremendous progress of Internet available bandwidth does not increase as quickly as computing power. Nowhere close, and it never has. If anything, due to increased scrutiny and "shaping" by ISPs (they are not a charity and need to be profitable) bandwidth "per user" might recently start decreasing as such resource hogs as YouTube and video programming distribution services (movies on demand) are becoming more and more prominent. Ability of video streams and P2P services to clog the Net in the most inopportune moment now is well established fact and is a real curse for university networks.

For i/o intensive tasks, unless you pay for the quality of service, "in the cloud" computing model stands on a very shaky ground. Reliable 24x7 bandwidth cannot be free for all users in all circumstances and for all protocols. Substantial amount of traffic with the remote datacenter is not that easy to transmit reliably and with minimal latency via public channels in rush hours. But buying private links to remote datacenters can be extremely expensive: for mid-side companies it is usually as expensive as keeping everything in house. For multinationals it is more expensive, so only "other considerations" (like "jurisdiction over the data") can sustain the centralization wave to the large remote datacenters. For many multinationals SOX was the last straw that made move of datacenters out of the USA desirable, costs be damned. Now the shadow of NSA serves as keeping this scare alive and well. Cost of private high speed links limits cost efficiency of the "in the cloud" approach to any service where disruptions or low bandwidth in certain times of the day cannot lead to substantial monetary losses. For critical business services such as ERP public data channel can be too brittle.

But it is fair to state that the situation is different for different services. For example for SMTP mail outsourcers like Google/Postini, this problem is not relevant due to the properties of the SMTP protocol: they can communicate via regular Internet. The same is true to DNS services providers, webmail and instant messaging. CRM is also pretty close. But for ERP, file sharing and WAN based backup the situation is very different: providing high speed networking services over WAN is a very challenging engineering task to say the least. The cost of bandwidth also puts natural limits on service providers growth as local networks are usually much cheaper and much faster. Achieving 1Gbit/s speed on LAN is extremely cheap (even laptops now have 1Gbit adapters) while it is quite expensive over WAN. There are also other limiting factors:

1. The possibility of creation of local LAN backbone with speeds higher the 1 Gbit/s. 10Gbit/s backbones are becoming more and more common.
   
2. Limited bandwidth at the point of connection of provider to the Internet. Every provider is connected to the Internet via a pipe and that pipe is only so big. For example OC-1 and OC-3 have their upper limit of 51.84Mbit/s and 155.2 Mbit/s correspondingly. Funny the upper speed of OC-3 (which is pretty expensive) is only slightly higher that 100Mbit/s which long ago became the lowest common denominator for LANs. Large service providers typically use OC-46 with speed up to 2488.32 Mbit/s which is similar to the speed of gigabit Ethernet. 10 Gigabit Ethernet is the fastest commonly available network standard for LANs. It is still emerging technology with only 1 million ports shipped in 2007 but it is quickly growing in importance. It might be eventually used in modified form for WANs too. Anyway as WAN bandwidth is limited and shared between multiple customers the spike in activity of one customer might negatively affect others. Networking problems at the provider level affect all its customers and recovery period might lead to additional spikes of activity.
   
3. Local vs. remote storage of data. Recent enterprise level hardrives (Cheetah 15K) have speed up to 164 MB/sec (Megabytes, not megabits). From the speed and cost point of view the ability to keep data/programs local is a big technological advantage. For I/O intensive applications it might be that the only viable role for remote providers is synchronization with local data ;-). Example of this approach is Microsoft's Live Mesh
   
4. Interdependence of customers on the transport level. This is jut another incarnation of "tragedy of commons" problem. Bandwidth hogs like game, P2P, music and video enthusiasts do not care a dime about your SLA and can easily put a company that uses public links into disadvantage any time of the day if and when something new and exiting like a new HD movie was released. Also providers are not willing to sacrifice their revenue to accommodate "free-riders.": as soon as usage of bandwidth cuts into profits it is punishable and no amount of rhetoric about "Internet freedom" and "Net neutrality" can change that. That means that enterprise customers relying on public bandwidth can suffer from the effort of providers to manage free-riding. That means the corporation which moved services to the cloud competes with various bandwidth hogs who do not want to scarifies any ground and ready to go quite far to satisfy their real or perceived needs. My university experience suggest that corporate users can suffer from Internet clogging in the form of sluggish download speeds, slow response times and frustration with i/o intensive services that become much less useful and/or enjoyable. See for example Time Warner Cable Vs. The Horde.
   
5. Competition for the resources at remote datacenter level. For any successful service providing all the necessary bandwidth is costly and cuts into margins. Recently Amazon faced the situation when bandwidth required for its Elastic Compute Cloud (EC2) proved to be higher then by all of Amazon.com’s global websites combined. You can read between lines how that affect profitability:

Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com’s global websites combined.

Web services providers which offer customers unlimited bandwidth are banking on the fact that the majority of their customers will not use much of their bandwidth. This is essentially a marketing trick. As soon as you exceed a fraction of what is promised they may well kick you out. People who tried to implement software , mp3 or video sharing services on low cost ISP accounts realized that very soon. See for example references that I collected under "Unlimited bandwidth myth". Web neutrality does not mean the tragedy of commons is not applicable. As Bernardo A. Huberman, Rajan M. Lukose noted:

Because the Internet is a public good and its numerous users are not charged in proportion to their use, it appears rational for individuals to consume bandwidth greedily while thinking that their actions have little effect on the overall performance of the Internet. Because every individual can reason this way, the whole Internet's performance can degrade considerably, which makes everyone worse off. An analysis of the congestions created by such dilemmas predicts that they are intermittent in nature with definite statistical properties leading to short-lived spikes in congestion. Internet latencies were measured over a wide range of conditions and locations and were found to confirm these predictions, thus providing a possible microscopic mechanism for the observed intermittent congestions of the Internet.

So a company which will try to implement Web based streaming of say corporate video conference via cloud is up to nasty surprises unless it paid "arm and leg" for dedicated lines to its headquarters and other major locations (which make the whole idea much less attractive in comparison with the local datacenter). The ability to stream video of any considerable quality in real-time between two (or more!) arbitrary points in the network is not really something that can be easily done over the current Internet.

The main point to make is that a reliable WAN network connectivity cost a lot of money is difficult to achieve. This problem is unavoidable if your major components are "in the cloud" (in WAN). Also in the "free internet" enterprises are starting to compete for bandwidth with streaming media (films over Internet). The latter proved to be a huge resource hog and quality of a typical Internet connection now fluctuates widely during the day. That means that in order to achieve respectable quality of service for bandwidth intensive applications enterprises need to buy dedicated WAN connections. That is a very expensive habit to say the least. In typical for multinationals moves, say, relocation of SAP/R3 instance from USA to Europe (just from one continent to another) to achieve reasonable latency for requests coming from the USA is not that easy and definitely not cheap. The cost of high bandwidth transatlantic connection is the major part of additional costs and eats all savings from the centralization. The same effect is true about any WAN connection: reliable high-bandwidth WAN connections are expensive. Moreover the reliability needs to be carefully monitored and that also cost money as anybody who was responsible for the company WAN SLA can attest.

Sweeping under the carpet bureaucratization and mainframe-style problems
with external providers of IT services

There is a shadow of mainframes over the whole idea of "in the cloud" software services providers. Excessive centralization has its own perils, perils well known from the days of IBM360/370 and "glass datacenters". While at the beginning growth of "cloud services" providers increase profit margins and may even increase the quality of the service eventually each organization hits "size" wall.

That's why businesses tend to perceive such providers as inflexible, and, due to natural pre-occupation with profit margins, hostile to their business needs: exactly like their perceived "grass datacenter" in not so distant past. So much for the improvement of the business model in comparison with traditional local datacenters, as problematic as they are (and I would be the last to deny that they have their own set of problems).

As one reader of Calculated Risk blog noted in his comments to the post Popular Google Product Suffers Major Disruption

Lucifer (profile) wrote on Sat, 6/20/2009 - 12:06 pm

They started employing more MBAs at Google? Seriously, any creative company that "grows up" and starts taking advice from bean counters/ entrail readers and sociopaths is doomed.

Microsoft's greatest strength was that its founders never lost control.. so even though their products were inferior (at least initially), their competitors took advice from MBAs and f**ked themselves up, leaving MS as the only survivor.

It's very hard to scale service-based tech firm and keep the mojo that made the startup successful in the first place, especially via acquisitions or employing 'professional managers' to operate the company. Basically I think the story is simple -- Parkinson's law -- bureaucracies naturally grow without limit. That includes the management of large "cloud-services" providers including Google. Excessive layers of middle managers and affiliated cronies ("shadow secretaries") count in the denominator of labor productivity. Everyone knows that many middle managers (often called PHBs) are mainly "inventing" work for themselves and other middle managers writing memos and calling meetings and stuff. Middle management is all about sustaining internal information flow; technology makes good middle management more efficient since it enables better information flow but it makes bad middle manager even more harmful as they "invent" useless information flow (spam) and block useful information in order to survive.

Issues of quality, loyalty, knowledge of the business are automatically surfaced and as a result customers suffer. Mainframe-style utility model encourages excessive bureaucratization with rigid procedures, stupid customer service and power concentrated at the top. That means that the issues of sociopathic leadership and sycophant managers replacing founders is even more acute then in regular IT firms. Corporate predators prefer large companies. As a result demoralization surface and IT personnel, that is cubicle serfs, now spend a large fraction of their time in the office, surfing the internet.

Contrary to simplistic description and assumptions typical for writers like Carr mainframe warts are very difficult, if not impossible to overcome. And they can be amplified by low cost of free services with no reliability guarantee. Disappearance of data usually is not covered. There is a danger of relying of semi-free (advertisement supported) services too much.

For example anybody who used low cost Web-hosting provider can attest that interests of providers run contrary to the interests of advanced users and as such often stifle advanced technology adoption even if they are supported by a particular provider because they, for example, might increase the load on the server. Also provision of the adequate bandwidth for multiple users (and decent responses times) can be a shaky area. Especially during rush period like 8-11 EST. Typically customer service is far from being responsive to any complains.

Security is another important (and costly to resolve) challenge: break-in into a large provider affects all its customers. There were several high profile break-ins into large Web hosting providers during the last two years, so this is not a theoretical threat. Claiming that Web provider are a total solution for any organization is like saying that just because the Big 4 accounting firms (with their the army of accountants, tax specialists and so on) exist, organizations can do away with internal accountants altogether. The hybrid platforms, such as Saleforce.com's application upgrades and quality of service issue still are of major concern.

Relying on advertisement is another mine field. Many people hesitate to send anything important to a Gmail address, knowing that the mail will be scanned (by what is an advertising company) in transit.

Still there is a lot of disappointments with this model as exemplified with the following characterization of "cloud-based" services and outsourcing:

'We thought that we are like a farmer shooing a fat rabbit, but it turned out that the rabbit is shooing at us."

This quote suggests that providers of such services as any outsourcers in the future might have difficulties to shake money loose from the organizations, as customers discover that the interests are highly divergent. IBM already discovered that this is an inherent limit of their push to the "'service oriented organization". Recently they even experienced a backlash.

Because we need to bridge interests of two or more different organization, there are several significant problems in interaction with "in the cloud" providers. Among them

1. Problem of loyalty. It cuts both ways. First of all in case you use external providers loyalty of staff disappears and for any complex service you face "all or nothing situation": If service works everything is wonderful, but if it does not troubleshooting is extremely difficult and fixing the problem is almost impossible even if you understands what is happening -- infrastructure belongs to other organization. Anybody who used Web hosting (the most successful example of such services) can attest that this is a wonderful service as long as it works. But if you have a problem you have a big problem: local staff has no loyalty to the particular organization and competition does not work as another provider can be as bad or even worse; so switching brings you nothing but additional headache and losses. Even elementary problem can take several months to be resolved and I am not joking I experience it myself. Oversubscription which leads to highly loaded servers and insufficient network bandwidth is another common problem. There are also problems related to the "race to the bottom" in such services: the main differentiator becomes price and to attract new customers Web providers often make claims that are untrue (unlimited bandwidth is one typical example). As a result naive customers who believe in such claims are burned.
   
2. Forced upgrades. In case of external providers that platform is owned by the provider and if his financial or technological interests dictate that upgrade is necessary it will be done. That means that customers have to adapt or leave. And upgrade for a provider with large number of customers can be huge mess that cost clients dearly. I experienced this situation myself and can attest that the level of frustration involved is substantial and the mess can easily last several months.
   
3. Compatibility problems. As provider uses specific technology the level of interoperability of this technology with other important for the company technologies is not under the control of the user of such services. It can lead to significant costs due to luck of interoperability. In the simplest example lack of interoperability with Microsoft Office is a serious problem for Sun which uses Open Office (Star Office to be exact).
   
4. The loss of operational flexibility When switch to "in the cloud" provider is done on cost grounds alone, that usually creates new (and initially mostly unrecognized) dependencies that deprive the company from much of the operational flexibility. Typically security departments are direct beneficiary as security-related spending tend to grow dramatically as a defensive reaction of the organism. The main consequence of the bureaucratization is the loss of flexibility, and sooner or later this lack of flexibility can come back and haunt the company.

   In the recent interview Bill Gates noted that "The IT systems are your brain. If you take your brain and outsource it then any adaptability you want (becomes) a contract negotiation". After you negotiated the contact with the "in the cloud" provider any flexibility you used to have is lost as every change explicitly or implicitly become a change of the contact negotiation. Moreover, if the company lost its key IT staff and key architectural issues are decided by outsourcers, then essentially the company becomes a hostage of outsourcers as it no longer has brain power to access the options and the architecture (and thus the costs) are by-and-large controlled by forces outside your control. That is much more serious risk that many PHB assume: the line between architectural decisions and implementation decisions is pretty fuzzy. There is also associated brain-drain risk – if you outsource important functions, you irrevocable erode the capabilities within your firm. When problems arise, internal IT staff can often resolve the problem more quickly, with less bureaucratic overhead inherent when two corporations are involved.

   The fist category of factors that lead to loss of flexibility is connected with additional standards, policies and procedures that are usually introduced in external service provider situation.

   The other important aspect of the loss of remnants of competitive advantage, if any, as the service provider is now the place were the critical mass of know-how and talent pool reside. That somewhat reverses "client-provider" relationship: it is service provider who now can dictate the rules of the game and who is the only party who understands the precise nature of the tasks involved and can conceal this knowledge for his own advantage from the other party. That usually is helped by the demoralization of the remaining staff in the company
   

5. Amplification of the management missteps.

   Their first and most important desire of service provider is to keep the client, even if client's situation changed and no more lend itself easily to the services provided. That can lead to huge architectural blunders. Those things are not limited to offshoring but happened often with complex projects like ERP were external consultants are involved, especially in case big consultant firms are involved. Several large companies went out of business or were bought as a result. Among example we can list FoxMeyer, AT&T Wireless. Several companies were severely hurt (Herschel, etc). This is might actually include Compaq.
   
   As for Compaq, the story is more complex. Dot-com bust hurt "value added" companies like HP and Compaq disproportionally and increased appetite for mergers and acquisitions activities. Dogged determination of Carly Fiorina about the merger (which served both for self-promotion and as a smokescreen for her own problems at struggling HP) and the ability of former Compaq boss, a certified public accountant Michael Capellas, to understand personal benefits from Carly Fiorina proposition proved to be a winning combination. Capellas who proved to be a "serial CEO", , was a big fan of SAP and that probably was a contributed factor is Compaq troubles. When he was appointed, he promised to turn around struggling Compaq in 180 days. Now we know that after just 84 he found a much better financial solution, at least for himself. A shortcut for a turnaround ;-). It is interesting to note that in 2000, based on iPAQ success, he declared[ ComputerWeekly.com,Feb 2000] :

   "There will be an explosion in devices, all attached to the Web. We are going to simplify the PC radically." Capellas promises that future desktop computers will be easier to manage than today's machines.

Large companies have become very adept at establishing remote teams in places like India, Russia, etc. Due to their ability to offer higher wages and better environment they are able to attract local talent and run challenging research and development projects. Often, though, these outposts become disconnected from their parent companies because they fail to establish rapport with key participants in the central office. Foreign prophets are usually ignored. There is something fundamental in this "tragedy of local talent" and communication is still a problem even in the age of videoconferences. Without "warm-body" personal contacts it is difficult to build long-term trust based relationships.

Many organizations who thought outsourcing IT was the key to success miserably failed. Whose who did not failed lost competitive advantage, experienced the demoralizing effect of vendor lockdown and QOC hokey-pokey which just simply made no sense. Some in-sourced the IT back, some recreated it from scratch, are still in denial.

That means that client of service providers will be implicitly pushed to lowest common denominator and cannot utilize local expertise, even if such exists. They face "helpdesk level" people and instead of benefiting from specialized provider are often proposed wrong solutions to misdiagnosed problems. my experience with WEB providers suggests that trivial problems like an error in DNS record or wrong permissions can became real production issues.

Service providers can evolve and upgrade software independently of wishes of some or all of the customers. That means that customers who are not satisfied with the direction taken need either to adapt or abandon the service.

When the cloud evaporates: performance issues and dealing with WAN and provider outages

Public internet is unsuitable for handling large volume of transaction with stringent performance criteria. That means that it is dangerous to put databases at "in the cloud providers" : the more successful "in the cloud" providers are ( or if there just are too many P2P and or multiplayer videogames enthusiasts in the same subnet), the slower your WAN connection will be ("tragedy of commons").

Moreover, in comparison with LAN, WAN-based provision of software services is more complex system and as such is less reliable especially at bottlenecks (which are service provider "entry points" and associated infrastructure (DNS, routers, switches, etc). With WAN outage the situation can became a lot worse then when just when spreadsheets or MS Word documents suddenly are inaccessible on the local server due to LAN outage (but you can still download then into USB stick directly from the server and work with the local copy until network connectivity is restored, because your departmental file server is just several dozens of yards away and friendly administrator probably can help you to get to the data. In case of WAN there is no way to put a USB stick on the server or use other shortcut to avoid effects of network downtime: if WAN connection is down you are really down. Generally not only you can do nothing about the outage, its effects might be amplified by the fact that there are many customers affected. All you will get is the message like this:

The service is experiencing technical difficulties. We apologize for the inconvenience. Please try again later .

That means that in some cases the effect on organization of external outage might be such that the head of the person who enthusiastically recommended company to move "into the cloud" rolls down independent of his position, real or faked IQ and technical competence. Recently both Gmail and Amazon services experienced multiple outages. As Brad Stone noted in NYT:

There is plenty of disappointment to go around these days. Such technology stalwarts as Yahoo, Amazon.com and Research in Motion, the company behind the BlackBerry, have all suffered embarrassing technical problems in the last few months.

About a month ago, a sudden surge of visitors to Mr. Payne’s site began asking about the normally impervious Amazon. That site was ultimately down for several hours over two business days, and Amazon, by some estimates, lost more than a million dollars an hour in sales.

The Web, like any technology or medium, has always been susceptible to unforeseen hiccups. Particularly in the early days of the Web, sites like eBay and Schwab.com regularly went dark.

But since fewer people used the Internet back then, the stakes were much lower. Now the Web is an irreplaceable part of daily life, and Internet companies have plans to make us even more dependent on it.

Companies like Google want us to store not just e-mail online but also spreadsheets, photo albums, sales data and nearly every other piece of personal and professional information. That data is supposed to be more accessible than information tucked away in the office computer or filing cabinet.

The problem is that this ideal requires Web services to be available around the clock — and even the Internet’s biggest companies sometimes have trouble making that happen.

Last holiday season, Yahoo’s system for Internet retailers, Yahoo Merchant Solutions, went dark for 14 hours, taking down thousands of e-commerce companies on one of the busiest shopping days of the year. In February, certain Amazon services that power the sites of many Web start-up companies had a day of intermittent failures, knocking many of those companies offline.

The causes of these problems range widely: it might be system upgrades with unintended consequences, human error (oops, wrong button) or even just old-fashioned electrical failures. Last month, an electrical explosion in a Houston data center of the Planet, a Web hosting company, knocked thousands of Web businesses off the Internet for up to five days.

“It was prolonged torture,” said Grant Burhans, a Web entrepreneur from Florida whose telecommunications- and real-estate-related Web sites were down for four days, costing him thousands of dollars in lost business.

I was actually surprised how much posts each "in the cloud" service outage generates and how significant were losses reported by some users; in addition to Official Gmail Blog one of the best places to track Gmail outages proved to be Twitter; there is also a site http://downforeveryoneorjustme.com/ which provides a free check for frustrated Gmail and other "in the cloud" services users). Some reactions are pretty funny:

• Thank you so much for coming back. I felt lonely without your constant, vigilant presence.
• I was so relieved Gmail was down. I thought my employer decided to block it on my berry.
• Gmail ruined my day. Damned cloud!
• Gmail is really pissing me off today, which doesn't happen often. S'like having your first real fight in a relationship. Very disconcerting!
• Gmail's back, but now my IndyChristian provider is down. And Facebook.com . Go figure.
• Thank God I never jumped aboard the evil gmail ship!
• To all Gmail outage whiners: if you want the right to bitch, don't use free email. Also, look up the definition of "BETA" sometime.
• Very comforting to know that I wasn't the only one that had no Gmail for a bit there. I'm not alone in this world? Long but good day.
• Man, gmail goes down for one moment and the whole world is up in arms :)
• How did I miss the gmail outage? I feel so out of the loop!
• the weather is hot, the a/c is b0rked, gmail was down this afternoon. expect to see four horsemen walk by any minute...
• Sorry, completely missed the gmail outage because I was out biking and enjoying the amazing evening we're having here.
• Having Gmail down about made me bang my head on the table. Oy.
• Boycotting gmail and getting behind the olympics
• For Twitterers, Gmail going down is like JFK being shot, innit?
• Ignorance IS bliss-so busy today I did not even know about the great gmail outage of '08.
• The cloud of unreliability: It's not clear why anyone should be surprised that Gmail, Amazon.com's cloud .. http://tinyurl.com/649amj
• Was thinking about going to back to Gmail from .Mac since MobileMe was down. Now I hear Gmail was down. Thinking about envelopes and stamps
• so mad at gmail for 502-ing her earlier today. what did we do before google?
• With Gmail having gone down and me not noticing, I am undiagnosing myself of Internet Addiction and OCD.
• Maybe gmail is down to express solidarity with MobileMe?

As any self-respecting obsessive business e-mail checker could tell you, each outage is a real shock and fails on the most inopportune moment. In reality most email outages does not make users less productive, they just deprive them from their favorite tool of wasting own and other time and procrastination ;-)

Here is a short but pretty sobering list for 2008. It neither complete not contains the most important posts about particular outage.

[Jan 28, 2008] Gmail OUTAGE, relying on Google too much - TECH.BLORGE.com

Google services are usually rock solid reliable, but earlier today some Gmail users lost service for a couple of hours. That begs the question, are we relying on Google too much?

Gmail outages are peppered over the last few years and every time it creates a flurry of activity as businesses and individuals realize how much they rely on a single business and platform for all of their messaging needs.

The same concept is mirrored with search engine traffic, where many people don’t even realize there are search engines out there other than Google.

[Feb 16, 2008] Amazon explains its S3 outage Between the Lines ZDNet.com

Amazon has issued a statement that adds a little more clarity to its Web services outage on Friday.

Here’s Amazon’s explanation of the S3 outage, which wreaked havoc on startups and other enterprises relying on Amazon’s cloud.

Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.

Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.

As we said earlier today, though we’re proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable. As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements. We are taking immediate action on the following: (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls. Additionally, we’ve begun work on a service health dashboard, and expect to release that shortly.

Sincerely,
The Amazon Web Services Team

[Jun 9, 2008] Surviving gMail 502 outages SKFox

I use mint to see live stats for the traffic to skfox.com and Google Analytics for the long term statistics. I’ve seen a resurgence today of traffic to my two blog posts (in March and again in May) about my gMail account being down. The most common for me is a “Temporary Error (502)” and it would seem to be happening to other users too. I hate to be the bearer of bad news, but there isn’t a whole lot you can do about it. On the short side, most of the outages only last 30 minutes or so with the longest being 3.5 hours. It can be terribly frustrating, but there are a few things you can do to alleviate the pain.

• Use your email client like Outlook or Thunderbird to download your messages to your local machine with POP3 while keeping your gmail account unchanged. That way, even if gMail is inaccessible, you can get to your old email for reference. Of course, you have to do this once your account is up and running again.
• Visit the gMail Google group to at the least find others in your boat and get the latest updates.

Some users have reported success getting to their mail during outages by using any number of alternative links to gmail. Your mileage may vary, but here they are:

• Basic HTML version: https://mail.google.com/mail/h/
• Basic HTML version #2: http://mail.google.com/mail/?ui=html
• Older version: http://mail.google.com/mail/?ui=1

Today’s outage is a known issue, and I hope for all of your coming here from search engines, that it comes back up quickly for you.

[Jul 20, 2008] Amazon's S3 Storage Outage Busts Web Sites, Crashes iPhone Apps

What happened? Sometime this morning, Amazon's (AMZN) S3 storage service went down. Or, according to Amazon's official note, S3 is "experiencing elevated error rates." A lot of companies -- Twitter, 37signals, etc. -- rely on S3 to host static files like images, style sheets, etc. for their Web apps. So when S3 goes down, those sites lose some/most/all functionality, depending on what the company uses S3 for.

So how'd Amazon's storage service bork my iPhone? Tapulous relies on S3 for images like Twinkle user icons. And they must not have included a "plan B" in their code to handle an image server outage. So when S3 hiccuped, and Twinkle couldn't download images, the app crashed, taking me back to the iPhone home screen. (And, hours later, it's still crashing.)

[Aug 11, 2008] TheStar.com Business Google Gmail users having troubles

SEATTLE–Google Inc said Monday that many users of its Gmail service are having trouble accessing their online e-mail accounts due to a temporary outage in a contacts system used by Gmail.

The problems began at about 5 p.m., and a company spokesman said Google is starting to implement a fix for the problem right now.

"We are starting to roll out a fix now and hope to have the problem resolved as quickly as possible," said a Google spokesman.

[Aug 25, 2008] FOXNews.com - Amazon's Site Outage Raises Scalability Questions - Science News Science & Technology Technology News

Amazon.com has built a reputation for being an aggressive, take-no-prisoners kind of company, but it showed its more cooperative side this week.

Late last week, Internet traffic monitoring firm Keynote Systems issued a report that said many of the largest e-commerce players will face major load-handling challenges for the holidays if they don't make big changes by the fourth quarter.

However, after so many years of holiday shopping seasons, some in the industry scoffed that the majors could be caught so short.

To help out, Amazon (AMZN) generously knocked out its own servers for almost 2 hours on Aug. 21.

OK, OK. Amazon probably didn't crash just to help Keynote make a point. But its timing was impeccable nonetheless.

It's interesting to note that within a few days this month, three of the industry's most powerful retail forces — Wal-Mart (WMT), Amazon and Google (GOOG) — all suffered problems that, one way or the other, can be classified as scalability-related.

But the most important difference between "in the cloud" services and local services is not duration or frequency of the outages. The most important is the level and quality of information available about the situation. In case of local services all information about the situation is readily available and thus some countermeasures can be taken. In military world such difference is of paramount important. IT is not the different in this respect from military world. In case of "in the cloud" the amount and quality of information about the outage are very low; services customers are essentially in the dark. Services just abruptly seizes to exists and then magically comes back. This lack of information has its own costs.

Virtualization generally magnifies failures. In the physical world, a server failure typically would mean that backup servers would step in to prevent downtime. In the virtual world, depending on the number of virtual machines residing on a physical box, a hardware failure could impact multiple virtual servers and the applications they host. As a result failures have a much larger impact, effecting multiple applications and multiple users. Little fires turn into big fires. Virtualization might also increase two other problems:

• Performance-related problems. Companies ensure top performance of critical applications using powerful dedicated servers, network and storage resources for those applications, segmenting them from other traffic to ensure they get the resources they need. "In the cloud" provider model of solving those problems is based on allocation of additional resources on demand as his goal is to minimize cost of infrastructure. That means that at any given time, performance of an application could degrade, perhaps not to a failure, but to a crawl making users less productive or paralyzed.
  
• Troubleshooting complexity. Server problems in the past could be limited to one box, but now the problem can be related to hypervisor, not the server per se or move with the application instance from on virtual machine to another. If problem existed on one server, but disappears after application is moved to another and later reappears the question arise: "Is the problem fixed?" Often temporary disappearance of the problem lull the staff into a vicious activity cycle and they start to transfer the problem around from virtual server to virtual server instead of trying to solve it

All-in-all the more a particular company relies on "in the cloud computing", the more severe will be effect of the outages. And in comparison with traditional local LAN-based "physical server" deployment there are several sources of them:

• In the cloud provider outages: upgrades, equipment replacements, DNS blunders (a very popular type of blunder ;-), blunders of personnel, problems connected with the personnel incompetence, etc.
• Links from in the cloud provider to ISP
• Problems in ISP (again here there are issues of upgrades, blunders of personnel, physical damages, DNS problems, etc)
• Problems in the link from ISP to the company
• Virtualization-related problems, if it is used by "in the cloud" provider.

The first thing you notice during large outage of "in the cloud" service provider is that customer service is overloaded and you need to wait a long time to get to the human voice. There are usually just too many angry and frustrated customers before you. Also bad is that the nature of the problem and the estimate of the time needed to resolve is usually are not communicated, keeping you in the dark. Moreover even after you get to a support person the information you get will be sketchy. Here is one example:

I was working on a client webinar using WebEx Event Center. I tried to log onto our site and got a cryptic message saying the site was unavailable. Whammo. I got on the line to technical support and waited my turn in queue, along with what must have been quite a few other panicked or angry customers. My support person was surprisingly calm and good natured, given the call volume they must have been processing. He confirmed that there were network problems on their side and said that while he didn't know details of the problem, he was certain it would be all right soon. I had a little time before our event, so I agreed to try again later.

Sure enough, our site came up, but on a backup network. This wasn't completely clean, as one of our scheduled future events had disappeared from our registration page, meaning people couldn't sign up. But at least we could carry on with today's show. Unfortunately, performance was quite a bit below normal standards, with slides and annotations taking inconsistent and sometimes very long time lags to appear on participants' computers.

After the event, strange behavior continued, with an inability to access post-event reports through the WebEx administrator interface. I called technical support again and got agreement that it certainly was strange behavior and we should all hope it would correct itself once the system was back to normal again. Whenever that might be.

Now, I'm not singling out WebEx for any particular acrimonious treatment here. I happened to be using them when this problem occurred. Any provider can have a similar problem. At least WebEx had a backup network plan in place and we were technically able to carry on with our scheduled event. But it's worth noting that there is a frustrating sense of powerlessness while a problem like this is going on. You can't prod anybody for more details, because you don't have access to the people trying to fix the problem as you might if a similar situation occurred with your own business network. You can't get much in the way of progress updates. And you can't put your own backup plan into effect.

One interesting difference in "horror stories" between local and "in the cloud" datacenter was aptly outlined in the following Computerworld quote:

Just after midnight on Jan. 22, 2006, Con Edison began telling the Internet that it was Martha Stewart. That is, Con Edison erroneously began sending out routing information to redirect Internet traffic intended for Martha Stewart Living Omnimedia to its own servers.

The publishing company was a Con Edison customer. So were other businesses and Internet providers whose routing information Con Edison hijacked at the same time. The result was a mess that wasn't completely cleared up for 18 hours — and some businesses were offline for most of that time.

But not Martha Stewart, whose CIO, Mike Plonski, wrote to me to clarify what happened at his company.

Plonski's secret sauce? No big secret — just network monitoring and redundancy.

Plonski said: "While one of the Internet connections at our corporate offices was impacted by the ConEd issue you describe, we, as a company, are smart enough to have employed redundancy, both by location and carrier, for our network operations. As a result, during this time frame, we simply flipped all of our Internet traffic to run over our secondary line until ConEd resolved their issue."

OK, it was a little more complicated than that. Plonski said his staff spotted the problem through routine network monitoring. There was clearly something wrong with network traffic coming to the corporate office over the Con Edison line. Thanks to the monitoring, the company knew about the problem about 30 minutes after it started.

Because of the type of outage, an IT staffer had to connect and manually switch over to a redundant line. That took another hour.

Total time for the outage: about 90 minutes in the wee hours of a Sunday morning. Total impact: minimal.

An outage? Yes. A knockout? No way. And handling the problem didn't require rocket science — just monitoring, redundancy and sharp IT staff work.

Repeat after me: [in case of local datacenters "handling the problem didn't require rocket science — just monitoring, redundancy and sharp IT staff work."

Interoperability problems

While individual "in the cloud" service provider can do a decent job in providing the required functionality, the problem arise when you need to use several such providers and issues of interoperability arise.

The lack of interoperability between different SaaS applications is one of the most well known and, at the same time, largest problems with SaaS. In a way companies who are using several different providers spending way too much money with the side effect of creating a problematic, cumbersome IT architecture that lack flexibility and efficiency and as such is inferior to the integrated datacenter architecture. The focus of vendors is on the adaptation of the SaaS model to enterprise requirements (enterprise-level integration), but there is a growing need for vendor-to-vendor integration. How and when those needs will be addressed remains to be seen.

In a way SaaS application emphasize too much GUI portion of functionality of application at the expense of the ability of smoothly exchange data with other, often similar applications.

The emergence of a several classes of enterprise SaaS (for email, CRM, supply chain management, benefits and other HR-related applications, etc ) creates problems of similar or identical data existing in various formats in different providers and their synchronization and timely updates. It providers does not share common "hardware cloud" we have a huge problems as not only protocols of data exchange, but reliability and security issues are pretty complex.

Also the existing interoperability can be broken anytime by software updates by one of several for existing "in the cloud" providers.

NEWS CONTENTS

• 20210612 : A Big Chunk of the Internet Goes Offline Because of a Faulty CDN Provider ( Jun 10, 2021 , tech.slashdot.org )
• 20210608 : Technical Evaluations- 6 questions to ask yourself ( Apr 21, 2021 , www.redhat.com )
• 20210503 : Do You Replace Your Server Or Go To The Cloud- The Answer May Surprise You ( May 03, 2021 , www.forbes.com )
• 20210305 : Edge servers can be strategically placed within the topography of a network to reduce the latency of connecting with them and serve as a buffer to help mitigate overloading a data center ( Mar 05, 2021 , opensource.com )
• 20200329* Why Didn't We Test Our Trade's 'Antifragility' Before COVID-19 by Gene Callahan and Joe Norman ( Mar 28, 2020 , www.theamericanconservative.com ) [Recommended]
• 20200305 : Micro data center ( Mar 05, 2020 , en.wikipedia.org )
• 20200305 : What's next for data centers Think micro data centers by Larry Dignan ( Apr 14, 2019 , www.zdnet.com )
• 20191108 : Multiple Linux sysadmins working as root ( May 21, 2012 , serverfault.com )
• 20191022 : Bank of America Says It Saves $2 Billion Per Year By Ignoring Amazon and Microsoft and Building Its Own Cloud Instead ( Oct 22, 2019 , slashdot.org )
• 20190822 : Why Micro Data Centers Deliver Good Things in Small Packages by Calvin Hennick ( Aug 22, 2019 , solutions.cdw.com )
• 20181113 : GridFTP : User s Guide ( toolkit.globus.org )
• 20181112 : Edge Computing vs. Cloud Computing What's the Difference by Andy Patrizio , ( Jan 23, 2018 , www.datamation.com )
• 20181109 : Cloud-hosted date must be accessed by users over existing WAN which creates performance issues due to bandwidth and latency constraints ( Nov 09, 2018 , www.eiseverywhere.com )
• 20181109 : Troubleshoot WAN Performance Issues SD Wan Experts by Steve Garson ( Feb 08, 2013 , www.sd-wan-experts.com )
• 20181109 : Storage in private clouds ( Nov 09, 2018 , www.redhat.com )
• 20181109 : Cloud Computing vs Edge Computing Which Will Prevail ( Nov 09, 2018 , www.lannerinc.com )
• 20181109 : What is Hybrid Cloud Computing ( Nov 09, 2018 , www.dummies.com )
• 20181109 : Why Micro Data Centers Deliver Good Things in Small Packages by Calvin Hennick ( Nov 09, 2018 , solutions.cdw.com )
• 20181109 : Solving Office 365 and SaaS Performance Issues with SD-WAN ( Nov 09, 2018 , www.brighttalk.com )
• 20181109 : Make sense of edge computing vs. cloud computing ( Nov 09, 2018 , www.infoworld.com )
• 20181108 : GT 6.0 GridFTP ( Nov 08, 2018 , toolkit.globus.org )
• 20181108 : globus-gridftp-server-control-6.2-1.el7.x86_64.rpm ( Nov 08, 2018 , centos.pkgs.org )
• 20181108 : 9 Aspera Sync Alternatives Top Best Alternatives ( Nov 08, 2018 , www.topbestalternatives.com )
• 20181103 : Technology is dominated by two types of people; those who understand what they don t manage; and those who manage what they don t understand – ARCHIBALD PUTT ( PUTTS LAW ) ( Nov 03, 2018 , brummieruss.wordpress.com )
• 20181030 : Cloud has a rich future, but I didn't even know Red Hat had any presence there, let alone $35 billion worth. ( Oct 30, 2018 , arstechnica.com )
• 20181027 : We are effectively back to the centralized systems ("clouds" controlled by Amazon (CIA), Google (NSA)). The peripheral components of autonomous networked systems are akin to sensory devices with heavy computational lifting in the (local) regime controlled servers. ( Oct 27, 2018 , www.moonofalabama.org )
• 20170916 : Google Drive Faces Outage, Users Report ( Sep 16, 2017 , tech.slashdot.org )
• 20170702 : The Details About the CIAs Deal With Amazon by Frank Konkel ( Jul 17, 2014 , www.theatlantic.com )
• 20170609 : Amazon's S3 web-based storage service is experiencing widespread issues on Feb 28 2017 ( Jun 09, 2017 , techcrunch.com )
• 20170401 : Amazon Web Services outage causes widespread internet problems ( Apr 01, 2017 , www.cbsnews.com )
• 20170401 : After Amazon outage, HealthExpense worries about cloud lock-in by Maria Korolov ( Apr 01, 2017 , www.networkworld.com )
• 20170303 : Do You Replace Your Server Or Go To The Cloud The Answer May Surprise You ( Mar 03, 2017 , www.forbes.com )
• 20170112 : Digitopoly Congestion on the Last Mile ( Jan 12, 2017 , www.digitopoly.org )
• 20151109 : Thoughts on the Amazon outage ( nickgeoghegan.net )
• 20151109 : MAT Monitoring and Administration Tool ( MAT Monitoring and Administration Tool, )
• 20151109 : Unix SysAdm Resources Automated Unix SysMgmt Software ( Unix SysAdm Resources Automated Unix SysMgmt Software , )

Old News ;-)

[Jun 12, 2021] A Big Chunk of the Internet Goes Offline Because of a Faulty CDN Provider

Jun 10, 2021 | tech.slashdot.org

(techcrunch.com) 154 Countless popular websites including Reddit, Spotify, Twitch, Stack Overflow, GitHub, gov.uk, Hulu, HBO Max, Quora, PayPal, Vimeo, Shopify, Stripe, and news outlets CNN, The Guardian, The New York Times, BBC and Financial Times are currently facing an outage . A glitch at Fastly, a popular CDN provider, is thought to be the reason, according to a product manager at Financial Times. Fastly has confirmed it's facing an outage on its status website.

[Jun 08, 2021] Technical Evaluations- 6 questions to ask yourself

Average but still useful enumeration of factors what should be considered. One question stands out "Is that SaaS app really cheaper than more headcount?" :-)

Notable quotes:

"... You may decide that this is not a feasible project for the organization at this time due to a lack of organizational knowledge around containers, but conscientiously accepting this tradeoff allows you to put containers on a roadmap for the next quarter. ..."

"... Bells and whistles can be nice, but the tool must resolve the core issues you identified in the first question. ..."

"... Granted, not everything has to be a cost-saving proposition. Maybe it won't be cost-neutral if you save the dev team a couple of hours a day, but you're removing a huge blocker in their daily workflow, and they would be much happier for it. That happiness is likely worth the financial cost. Onboarding new developers is costly, so don't underestimate the value of increased retention when making these calculations. ..."

Apr 21, 2021 | www.redhat.com

When introducing a new tool, programming language, or dependency into your environment, what steps do you take to evaluate it? In this article, I will walk through a six-question framework I use to make these determinations.

What problem am I trying to solve?

We all get caught up in the minutiae of the immediate problem at hand. An honest, critical assessment helps divulge broader root causes and prevents micro-optimizations.

[ You might also like: Six deployment steps for Linux services and their related tools ]

Let's say you are experiencing issues with your configuration management system. Day-to-day operational tasks are taking longer than they should, and working with the language is difficult. A new configuration management system might alleviate these concerns, but make sure to take a broader look at this system's context. Maybe switching from virtual machines to immutable containers eases these issues and more across your environment while being an equivalent amount of work. At this point, you should explore the feasibility of more comprehensive solutions as well. You may decide that this is not a feasible project for the organization at this time due to a lack of organizational knowledge around containers, but conscientiously accepting this tradeoff allows you to put containers on a roadmap for the next quarter.

This intellectual exercise helps you drill down to the root causes and solve core issues, not the symptoms of larger problems. This is not always going to be possible, but be intentional about making this decision.

In the cloud

• Understanding cloud computing
• Free course: Red Hat OpenStack Technical Overview
• Free e-book: Hybrid Cloud Strategy for Dummies

Does this tool solve that problem?

Now that we have identified the problem, it is time for critical evaluation of both ourselves and the selected tool.

A particular technology might seem appealing because it is new because you read a cool blog post about it or you want to be the one giving a conference talk. Bells and whistles can be nice, but the tool must resolve the core issues you identified in the first question.

What am I giving up?

The tool will, in fact, solve the problem, and we know we're solving the right problem, but what are the tradeoffs?

These considerations can be purely technical. Will the lack of observability tooling prevent efficient debugging in production? Does the closed-source nature of this tool make it more difficult to track down subtle bugs? Is managing yet another dependency worth the operational benefits of using this tool?

Additionally, include the larger organizational, business, and legal contexts that you operate under.

Are you giving up control of a critical business workflow to a third-party vendor? If that vendor doubles their API cost, is that something that your organization can afford and is willing to accept? Are you comfortable with closed-source tooling handling a sensitive bit of proprietary information? Does the software licensing make this difficult to use commercially?

While not simple questions to answer, taking the time to evaluate this upfront will save you a lot of pain later on.

Is the project or vendor healthy?

This question comes with the addendum "for the balance of your requirements." If you only need a tool to get your team over a four to six-month hump until Project X is complete, this question becomes less important. If this is a multi-year commitment and the tool drives a critical business workflow, this is a concern.

When going through this step, make use of all available resources. If the solution is open source, look through the commit history, mailing lists, and forum discussions about that software. Does the community seem to communicate effectively and work well together, or are there obvious rifts between community members? If part of what you are purchasing is a support contract, use that support during the proof-of-concept phase. Does it live up to your expectations? Is the quality of support worth the cost?

Make sure you take a step beyond GitHub stars and forks when evaluating open source tools as well. Something might hit the front page of a news aggregator and receive attention for a few days, but a deeper look might reveal that only a couple of core developers are actually working on a project, and they've had difficulty finding outside contributions. Maybe a tool is open source, but a corporate-funded team drives core development, and support will likely cease if that organization abandons the project. Perhaps the API has changed every six months, causing a lot of pain for folks who have adopted earlier versions.

What are the risks?

As a technologist, you understand that nothing ever goes as planned. Networks go down, drives fail, servers reboot, rows in the data center lose power, entire AWS regions become inaccessible, or BGP hijacks re-route hundreds of terabytes of Internet traffic.

Ask yourself how this tooling could fail and what the impact would be. If you are adding a security vendor product to your CI/CD pipeline, what happens if the vendor goes down?

Kubernetes and OpenShift

• Free cheatsheet: Kubernetes and Minikube
• Free ebook: Designing Cloud-Native Applications
• Interactive course: Getting Started with OpenShift
• Free ebook: Build Applications with Kubernetes and Openshift

This brings up both technical and business considerations. Do the CI/CD pipelines simply time out because they can't reach the vendor, or do you have it "fail open" and allow the pipeline to complete with a warning? This is a technical problem but ultimately a business decision. Are you willing to go to production with a change that has bypassed the security scanning in this scenario?

Obviously, this task becomes more difficult as we increase the complexity of the system. Thankfully, sites like k8s.af consolidate example outage scenarios. These public postmortems are very helpful for understanding how a piece of software can fail and how to plan for that scenario.

What are the costs?

The primary considerations here are employee time and, if applicable, vendor cost. Is that SaaS app cheaper than more headcount? If you save each developer on the team two hours a day with that new CI/CD tool, does it pay for itself over the next fiscal year?

Granted, not everything has to be a cost-saving proposition. Maybe it won't be cost-neutral if you save the dev team a couple of hours a day, but you're removing a huge blocker in their daily workflow, and they would be much happier for it. That happiness is likely worth the financial cost. Onboarding new developers is costly, so don't underestimate the value of increased retention when making these calculations.

[ A free guide from Red Hat: 5 steps to automate your business . ]

Wrap up

I hope you've found this framework insightful, and I encourage you to incorporate it into your own decision-making processes. There is no one-size-fits-all framework that works for every decision. Don't forget that, sometimes, you might need to go with your gut and make a judgment call. However, having a standardized process like this will help differentiate between those times when you can critically analyze a decision and when you need to make that leap.

[May 03, 2021] Do You Replace Your Server Or Go To The Cloud- The Answer May Surprise You

May 03, 2021 | www.forbes.com

Is your server or servers getting old? Have you pushed it to the end of its lifespan? Have you reached that stage where it's time to do something about it? Join the crowd. You're now at that decision point that so many other business people are finding themselves this year. And the decision is this: do you replace that old server with a new server or do you go to: the cloud.

Everyone's talking about the cloud nowadays so you've got to consider it, right? This could be a great new thing for your company! You've been told that the cloud enables companies like yours to be more flexible and save on their IT costs. It allows free and easy access to data for employees from wherever they are, using whatever devices they want to use. Maybe you've seen the recent survey by accounting software maker MYOB that found that small businesses that adopt cloud technologies enjoy higher revenues. Or perhaps you've stumbled on this analysis that said that small businesses are losing money as a result of ineffective IT management that could be much improved by the use of cloud based services. Or the poll of more than 1,200 small businesses by technology reseller CDW which discovered that " cloud users cite cost savings, increased efficiency and greater innovation as key benefits" and that " across all industries, storage and conferencing and collaboration are the top cloud services and applications."

So it's time to chuck that old piece of junk and take your company to the cloud, right? Well just hold on.

There's no question that if you're a startup or a very small company or a company that is virtual or whose employees are distributed around the world, a cloud based environment is the way to go. Or maybe you've got high internal IT costs or require more computing power. But maybe that's not you. Maybe your company sells pharmaceutical supplies, provides landscaping services, fixes roofs, ships industrial cleaning agents, manufactures packaging materials or distributes gaskets. You are not featured in Fast Company and you have not been invited to presenting at the next Disrupt conference. But you know you represent the very core of small business in America. I know this too. You are just like one of my company's 600 clients. And what are these companies doing this year when it comes time to replace their servers?

These very smart owners and managers of small and medium sized businesses who have existing applications running on old servers are not going to the cloud. Instead, they've been buying new servers.

Wait, buying new servers? What about the cloud?

At no less than six of my clients in the past 90 days it was time to replace servers. They had all waited as long as possible, conserving cash in a slow economy, hoping to get the most out of their existing machines. Sound familiar? But the servers were showing their age, applications were running slower and now as the companies found themselves growing their infrastructure their old machines were reaching their limit. Things were getting to a breaking point, and all six of my clients decided it was time for a change. So they all moved to cloud, right?

PROMOTED

https://642be7d830a988d07ed5dd23076ca4e7.safeframe.googlesyndication.com/safeframe/1-0-38/html/container.html

https://642be7d830a988d07ed5dd23076ca4e7.safeframe.googlesyndication.com/safeframe/1-0-38/html/container.html

https://642be7d830a988d07ed5dd23076ca4e7.safeframe.googlesyndication.com/safeframe/1-0-38/html/container.html

Nope. None of them did. None of them chose the cloud. Why? Because all six of these small business owners and managers came to the same conclusion: it was just too expensive. Sorry media. Sorry tech world. But this is the truth. This is what's happening in the world of established companies.

Consider the options. All of my clients' evaluated cloud based hosting services from Amazon , Microsoft and Rackspace . They also interviewed a handful of cloud based IT management firms who promised to move their existing applications (Office, accounting, CRM, databases) to their servers and manage them offsite. All of these popular options are viable and make sense, as evidenced by their growth in recent years. But when all the smoke cleared, all of these services came in at about the same price: approximately $100 per month per user. This is what it costs for an existing company to move their existing infrastructure to a cloud based infrastructure in 2013. We've got the proposals and we've done the analysis.

You're going through the same thought process, so now put yourself in their shoes. Suppose you have maybe 20 people in your company who need computer access. Suppose you are satisfied with your existing applications and don't want to go through the agony and enormous expense of migrating to a new cloud based application. Suppose you don't employ a full time IT guy, but have a service contract with a reliable local IT firm.

Now do the numbers: $100 per month x 20 users is $2,000 per month or $24,000 PER YEAR for a cloud based service. How many servers can you buy for that amount? Imagine putting that proposal out to an experienced, battle-hardened, profit generating small business owner who, like all the smart business owners I know, look hard at the return on investment decision before parting with their cash.

For all six of these clients the decision was a no-brainer: they all bought new servers and had their IT guy install them. But can't the cloud bring down their IT costs? All six of these guys use their IT guy for maybe half a day a month to support their servers (sure he could be doing more, but small business owners always try to get away with the minimum). His rate is $150 per hour. That's still way below using a cloud service.

No one could make the numbers work. No one could justify the return on investment. The cloud, at least for established businesses who don't want to change their existing applications, is still just too expensive.

Please know that these companies are, in fact, using some cloud-based applications. They all have virtual private networks setup and their people access their systems over the cloud using remote desktop technologies. Like the respondents in the above surveys, they subscribe to online backup services, share files on DropBox and Microsoft 's file storage, make their calls over Skype, take advantage of Gmail and use collaboration tools like Google Docs or Box. Many of their employees have iPhones and Droids and like to use mobile apps which rely on cloud data to make them more productive. These applications didn't exist a few years ago and their growth and benefits cannot be denied.

Paul-Henri Ferrand, President of Dell North America, doesn't see this trend continuing. "Many smaller but growing businesses are looking and/or moving to the cloud," he told me. "There will be some (small businesses) that will continue to buy hardware but I see the trend is clearly toward the cloud. As more business applications become more available for the cloud, the more likely the trend will continue."

He's right. Over the next few years the costs will come down. Your beloved internal application will become out of date and your only option will be to migrate to a cloud based application (hopefully provided by the same vendor to ease the transition). Your technology partners will help you and the process will be easier, and less expensive than today. But for now, you may find it makes more sense to just buy a new server. It's OK. You're not alone.

Besides Forbes, Gene Marks writes weekly for The New York Times and Inc.com .

Related on Forbes:

[Mar 05, 2021] Edge servers can be strategically placed within the topography of a network to reduce the latency of connecting with them and serve as a buffer to help mitigate overloading a data center

Mar 05, 2021 | opensource.com

... Edge computing is a model of infrastructure design that places many "compute nodes" (a fancy word for a server ) geographically closer to people who use them most frequently. It can be part of the open hybrid-cloud model, in which a centralized data center exists to do all the heavy lifting but is bolstered by smaller regional servers to perform high frequency -- but usually less demanding -- tasks...

Historically, a computer was a room-sized device hidden away in the bowels of a university or corporate head office. Client terminals in labs would connect to the computer and make requests for processing. It was a centralized system with access points scattered around the premises. As modern networked computing has evolved, this model has been mirrored unexpectedly. There are centralized data centers to provide serious processing power, with client computers scattered around so that users can connect. However, the centralized model makes less and less sense as demands for processing power and speed are ramping up, so the data centers are being augmented with distributed servers placed on the "edge" of the network, closer to the users who need them.

The "edge" of a network is partly an imaginary place because network boundaries don't exactly map to physical space. However, servers can be strategically placed within the topography of a network to reduce the latency of connecting with them and serve as a buffer to help mitigate overloading a data center.

... ... ...

While it's not exclusive to Linux, container technology is an important part of cloud and edge computing. Getting to know Linux and Linux containers helps you learn to install, modify, and maintain "serverless" applications. As processing demands increase, it's more important to understand containers, Kubernetes and KubeEdge , pods, and other tools that are key to load balancing and reliability.

... ... ...

The cloud is largely a Linux platform. While there are great layers of abstraction, such as Kubernetes and OpenShift, when you need to understand the underlying technology, you benefit from a healthy dose of Linux knowledge. The best way to learn it is to use it, and Linux is remarkably easy to try . Get the edge on Linux so you can get Linux on the edge.

[Mar 29, 2020] Why Didn't We Test Our Trade's 'Antifragility' Before COVID-19 by Gene Callahan and Joe Norman

Highly recommended!

Mar 28, 2020 | www.theamericanconservative.com

On April 21, 2011, the region of Amazon Web Services covering eastern North America crashed. The crash brought down the sites of large customers such as Quora, Foursquare, and Reddit. It took Amazon over a week to bring its system fully back online, and some customer data was lost permanently.

But one company whose site did not crash was Netflix. It turns out that Netflix had made themselves "antifragile" by employing software they called "Chaos Monkey," which regularly and randomly brought down Netflix servers. By continually crashing their own servers, Netflix learned how to nevertheless keep other portions of their network running. And so when Amazon US-East crashed, Netflix ran on, unfazed.

This phenomenon is discussed by Nassim Taleb in his book Antifragile : a system that depends on the absence of change is fragile. The companies that focused on keeping all of their servers up and running all the time went completely offline when Amazon crashed from under them. But the company that had exposed itself to lots of little crashes could handle the big crash. That is because the minor, "undesirable" changes stress the system in a way that can make it stronger.

The idea of antifragility does not apply only to computer networks. For instance, by trying to eliminate minor downturns in the economy, central bank policy can make that economy extremely vulnerable to a major recession. Running only on treadmills or tracks makes the joints extremely vulnerable when, say, one steps in a pothole in the sidewalk.

What does this have to do with trade policy? For many reasons, such as the recent coronavirus outbreak, flows of goods are subject to unexpected shocks.

Both a regime of "unfettered" free trade, and its opposite, that of complete autarchy, are fragile in the face of such shocks. A trade policy aimed not at complete free trade or protectionism, but at making an economy better at absorbing and adapting to rapid change, is more sane and salutary than either extreme. Furthermore, we suggest practicing for shocks can help make an economy antifragile.

Amongst academic economists, the pure free-trade position is more popular. The case for international trade, absent the artificial interference of government trade policy, is generally based upon the "principle of comparative advantage," first formulated by the English economist David Ricardo in the early 19th century. Ricardo pointed out, quite correctly, that even if, among two potential trading partners looking to trade a pair of goods, one of them is better at producing both of them, there still exist potential gains from trade -- so long as one of them is relatively better at producing one of the goods, and the other (as a consequence of this condition) relatively better at producing the other. For example, Lebron James may be better than his local house painter at playing basketball, and at painting houses, given his extreme athleticism and long reach. But he is so much more "better" at basketball that it can still make sense for him to concentrate on basketball and pay the painter to paint his house.

And so, per Ricardo, it is among nations: even if, say, Sweden can produce both cars and wool sweaters more efficiently than Scotland, if Scotland is relatively less bad at producing sweaters than cars, it still makes sense for Scotland to produce only wool sweaters, and trade with Sweden for the cars it needs.

When we take comparative advantage to its logical conclusion at the global scale, it suggests that each agent (say, nation) should focus on one major industry domestically and that no two agents should specialize in the same industry. To do so would be to sacrifice the supposed advantage of sourcing from the agent who is best positioned to produce a particular good, with no gain for anyone.

Good so far, but Ricardo's case contains two critical hidden assumptions: first, that the prices of the goods in question will remain more or less stable in the global marketplace, and second that the availability of imported goods from specialized producers will remain uninterrupted, such that sacrificing local capabilities for cheaper foreign alternatives.

So what happens in Scotland if the Swedes suddenly go crazy for yak hair sweaters (produced in Tibet) and are no longer interested in Scottish sweaters at all? The price of those sweaters crashes, and Scotland now finds itself with most of its productive capacity specialized in making a product that can only be sold at a loss.

Or what transpires if Scotland is no longer able, for whatever reason, to produce sweaters, but the Swedes need sweaters to keep warm? Swedes were perhaps once able to make their own sweaters, but have since funneled all their resources into making cars, and have even lost the knowledge of sweater-making. Now to keep warm, the Swedes have to rapidly build the infrastructure and workforce needed to make sweaters, and regain the knowledge of how to do so, as the Scots had not only been their sweater supplier, but the only global sweater supplier.

So we see that the case for extreme specialization, based on a first-order understanding of comparative advantage, collapses when faced with a second-order effect of a dramatic change in relative prices or conditions of supply.

That all may sound very theoretical, but collapses due to over-specialization, prompted by international agencies advising developing economies based on naive comparative-advantage analysis, have happened all too often. For instance, a number of African economies, persuaded to base their entire economy on a single good in which they had a comparative advantage (e.g, gold, cocoa, oil, or bauxite), saw their economies crash when the price of that commodity fell. People who had formerly been largely self-sufficient found themselves wage laborers for multinationals in good times, and dependents on foreign charity during bad times.

While the case for extreme specialization in production collapses merely by letting prices vary, it gets even worse for the "just specialize in the single thing you do best" folks once we add in considerations of pandemics, wars, extreme climate change, and other such shocks. We have just witnessed how relying on China for such a high percentage of our medical supplies and manufacturing has proven unwise when faced with an epidemic originating in China.

On a smaller scale, the great urban theorist Jane Jacobs stressed the need for economic diversity in a city if it is to flourish. Detroit's over-reliance on the automobile industry, and its subsequent collapse when that industry largely deserted it, is a prominent example of Jacobs' point. And while Detroit is perhaps the most famous example of a city collapsing due to over-specialization, it is far from the only one .

All of this suggests that trade policy, at any level, should have, as its primary goal, the encouragement of diversity in that level's economic activity. To embrace the extremes of "pure free trade" or "total self-sufficiency" is to become more susceptible to catastrophe from changing conditions. A region that can produce only a few goods is fragile in the face of an event, like the coronavirus, that disrupts the flow of outside goods. On the other hand, turning completely inward, and cutting the region off from the outside, leaves it without outside help when confronting a local disaster, like an extreme drought.

To be resilient as a social entity, whether a nation, region, city, or family, will have a diverse mix of internal and external resources it can draw upon for sustenance. Even for an individual, total specialization and complete autarchy are both bad bets. If your only skill is repairing Sony Walkmen, you were probably pretty busy in 2000, but by today you likely don't have much work. Complete individual autarchy isn't ever really even attempted: if you watch YouTube videos of supposedly "self-reliant" people in the wilderness, you will find them using axes, radios, saws, solar panels, pots and pans, shirts, shoes, tents, and many more goods produced by others.

In the technical literature, having such diversity at multiple scales is referred to as "multiscale variety." In a system that displays multiscale variety, no single scale accounts for all of the diversity of behavior in the system. The practical importance of this is related to the fact that shocks themselves come at different scales. Some shocks might be limited to a town or a region, for instance local weather events, while others can be much more widespread, such as the coronavirus pandemic we are currently facing.

A system with multiscale variety is able to respond to shocks at the scale at which they occur: if one region experiences a drought while a neighboring region does not, agricultural supplementation from the currently abundant region can be leveraged. At a smaller scale, if one field of potatoes becomes infested with a pest, while the adjacent cows in pasture are spared, the family who owns the farm will still be able to feed themselves and supply products to the market.

Understanding this, the question becomes how can trade policy, conceived broadly, promote the necessary variety and resiliency to mitigate and thrive in the face of the unexpected? Crucially, we should learn from the tech companies: practice disconnecting, and do it randomly. In our view there are two important components to the intentional disruption: (1) it is regular enough to generate "muscle memory" type responses; and (2) it is random enough that responses are not "overfit" to particular scenarios.

For an individual or family, implementing such a policy might create some hardships, but there are few institutional barriers to doing so. One week, simply declare, "Let's pretend all of the grocery stores are empty, and try getting by only on what we can produce in the yard or have stockpiled in our house!" On another occasion, perhaps, see if you can keep your house warm for a few days without input from utility companies.

Businesses are also largely free of institutional barriers to practicing disconnecting. A company can simply say, "We are awfully dependent on supplier X: this week, we are not going to order from them, and let's see what we can do instead!" A business can also seek out external alternatives to over-reliance on crucial internal resources: for instance, if your top tech guy can hold your business hostage, it is a good idea to find an outside consulting firm that could potentially fill his role.

When we get up to the scale of the nation, things become (at least institutionally) trickier. If Freedonia suddenly bans the import of goods from Ruritania, even for a week, Ruritania is likely to regard this as a "trade war," and may very well go to the WTO and seek relief. However, the point of this reorientation of trade policy is not to promote hostility to other countries, but to make one's own country more resilient. A possible solution to this problem is that a national government could periodically, at random times, buy all of the imports of some good from some other country, and stockpile them. Then the foreign supplier would have no cause for complaint: its goods are still being purchased! But domestic manufacturers would have to learn to adjust to a disappearance of the supply of palm oil from Indonesia, or tin from China, or oil from Norway.

Critics will complain that such government management of trade flows, even with the noble aim of rendering an economy antifragile, will inevitably be turned to less pure purposes, like protecting politically powerful industrialists. But so what? It is not as though the pursuit of free trade hasn't itself yielded perverse outcomes, such as the NAFTA trade agreement that ran to over one thousand pages. Any good aim is likely to suffer diversion as it passes through the rough-and-tumble of political reality. Thus, we might as well set our sites on an ideal policy, even though it won't be perfectly realized.

We must learn to deal with disruptions when success is not critical to survival. The better we become at responding to unexpected shocks, the lower the cost will be each time we face an event beyond our control that demands an adaptive response. To wait until adaptation is necessary makes us fragile when a real crisis appears. We should begin to develop an antifragile economy today, by causing our own disruptions and learning to overcome them. Deliberately disrupting our own economy may sound crazy. But then, so did deliberately crashing one's own servers, until Chaos Monkey proved that it works.

Gene Callahan teaches at the Tandon School of Engineering at New York University. Joe Norman is a data scientist and researcher at the New England Complex Systems Institute.

Most disruptive force is own demographic change of which govts have known for decades. Caronovirus challenge is nothing compared to what will happen because US ed system discriminated against the poor who will be the majority!

What Winston Churchill once said about the Americans is in fact true of all humans: "Americans always end up doing
the right thing once they have exhausted all other options". That's just as true of the French (I write from France) since our government stopped stocking a strategic reserve of a billion breathing-masks in 2013 because "we could buy them in Chine for a lower costs". Now we can't produce enough masks even for our hospitals.

[Mar 05, 2020] Micro data center

Mar 05, 2020 | en.wikipedia.org

A micro data center ( MDC ) is a smaller or containerized (modular) data center architecture that is designed for computer workloads not requiring traditional facilities. Whereas the size may vary from rack to container, a micro data center may include fewer than four servers in a single 19-inch rack. It may come with built-in security systems, cooling systems, and fire protection. Typically there are standalone rack-level systems containing all the components of a 'traditional' data center, ^[1] including in-rack cooling, power supply, power backup, security, fire and suppression. Designs exist where energy is conserved by means of temperature chaining , in combination with liquid cooling. ^[2]

In mid-2017, technology introduced by the DOME project was demonstrated enabling 64 high-performance servers, storage, networking, power and cooling to be integrated in a 2U 19" rack-unit. This packaging, sometimes called 'datacenter-in-a-box' allows deployments in spaces where traditional data centers do not fit, such as factory floors ( IOT ) and dense city centers, especially for edge-computing and edge-analytics.

MDCs are typically portable and provide plug and play features. They can be rapidly deployed indoors or outdoors, in remote locations, for a branch office, or for temporary use in high-risk zones. ^[3] They enable distributed workloads , minimizing downtime and increasing speed of response.

[Mar 05, 2020] What's next for data centers Think micro data centers by Larry Dignan

Apr 14, 2019 | www.zdnet.com

A micro data center, a mini version of a data center rack, could work as edge computing takes hold in various industries. Here's a look at the moving parts behind the micro data center concept.

[Nov 08, 2019] Multiple Linux sysadmins working as root

No new interesting ideas for such an important topic whatsoever. One of the main problems here is documenting actions of each administrator in such a way that the set of actions was visible to everybody in a convenient and transparent matter. With multiple terminal opened Unix history is not the file from which you can deduct each sysadmin actions as parts of the history from additional terminals are missing. , not smooch access. Actually Solaris has some ideas implemented in Solaris 10, but they never made it to Linux

May 21, 2012 | serverfault.com

In our team we have three seasoned Linux sysadmins having to administer a few dozen Debian servers. Previously we have all worked as root using SSH public key authentication. But we had a discussion on what is the best practice for that scenario and couldn't agree on anything.

Everybody's SSH public key is put into ~root/.ssh/authorized_keys2

• Advantage: easy to use, SSH agent forwarding works easily, little overhead
• Disadvantage: missing auditing (you never know which "root" made a change), accidents are more likely

Using personalized accounts and sudo

That way we would login with personalized accounts using SSH public keys and use sudo to do single tasks with root permissions. In addition we could give ourselves the "adm" group that allows us to view log files.

• Advantage: good auditing, sudo prevents us from doing idiotic things too easily
• Disadvantage: SSH agent forwarding breaks, it's a hassle because barely anything can be done as non-root

Using multiple UID 0 users

This is a very unique proposal from one of the sysadmins. He suggest to create three users in /etc/passwd all having UID 0 but different login names. He claims that this is not actually forbidden and allow everyone to be UID 0 but still being able to audit.

• Advantage: SSH agent forwarding works, auditing might work (untested), no sudo hassle
• Disadvantage: feels pretty dirty - couldn't find it documented anywhere as an allowed way

Comments:

• About your "couldn't find it documented anywhere as an allowed way" statement: Have a look to the -o flag in the useradd manual page. This flag is there to allow multiple users sharing the same uid. – jlliagre May 21 '12 at 11:38
• Can you explain what you mean by "SSH agent forwarding breaks" in the second option? We use this at my work and ssh agent forwarding works just fine. – Patrick May 21 '12 at 12:01
• You should ssh out from your non-root account rather than from within sudo. – Random832 May 21 '12 at 12:33
• Another consequence of sudo method: You can no longer SCP/FTP as root. Any file transfers will first need to be moved into the person's home directory and then copied over in the terminal. This is an advantage and a disadvantage depending on perspective. – user606723 May 21 '12 at 14:43
• Why aren't puppet/chef/ansible-type systems being considered? – Alex Holst May 21 '12 at 20:29

The second option is the best one IMHO. Personal accounts, sudo access. Disable root access via SSH completely. We have a few hundred servers and half a dozen system admins, this is how we do it.

How does agent forwarding break exactly?

Also, if it's such a hassle using sudo in front of every task you can invoke a sudo shell with sudo -s or switch to a root shell with sudo su -

thepearson thepearson 775 8 8 silver badges 18 18 bronze badges

• 10 Rather than disable root access by SSH completely, I recommend making root access by SSH require keys, creating one key with a very strong keyphrase and keeping it locked away for emergency use only. If you have permanent console access this is less useful, but if you don't, it can be very handy. – EightBitTony May 21 '12 at 13:02
• 17 I recommend disabling root login over SSH for security purposes. If you really need to be logged in as root, log in as a non-root user and su. – taz May 21 '12 at 14:29
• +1.. I'd go further than saying "The second option is the best". I'd it's the only reasonable option. Options one and three vastly decrease the security of the system from both outside attacks and mistakes. Besides, #2 is how the system was designed to be primarily used. – Ben Lee May 23 '12 at 18:28
• 2 Please, elaborate on sudo -s . Am I correct to understand that sudo -i is no difference to using su - or basically logging in as root apart from additional log entry compared to plain root login? If that's true, how and why is it better than plain root login? – PF4Public Jun 3 '16 at 20:32

add a comment | 9 With regard to the 3rd suggested strategy, other than perusal of the useradd -o -u userXXX options as recommended by @jlliagre, I am not familiar with running multiple users as the same uid. (hence if you do go ahead with that, I would be interested if you could update the post with any issues (or sucesses) that arise...)

I guess my first observation regarding the first option "Everybody's SSH public key is put into ~root/.ssh/authorized_keys2", is that unless you absolutely are never going to work on any other systems;

1. then at least some of the time, you are going to have to work with user accounts and sudo

The second observation would be, that if you work on systems that aspire to HIPAA, PCI-DSS compliance, or stuff like CAPP and EAL, then you are going to have to work around the issues of sudo because;

1. It an industry standard to provide non-root individual user accounts, that can be audited, disabled, expired, etc, typically using some centralized user database.

So; Using personalized accounts and sudo

It is unfortunate that as a sysadmin, almost everything you will need to do on a remote machine is going to require some elevated permissions, however it is annoying that most of the SSH based tools and utilities are busted while you are in sudo

Hence I can pass on some tricks that I use to work-around the annoyances of sudo that you mention. The first problem is that if root login is blocked using PermitRootLogin=no or that you do not have the root using ssh key, then it makes SCP files something of a PITA.

Problem 1 : You want to scp files from the remote side, but they require root access, however you cannot login to the remote box as root directly.

Boring Solution : copy the files to home directory, chown, and scp down.

ssh userXXX@remotesystem , sudo su - etc, cp /etc/somefiles to /home/userXXX/somefiles , chown -R userXXX /home/userXXX/somefiles , use scp to retrieve files from remote.

Less Boring Solution : sftp supports the -s sftp_server flag, hence you can do something like the following (if you have configured password-less sudo in /etc/sudoers );

sftp  -s '/usr/bin/sudo /usr/libexec/openssh/sftp-server' \
userXXX@remotehost:/etc/resolv.conf

(you can also use this hack-around with sshfs, but I am not sure its recommended... ;-)

If you don't have password-less sudo rights, or for some configured reason that method above is broken, I can suggest one more less boring file transfer method, to access remote root files.

Port Forward Ninja Method :

Login to the remote host, but specify that the remote port 3022 (can be anything free, and non-reserved for admins, ie >1024) is to be forwarded back to port 22 on the local side.

 [localuser@localmachine ~]$ ssh userXXX@remotehost -R 3022:localhost:22
Last login: Mon May 21 05:46:07 2012 from 123.123.123.123
------------------------------------------------------------------------
This is a private system; blah blah blah
------------------------------------------------------------------------

Get root in the normal fashion...

-bash-3.2$ sudo su -
[root@remotehost ~]#

Now you can scp the files in the other direction avoiding the boring boring step of making a intermediate copy of the files;

[root@remotehost ~]#  scp -o NoHostAuthenticationForLocalhost=yes \
 -P3022 /etc/resolv.conf localuser@localhost:~
localuser@localhost's password: 
resolv.conf                                 100%  
[root@remotehost ~]#

Problem 2: SSH agent forwarding : If you load the root profile, e.g. by specifying a login shell, the necessary environment variables for SSH agent forwarding such as SSH_AUTH_SOCK are reset, hence SSH agent forwarding is "broken" under sudo su - .

Half baked answer :

Anything that properly loads a root shell, is going to rightfully reset the environment, however there is a slight work-around your can use when you need BOTH root permission AND the ability to use the SSH Agent, AT THE SAME TIME

This achieves a kind of chimera profile, that should really not be used, because it is a nasty hack , but is useful when you need to SCP files from the remote host as root, to some other remote host.

Anyway, you can enable that your user can preserve their ENV variables, by setting the following in sudoers;

 Defaults:userXXX    !env_reset

this allows you to create nasty hybrid login environments like so;

login as normal;

[localuser@localmachine ~]$ ssh userXXX@remotehost 
Last login: Mon May 21 12:33:12 2012 from 123.123.123.123
------------------------------------------------------------------------
This is a private system; blah blah blah
------------------------------------------------------------------------
-bash-3.2$ env | grep SSH_AUTH
SSH_AUTH_SOCK=/tmp/ssh-qwO715/agent.1971

create a bash shell, that runs /root/.profile and /root/.bashrc . but preserves SSH_AUTH_SOCK

-bash-3.2$ sudo -E bash -l

So this shell has root permissions, and root $PATH (but a borked home directory...)

bash-3.2# id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) context=user_u:system_r:unconfined_t
bash-3.2# echo $PATH
/usr/kerberos/sbin:/usr/local/sbin:/usr/sbin:/sbin:/home/xtrabm/xtrabackup-manager:/usr/kerberos/bin:/opt/admin/bin:/usr/local/bin:/bin:/usr/bin:/opt/mx/bin

But you can use that invocation to do things that require remote sudo root, but also the SSH agent access like so;

bash-3.2# scp /root/.ssh/authorized_keys ssh-agent-user@some-other-remote-host:~
/root/.ssh/authorized_keys              100%  126     0.1KB/s   00:00    
bash-3.2#

Tom H Tom H 8,793 3 3 gold badges 34 34 silver badges 57 57 bronze badges

• 1 I like the hacks. – sjbotha May 21 '12 at 20:16

add a comment | 2 The 3rd option looks ideal - but have you actually tried it out to see what's happenning? While you might see the additional usernames in the authentication step, any reverse lookup is going to return the same value.

Allowing root direct ssh access is a bad idea, even if your machines are not connected to the internet / use strong passwords.

Usually I use 'su' rather than sudo for root access.

symcbean symcbean 18.8k 1 1 gold badge 24 24 silver badges 40 40 bronze badges

• 4 Adding multiple users with the same UID adds problems. When applications go to look up the username for a UID number, they can look up the wrong username. Applications which run under root can think they're running as the wrong user, and lots of weird errors will start popping up (I tried it once). – Patrick May 21 '12 at 11:59
• 8 The third option is just a bloody bad idea . You're essentially breaking the 1:1 relation between UIDs and usernames, and literally everything in unix expects that relation to hold. Just because there's no explicit rule not to do it doesn't mean it's a good idea. – Shadur May 21 '12 at 14:28
• Sorry, but the third option is a horrible idea. Having multiple UID 0 people logging in is just asking for problems to be multiplied. Option number 2 is the only sane one. – Doug May 21 '12 at 20:01
• The third option doesn't deserve that many downvotes. There is no command in Unix I'm aware of that is confused by this trick, people might be but commands should not care. It is just a different login name but as soon as you are logged in, the first name found matching the uid in the password database is used so just make sure the real username (here root) appears first there. – jlliagre May 21 '12 at 21:38
• @Patrick Have you seen this in practice? As much as I tested, then applications pick root user if root user is the first one in /etc/passwd with UID 0. I tend to agree with jlliagre. The only downside that I see, is that each user is a root user and sometimes it might be confusing to understand who did what. – Martin Sep 4 '16 at 18:31

add a comment | 2 I use (1), but I happened to type

rm -rf / tmp *

on one ill-fated day.I can see to be bad enough if you have more than a handful admins.

(2) Is probably more engineered - and you can become full-fledged root through sudo su -. Accidents are still possible though.

(3) I would not touch with a barge pole. I used it on Suns, in order to have a non-barebone-sh root account (if I remember correctly) but it was never robust - plus I doubt it would be very auditable.

add a comment | 2 Definitely answer 2.

1. Means that you're allowing SSH access as root . If this machine is in any way public facing, this is just a terrible idea; back when I ran SSH on port 22, my VPS got multiple attempts hourly to authenticate as root. I had a basic IDS set up to log and ban IPs that made multiple failed attempts, but they kept coming. Thankfully, I'd disabled SSH access as the root user as soon as I had my own account and sudo configured. Additionally, you have virtually no audit trail doing this.
2. Provides root access as and when it is needed. Yes, you barely have any privileges as a standard user, but this is pretty much exactly what you want; if an account does get compromised, you want it to be limited in its abilities. You want any super user access to require a password re-entry. Additionally, sudo access can be controlled through user groups, and restricted to particular commands if you like, giving you more control over who has access to what. Additionally, commands run as sudo can be logged, so it provides a much better audit trail if things go wrong. Oh, and don't just run "sudo su -" as soon as you log in. That's terrible, terrible practice.
3. Your sysadmin's idea is bad. And he should feel bad. No, *nix machines probably won't stop you from doing this, but both your file system, and virtually every application out there expects each user to have a unique UID. If you start going down this road, I can guarantee that you'll run into problems. Maybe not immediately, but eventually. For example, despite displaying nice friendly names, files and directories use UID numbers to designate their owners; if you run into a program that has a problem with duplicate UIDs down the line, you can't just change a UID in your passwd file later on without having to do some serious manual file system cleanup.

sudo is the way forward. It may cause additional hassle with running commands as root, but it provides you with a more secure box, both in terms of access and auditing.

Rohaq Rohaq 121 3 3 bronze badges

Definitely option 2, but use groups to give each user as much control as possible without needing to use sudo. sudo in front of every command loses half the benefit because you are always in the danger zone. If you make the relevant directories writable by the sysadmins without sudo you return sudo to the exception which makes everyone feel safer.

Julian Julian 121 4 4 bronze badges

In the old days, sudo did not exist. As a consequence, having multiple UID 0 users was the only available alternative. But it's still not that good, notably with logging based on the UID to obtain the username. Nowadays, sudo is the only appropriate solution. Forget anything else.

It is documented permissible by fact. BSD unices have had their toor account for a long time, and bashroot users tend to be accepted practice on systems where csh is standard (accepted malpractice ;)

add a comment | 0 Perhaps I'm weird, but method (3) is what popped into my mind first as well. Pros: you'd have every users name in logs and would know who did what as root. Cons: they'd each be root all the time, so mistakes can be catastrophic.

I'd like to question why you need all admins to have root access. All 3 methods you propose have one distinct disadvantage: once an admin runs a sudo bash -l or sudo su - or such, you lose your ability to track who does what and after that, a mistake can be catastrophic. Moreover, in case of possible misbehaviour, this even might end up a lot worse.

Instead you might want to consider going another way:

• Create your admin users as regular users
• Decide who needs to do what job (apache management / postfix management etc)
• Add users to related groups (such as add "martin" to "postfix" and "mail", "amavis" if you use it, etc.)
• Fix permissions (chmod -R g+w postfix:postfix /etc/postfix)
• give only relative sudo powers: (visudo -> let martin use /etc/init.d/postfix , /usr/bin/postsuper etc.)

This way, martin would be able to safely handle postfix, and in case of mistake or misbehaviour, you'd only lose your postfix system, not entire server.

Same logic can be applied to any other subsystem, such as apache, mysql, etc.

Of course, this is purely theoretical at this point, and might be hard to set up. It does look like a better way to go tho. At least to me. If anyone tries this, please let me know how it went.

Tuncay Göncüoğlu Tuncay Göncüoğlu 561 3 3 silver badges 9 9 bronze badges

• I should add handling SSH connections is pretty basic in this context. Whatever method you use, do not permit root logins over SSH, let the individual users ssh with their own credentials, and handle sudo/nosudo/etc from there. – Tuncay Göncüoğlu Nov 30 '12 at 17:53

[Oct 22, 2019] Bank of America Says It Saves $2 Billion Per Year By Ignoring Amazon and Microsoft and Building Its Own Cloud Instead

Oct 22, 2019 | slashdot.org

(businessinsider.com) 121 building its own private cloud software rather than outsourcing to companies like Amazon, Microsoft, and Google. From a report: The investment, including a $350 million charge in 2017, hasn't been cheap, but it has had a striking payoff, CEO Brian Moynihan said during the company's third-quarter earnings call. He said the decision helped reduce the firm's servers to 70,000 from 200,000 and its data centers to 23 from 60, and it has resulted in $2 billion in annual infrastructure savings.

[Aug 22, 2019] Why Micro Data Centers Deliver Good Things in Small Packages by Calvin Hennick

Aug 22, 2019 | solutions.cdw.com

Enterprises are deploying self-contained micro data centers to power computing at the network edge.

Calvin Hennick is a freelance journalist who specializes in business and technology writing. He is a contributor to the CDW family of technology magazines.

The location for data processing has changed significantly throughout the history of computing. During the mainframe era, data was processed centrally, but client/server architectures later decentralized computing. In recent years, cloud computing centralized many processing workloads, but digital transformation and the Internet of Things are poised to move computing to new places, such as the network edge .

"There's a big transformation happening," says Thomas Humphrey, segment director for edge computing at APC . "Technologies like IoT have started to require that some local computing and storage happen out in that distributed IT architecture."

For example, some IoT systems require processing of data at remote locations rather than a centralized data center , such as at a retail store instead of a corporate headquarters.

To meet regulatory requirements and business needs, IoT solutions often need low latency, high bandwidth, robust security and superior reliability . To meet these demands, many organizations are deploying micro data centers: self-contained solutions that provide not only essential infrastructure, but also physical security, power and cooling and remote management capabilities.

"Digital transformation happens at the network edge, and edge computing will happen inside micro data centers ," says Bruce A. Taylor, executive vice president at Datacenter Dynamics . "This will probably be one of the fastest growing segments -- if not the fastest growing segment -- in data centers for the foreseeable future."

What Is a Micro Data Center?

Delivering the IT capabilities needed for edge computing represents a significant challenge for many organizations, which need manageable and secure solutions that can be deployed easily, consistently and close to the source of computing . Vendors such as APC have begun to create comprehensive solutions that provide these necessary capabilities in a single, standardized package.

"From our perspective at APC, the micro data center was a response to what was happening in the market," says Humphrey. "We were seeing that enterprises needed more robust solutions at the edge."

Most micro data center solutions rely on hyperconverged infrastructure to integrate computing, networking and storage technologies within a compact footprint . A typical micro data center also incorporates physical infrastructure (including racks), fire suppression, power, cooling and remote management capabilities. In effect, the micro data center represents a sweet spot between traditional IT closets and larger modular data centers -- giving organizations the ability to deploy professional, powerful IT resources practically anywhere .

Standardized Deployments Across the Country

Having robust IT resources at the network edge helps to improve reliability and reduce latency, both of which are becoming more and more important as analytics programs require that data from IoT deployments be processed in real time .

"There's always been edge computing," says Taylor. "What's new is the need to process hundreds of thousands of data points for analytics at once."

Standardization, redundant deployment and remote management are also attractive features, especially for large organizations that may need to deploy tens, hundreds or even thousands of micro data centers. "We spoke to customers who said, 'I've got to roll out and install 3,500 of these around the country,'" says Humphrey. "And many of these companies don't have IT staff at all of these sites." To address this scenario, APC designed standardized, plug-and-play micro data centers that can be rolled out seamlessly. Additionally, remote management capabilities allow central IT departments to monitor and troubleshoot the edge infrastructure without costly and time-intensive site visits.

In part because micro data centers operate in far-flung environments, security is of paramount concern. The self-contained nature of micro data centers ensures that only authorized personnel will have access to infrastructure equipment , and security tools such as video surveillance provide organizations with forensic evidence in the event that someone attempts to infiltrate the infrastructure.

How Micro Data Centers Can Help in Retail, Healthcare

Micro data centers make business sense for any organization that needs secure IT infrastructure at the network edge. But the solution is particularly appealing to organizations in fields such as retail, healthcare and finance , where IT environments are widely distributed and processing speeds are often a priority.

In retail, for example, edge computing will become more important as stores find success with IoT technologies such as mobile beacons, interactive mirrors and real-time tools for customer experience, behavior monitoring and marketing .

"It will be leading-edge companies driving micro data center adoption, but that doesn't necessarily mean they'll be technology companies," says Taylor. "A micro data center can power real-time analytics for inventory control and dynamic pricing in a supermarket."

In healthcare, digital transformation is beginning to touch processes and systems ranging from medication carts to patient records, and data often needs to be available locally; for example, in case of a data center outage during surgery. In finance, the real-time transmission of data can have immediate and significant financial consequences. And in both of these fields, regulations governing data privacy make the monitoring and security features of micro data centers even more important.

Micro data centers also have enormous potential to power smart city initiatives and to give energy companies a cost-effective way of deploying resources in remote locations , among other use cases.

"The proliferation of edge computing will be greater than anything we've seen in the past," Taylor says. "I almost can't think of a field where this won't matter."

Learn more about how solutions and services from CDW and APC can help your organization overcome its data center challenges.

Micro Data Centers Versus IT Closets

Think the micro data center is just a glorified update on the traditional IT closet? Think again.

"There are demonstrable differences," says Bruce A. Taylor, executive vice president at Datacenter Dynamics. "With micro data centers, there's a tremendous amount of computing capacity in a very small, contained space, and we just didn't have that capability previously ."

APC identifies three key differences between IT closets and micro data centers:

Difference #1: Uptime Expectations. APC notes that, of the nearly 3 million IT closets in the U.S., over 70 percent report outages directly related to human error. In an unprotected IT closet, problems can result from something as preventable as cleaning staff unwittingly disconnecting a cable. Micro data centers, by contrast, utilize remote monitoring, video surveillance and sensors to reduce downtime related to human error.

Difference #2: Cooling Configurations. The cooling of IT wiring closets is often approached both reactively and haphazardly, resulting in premature equipment failure. Micro data centers are specifically designed to assure cooling compatibility with anticipated loads.

Difference #3: Power Infrastructure. Unlike many IT closets, micro data centers incorporate uninterruptible power supplies, ensuring that infrastructure equipment has the power it needs to help avoid downtime.

[Nov 13, 2018] GridFTP : User s Guide

Notable quotes:

"... file:///path/to/my/file ..."

"... gsiftp://hostname/path/to/remote/file ..."

"... third party transfer ..."

toolkit.globus.org

Table of Contents

1. Introduction

2. Usage scenarios

2.1. Basic procedure for using GridFTP (globus-url-copy)

2.2. Accessing data in...

3. Command line tools

4. Graphical user interfaces

4.1. Globus GridFTP GUI

4.2. UberFTP

5. Security Considerations

5.1. Two ways to configure your server

5.2. New authentication options

5.3. Firewall requirements

6. Troubleshooting

6.1. Establish control channel connection

6.2. Try running globus-url-copy

6.3. If your server starts...

7. Usage statistics collection by the Globus Alliance

1. Introduction The GridFTP User's Guide provides general end user-oriented information. 2. Usage scenarios 2.1. Basic procedure for using GridFTP (globus-url-copy) If you just want the "rules of thumb" on getting started (without all the details), the following options using globus-url-copy will normally give acceptable performance:

globus-url-copy -vb -tcp-bs 2097152 -p 4 source_url destination_url

The source/destination URLs will normally be one of the following:

• file:///path/to/my/file if you are accessing a file on a file system accessible by the host on which you are running your client .
• gsiftp://hostname/path/to/remote/file if you are accessing a file from a GridFTP server .

2.1.1. Putting files One of the most basic tasks in GridFTP is to "put" files, i.e., moving a file from your file system to the server. So for example, if you want to move the file /tmp/foo from a file system accessible to the host on which you are running your client to a file name /tmp/bar on a host named remote.machine.my.edu running a GridFTP server, you would use this command:

globus-url-copy -vb -tcp-bs 2097152 -p 4 file:///tmp/foo gsiftp://remote.machine.my.edu/tmp/bar

	Note
	In theory, remote.machine.my.edu could be the same host as the one on which you are running your client, but that is normally only done in testing situations.

2.1.2. Getting files A get, i.e, moving a file from a server to your file system, would just reverse the source and destination URLs:

	Tip
	Remember file: always refers to your file system.

globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://remote.machine.my.edu/tmp/bar file:///tmp/foo

2.1.3. Third party transfers Finally, if you want to move a file between two GridFTP servers (a third party transfer ), both URLs would use gsiftp: as the protocol:

globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://other.machine.my.edu/tmp/foo gsiftp://remote.machine.my.edu/tmp/bar

2.1.4. For more information If you want more information and details on URLs and the command line options , the Key Concepts Guide gives basic definitions and an overview of the GridFTP protocol as well as our implementation of it. 2.2. Accessing data in... 2.2.1. Accessing data in a non-POSIX file data source that has a POSIX interface If you want to access data in a non-POSIX file data source that has a POSIX interface, the standard server will do just fine. Just make sure it is really POSIX-like (out of order writes, contiguous byte writes, etc). 2.2.2. Accessing data in HPSS The following information is helpful if you want to use GridFTP to access data in HPSS. Architecturally, the Globus GridFTP server can be divided into 3 modules:

• the GridFTP protocol module,
• the (optional) data transform module, and
• the Data Storage Interface (DSI).

In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.

	Note
	This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).

2.2.2.1. GridFTP Protocol Module

The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.

2.2.2.2. Data Transform Functionality

The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.

2.2.2.3. Data Storage Interface (DSI) / Data Transform module

The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN). The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc.. Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.

2.2.2.4. HPSS info

Last Update: August 2005 Working with Los Alamos National Laboratory and the High Performance Storage System (HPSS) collaboration ( http://www.hpss-collaboration.org ), we have written a Data Storage Interface (DSI) for read/write access to HPSS. This DSI would allow an existing application that uses a GridFTP compliant client to utilize an HPSS data resources. This DSI is currently in testing. Due to changes in the HPSS security mechanisms, it requires HPSS 6.2 or later, which is due to be released in Q4 2005. Distribution for the DSI has not been worked out yet, but it will *probably* be available from both Globus and the HPSS collaboration. While this code will be open source, it requires underlying HPSS libraries which are NOT open source (proprietary).

	Note
	This is a purely server side change, the client does not know what DSI is running, so only a site that is already running HPSS and wants to allow GridFTP access needs to worry about access to these proprietary libraries.

2.2.3. Accessing data in SRB The following information is helpful if you want to use GridFTP to access data in SRB. Architecturally, the Globus GridFTP server can be divided into 3 modules:

• the GridFTP protocol module,
• the (optional) data transform module, and
• the Data Storage Interface (DSI).

In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.

	Note
	This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).

2.2.3.1. GridFTP Protocol Module

The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.

2.2.3.2. Data Transform Functionality

The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.

2.2.3.3. Data Storage Interface (DSI) / Data Transform module

The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN). The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc.. Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.

2.2.3.4. SRB info

Last Update: August 2005 Working with the SRB team at the San Diego Supercomputing Center, we have written a Data Storage Interface (DSI) for read/write access to data in the Storage Resource Broker (SRB) (http://www.npaci.edu/DICE/SRB). This DSI will enable GridFTP compliant clients to read and write data to an SRB server, similar in functionality to the sput/sget commands. This DSI is currently in testing and is not yet publicly available, but will be available from both the SRB web site (here) and the Globus web site (here). It will also be included in the next stable release of the toolkit. We are working on performance tests, but early results indicate that for wide area network (WAN) transfers, the performance is comparable. When might you want to use this functionality:

• You have existing tools that use GridFTP clients and you want to access data that is in SRB
• You have distributed data sets that have some of the data in SRB and some of the data available from GridFTP servers.

2.2.4. Accessing data in some other non-POSIX data source The following information is helpful If you want to use GridFTP to access data in a non-POSIX data source. Architecturally, the Globus GridFTP server can be divided into 3 modules:

• the GridFTP protocol module,
• the (optional) data transform module, and
• the Data Storage Interface (DSI).

In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.

	Note
	This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).

2.2.4.1. GridFTP Protocol Module

The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.

2.2.4.2. Data Transform Functionality

The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.

2.2.4.3. Data Storage Interface (DSI) / Data Transform module

Nov 13, 2018 | toolkit.globus.org

The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN).

The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc..

Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly. 3. Command line tools

Please see the GridFTP Command Reference .

[Nov 12, 2018] Edge Computing vs. Cloud Computing What's the Difference by Andy Patrizio ,

"... Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business ..."

Notable quotes:

"... Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business ..."

"... Edge computing is a term you are going to hear more of in the coming years because it precedes another term you will be hearing a lot, the Internet of Things (IoT). You see, the formally adopted definition of edge computing is a form of technology that is necessary to make the IoT work. ..."

"... Tech research firm IDC defines edge computing is a "mesh network of micro data centers that process or store critical data locally and push all received data to a central data center or cloud storage repository, in a footprint of less than 100 square feet." ..."

Jan 23, 2018 | www.datamation.com

Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business

The term cloud computing is now as firmly lodged in our technical lexicon as email and Internet, and the concept has taken firm hold in business as well. By 2020, Gartner estimates that a "no cloud" policy will be as prevalent in business as a "no Internet" policy. Which is to say no one who wants to stay in business will be without one.

You are likely hearing a new term now, edge computing . One of the problems with technology is terms tend to come before the definition. Technologists (and the press, let's be honest) tend to throw a word around before it is well-defined, and in that vacuum come a variety of guessed definitions, of varying accuracy.

Edge computing is a term you are going to hear more of in the coming years because it precedes another term you will be hearing a lot, the Internet of Things (IoT). You see, the formally adopted definition of edge computing is a form of technology that is necessary to make the IoT work.

Tech research firm IDC defines edge computing is a "mesh network of micro data centers that process or store critical data locally and push all received data to a central data center or cloud storage repository, in a footprint of less than 100 square feet."

It is typically used in IoT use cases, where edge devices collect data from IoT devices and do the processing there, or send it back to a data center or the cloud for processing. Edge computing takes some of the load off the central data center, reducing or even eliminating the processing work at the central location.

IoT Explosion in the Cloud Era

To understand the need for edge computing you must understand the explosive growth in IoT in the coming years, and it is coming on big. There have been a number of estimates of the growth in devices, and while they all vary, they are all in the billions of devices.

• Gartner estimates there were 6.4 billion connected devices in 2016 will it reach 20.8 billion by 2020. It estimates that in 2016, 5.5 million new "things" were connected every day.
• IDC predicts global IoT revenue will grow from $2.71 billion in 2015 to $7.065 billion by 2020, with the total installed base of devices reaching 28.1 billion in 2020.
• IHS Markit forecasts that the IoT market will grow from an installed base of 15.4 billion devices in 2015 to 30.7 billion devices by 2020 and 75.4 billion in 2025.
• McKinsey estimates the total IoT market size was about $900 million in 2015 and will grow to $3.7 billion by 2020.

This is taking place in a number of areas, most notably cars and industrial equipment. Cars are becoming increasingly more computerized and more intelligent. Gone are the days when the "Check engine" warning light came on and you had to guess what was wrong. Now it tells you which component is failing.

The industrial sector is a broad one and includes sensors, RFID, industrial robotics, 3D printing, condition monitoring, smart meters, guidance, and more. This sector is sometimes called the Industrial Internet of Things (IIoT) and the overall market is expected to grow from $93.9 billion in 2014 to $151.01 billion by 2020.

All of these sensors are taking in data but they are not processing it. Your car does some of the processing of sensor data but much of it has to be sent in to a data center for computation, monitoring and logging.

The problem is that this would overload networks and data centers. Imaging the millions of cars on the road sending in data to data centers around the country. The 4G network would be overwhelmed, as would the data centers. And if you are in California and the car maker's data center is in Texas, that's a long round trip.

[Nov 09, 2018] Cloud-hosted date must be accessed by users over existing WAN which creates performance issues due to bandwidth and latency constraints

Notable quotes:

"... Congestion problems lead to miserable performance. We have one WAN pipe, typically 1.5 Mbps to 10 MBps ..."

Nov 09, 2018 | www.eiseverywhere.com

However, cloud-hosted information assets must still be accessed by users over existing WAN infrastructures, where there are performance issues due to bandwidth and latency constraints.

THE EXTREMELY UNFUNNY PART - UP TO 20x SLOWER

• The "distance" problem
• "Skinny Straw" problem
• Latency and application chattiness
• Slower application access

Public/Private

Cloud

Thousands of companies
Millions of users
Varied bandwidth

♦ Per-unit provisioning costs does not decrease much with size after, say, 100 units.

> Cloud data centers are potentially "far away"

♦ Cloud infrastructure supports many enterprises

♦ Large scale drives lower per-unit cost for data center
services

> All employees will be "remote" from their data

♦ Even single-location companies will be remote from their data

♦ HQ employees previously local to servers, but not with Cloud model

> Lots of data needs to be sent over limited WAN bandwidth

Congestion problems lead to miserable performance. We have one WAN pipe, typically 1.5 Mbps to 10 MBps

• Files
• Email
• Web Apps
• Database U J
• Backup
• VOIP

> Disk-based deduplication technology

♦ Identify redundant data at the byte level, not application (e.g., file) level

♦ Use disks to store vast dictionaries of byte sequences for long periods of time

♦ Use symbols to transfer repetitive sequences of byte-level raw data

♦ Only deduplicated data stored on disk

[Nov 09, 2018] Troubleshoot WAN Performance Issues SD Wan Experts by Steve Garson

Feb 08, 2013 | www.sd-wan-experts.com

Troubleshooting MPLS Networks

How should you troubleshoot WAN performance issues. Your MPLS or VPLS network and your clients in field offices are complaining about slow WAN performance. Your network should be performing better and you can't figure out what the problem is. You can contact SD-WAN-Experts to have their engineers solve your problem, but you want to try to solve the problems yourself.

1. The first thing to check, seems trivial, but you need to confirm that the ports on your router and switch ports are configured for the same speed and duplex. Log into your switches and check the logs for mismatches of speed or duplex. Auto-negotiation sometimes does not work properly, so a 10M port connected to a 100M port is mismatched. Or you might have a half-duplex port connected to a full-duplex port. Don't assume that a 10/100/1000 port is auto-negotiating correctly!
2. Is your WAN performance problem consistent? Does it occur at roughly the same time of day? Or is it completely random? If you don't have the monitoring tools to measure this, you are at a big disadvantage in resolving the issues on your own.
3. Do you have Class of Service configured on your WAN? Do you have DSCP configured on your LAN? What is the mapping of your DSCP values to CoS?
4. What kind of applications are traversing your WAN? Are there specific apps that work better than others?
5. Have your reviewed bandwidth utilization on your carrier's web portal to determine if you are saturating the MPLS port of any locations? Even brief peaks will be enough to generate complaints. Large files, such as CAD drawings, can completely saturate a WAN link.
6. Are you backing up or synchronizing data over the WAN? Have you confirmed 100% that this work is completed before the work day begins.
7. Might your routing be taking multiple paths and not the most direct path? Look at your routing tables.
8. Next, you want to see long term trend statistics. This means monitoring the SNMP streams from all your routers, using tools such as MRTG, NTOP or Cacti. A two week sampling should provide a very good picture of what is happening on your network to help troubleshoot your WAN.

NTOP allows you to

• Sort network traffic according to many protocols
• Show network traffic sorted according to various criteria
• Display traffic statistics
• Store on disk persistent traffic statistics in RRD format
• Identify the identity (e.g. email address) of computer users
• Passively (i.e. without sending probe packets) identify the host OS
• Show IP traffic distribution among the various protocols
• Analyse IP traffic and sort it according to the source/destination
• Display IP Traffic Subnet matrix (who's talking to who?)
• Report IP protocol usage sorted by protocol type
• Act as a NetFlow/ sFlow collector for flows generated by routers (e.g. Cisco and Juniper) or switches (e.g. Foundry Networks)
• Produce RMON-like network traffic statistic

MRTG (Multi-Router Traffic Grapher) provides easy to understand graphs of your network bandwidth utilization.

Cacti requires a MySQL database. It is a complete network graphing solution designed to harness the power of RRDTool 's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices.

Both NTOP and MRTG are freeware applications to help troubleshoot your WAN that will run on the freeware versions of Linux. As a result, they can be installed on almost any desktop computer that has out-lived its value as a Windows desktop machine. If you are skilled with Linux and networking, and you have the time, you can install this monitoring system on your own. You will need to get your carrier to provide read-only access to your router SNMP traffic.

But you might find it more cost effective to have the engineers at SD-WAN-Experts do the work for you. All you need to do is provide an available machine with a Linux install (Ubuntu, CentOS, RedHat, etc) with remote access via a VPN. Our engineers will then download all the software remotely, install and configure the machine. When we are done with the monitoring, beside understanding how to solve your problem (and solving it!) you will have your own network monitoring system installed for your use on a daily basis. We'll teach you how to use it, which is quite simple using the web based tools, so you can view it from any machine on your network.

If you need assistance in troubleshooting your wide area network, contact SD-WAN-Experts today !

You might also find these troubleshooting tips of interest;

Troubleshooting MPLS Network Performance Issues

Packet Loss and How It Affects Performance

Troubleshooting VPLS and Ethernet Tunnels over MPLS

[Nov 09, 2018] Storage in private clouds

Nov 09, 2018 | www.redhat.com

Storage in private clouds

Storage is one of the most popular uses of cloud computing, particularly for consumers. The user-friendly design of service-based companies have helped make "cloud" a pretty normal term -- even reaching meme status in 2016.

However, cloud storage means something very different to businesses. Big data and the Internet of Things (IoT) have made it difficult to appraise the value of data until long after it's originally stored -- when finding that piece of data becomes the key to revealing valuable business insights or unlocking an application's new feature. Even after enterprises decide where to store their data in the cloud (on-premise, off-premise, public, or private), they still have to decide how they're going to store it. What good is data that can't be found?

It's common to store data in the cloud using software-defined storage . Software-defined storage decouples storage software from hardware so you can abstract and consolidate storage capacity in a cloud. It allows you to scale beyond whatever individual hardware components your cloud is built on.

Two of the more common software-defined storage solutions include Ceph for structured data and Gluster for unstructured data. Ceph is a massively scalable, programmable storage system that works well with clouds -- particularly those deployed using OpenStack ^® -- because of its ability to unify object, block, and file storage into 1 pool of resources. Gluster is designed to handle the requirements of traditional file storage and is particularly adept at provisioning and managing elastic storage for container-based applications.

[Nov 09, 2018] Cloud Computing vs Edge Computing Which Will Prevail

Notable quotes:

"... The recent widespread of edge computing in some 5G showcases, like the major sports events, has generated the ongoing discussion about the possibility of edge computing to replace cloud computing. ..."

"... For instance, Satya Nadella, the CEO of Microsoft, announced in Microsoft Build 2017 that the company will focus its strategy on edge computing. Indeed, edge computing will be the key for the success of smart home and driverless vehicles ..."

"... the edge will be the first to process and store the data generated by user devices. This will reduce the latency for the data to travel to the cloud. In other words, the edge optimizes the efficiency for the cloud. ..."

Nov 09, 2018 | www.lannerinc.com

The recent widespread of edge computing in some 5G showcases, like the major sports events, has generated the ongoing discussion about the possibility of edge computing to replace cloud computing.

In fact, there have been announcements from global tech leaders like Nokia and Huawei demonstrating increased efforts and resources in developing edge computing.

For instance, Satya Nadella, the CEO of Microsoft, announced in Microsoft Build 2017 that the company will focus its strategy on edge computing. Indeed, edge computing will be the key for the success of smart home and driverless vehicles.

... ... ...

Cloud or edge, which will lead the future?

The answer to that question is "Cloud – Edge Mixing". The cloud and the edge will complement each other to offer the real IoT experience. For instance, while the cloud coordinates all the technology and offers SaaS to users, the edge will be the first to process and store the data generated by user devices. This will reduce the latency for the data to travel to the cloud. In other words, the edge optimizes the efficiency for the cloud.

It is strongly suggested to implement open architecture white-box servers for both cloud and edge, to minimize the latency for cloud-edge synchronization and optimize the compatibility between the two. For example, Lanner Electronics offers a wide range of Intel x86 white box appliances for data centers and edge uCPE/vCPE.

http://www.lannerinc.com/telecom-datacenter-appliances/vcpe/ucpe-platforms/

[Nov 09, 2018] What is Hybrid Cloud Computing

Nov 09, 2018 | www.dummies.com

The hybrid cloud

A hybrid cloud is a combination of a private cloud combined with the use of public cloud services where one or several touch points exist between the environments. The goal is to combine services and data from a variety of cloud models to create a unified, automated, and well-managed computing environment.

Combining public services with private clouds and the data center as a hybrid is the new definition of corporate computing. Not all companies that use some public and some private cloud services have a hybrid cloud. Rather, a hybrid cloud is an environment where the private and public services are used together to create value.

A cloud is hybrid

• If a company uses a public development platform that sends data to a private cloud or a data center–based application.

• When a company leverages a number of SaaS (Software as a Service) applications and moves data between private or data center resources.

• When a business process is designed as a service so that it can connect with environments as though they were a single environment.

A cloud is not hybrid

• If a few developers in a company use a public cloud service to prototype a new application that is completely disconnected from the private cloud or the data center.

• If a company is using a SaaS application for a project but there is no movement of data from that application into the company's data center.

[Nov 09, 2018] Why Micro Data Centers Deliver Good Things in Small Packages by Calvin Hennick

Notable quotes:

"... "There's a big transformation happening," says Thomas Humphrey, segment director for edge computing at APC . "Technologies like IoT have started to require that some local computing and storage happen out in that distributed IT architecture." ..."

"... In retail, for example, edge computing will become more important as stores find success with IoT technologies such as mobile beacons, interactive mirrors and real-time tools for customer experience, behavior monitoring and marketing . ..."

Nov 09, 2018 | solutions.cdw.com

Enterprises are deploying self-contained micro data centers to power computing at the network edge.

The location for data processing has changed significantly throughout the history of computing. During the mainframe era, data was processed centrally, but client/server architectures later decentralized computing. In recent years, cloud computing centralized many processing workloads, but digital transformation and the Internet of Things are poised to move computing to new places, such as the network edge .

"There's a big transformation happening," says Thomas Humphrey, segment director for edge computing at APC . "Technologies like IoT have started to require that some local computing and storage happen out in that distributed IT architecture."

For example, some IoT systems require processing of data at remote locations rather than a centralized data center , such as at a retail store instead of a corporate headquarters.

To meet regulatory requirements and business needs, IoT solutions often need low latency, high bandwidth, robust security and superior reliability . To meet these demands, many organizations are deploying micro data centers: self-contained solutions that provide not only essential infrastructure, but also physical security, power and cooling and remote management capabilities.

"Digital transformation happens at the network edge, and edge computing will happen inside micro data centers ," says Bruce A. Taylor, executive vice president at Datacenter Dynamics . "This will probably be one of the fastest growing segments -- if not the fastest growing segment -- in data centers for the foreseeable future."

What Is a Micro Data Center?

Delivering the IT capabilities needed for edge computing represents a significant challenge for many organizations, which need manageable and secure solutions that can be deployed easily, consistently and close to the source of computing . Vendors such as APC have begun to create comprehensive solutions that provide these necessary capabilities in a single, standardized package.

"From our perspective at APC, the micro data center was a response to what was happening in the market," says Humphrey. "We were seeing that enterprises needed more robust solutions at the edge."

Most micro data center solutions rely on hyperconverged infrastructure to integrate computing, networking and storage technologies within a compact footprint . A typical micro data center also incorporates physical infrastructure (including racks), fire suppression, power, cooling and remote management capabilities. In effect, the micro data center represents a sweet spot between traditional IT closets and larger modular data centers -- giving organizations the ability to deploy professional, powerful IT resources practically anywhere .

Standardized Deployments Across the Country

Having robust IT resources at the network edge helps to improve reliability and reduce latency, both of which are becoming more and more important as analytics programs require that data from IoT deployments be processed in real time .

"There's always been edge computing," says Taylor. "What's new is the need to process hundreds of thousands of data points for analytics at once."

Standardization, redundant deployment and remote management are also attractive features, especially for large organizations that may need to deploy tens, hundreds or even thousands of micro data centers. "We spoke to customers who said, 'I've got to roll out and install 3,500 of these around the country,'" says Humphrey. "And many of these companies don't have IT staff at all of these sites." To address this scenario, APC designed standardized, plug-and-play micro data centers that can be rolled out seamlessly. Additionally, remote management capabilities allow central IT departments to monitor and troubleshoot the edge infrastructure without costly and time-intensive site visits.

In part because micro data centers operate in far-flung environments, security is of paramount concern. The self-contained nature of micro data centers ensures that only authorized personnel will have access to infrastructure equipment , and security tools such as video surveillance provide organizations with forensic evidence in the event that someone attempts to infiltrate the infrastructure.

How Micro Data Centers Can Help in Retail, Healthcare

Micro data centers make business sense for any organization that needs secure IT infrastructure at the network edge. But the solution is particularly appealing to organizations in fields such as retail, healthcare and finance , where IT environments are widely distributed and processing speeds are often a priority.

In retail, for example, edge computing will become more important as stores find success with IoT technologies such as mobile beacons, interactive mirrors and real-time tools for customer experience, behavior monitoring and marketing .

"It will be leading-edge companies driving micro data center adoption, but that doesn't necessarily mean they'll be technology companies," says Taylor. "A micro data center can power real-time analytics for inventory control and dynamic pricing in a supermarket."

In healthcare, digital transformation is beginning to touch processes and systems ranging from medication carts to patient records, and data often needs to be available locally; for example, in case of a data center outage during surgery. In finance, the real-time transmission of data can have immediate and significant financial consequences. And in both of these fields, regulations governing data privacy make the monitoring and security features of micro data centers even more important.

Micro data centers also have enormous potential to power smart city initiatives and to give energy companies a cost-effective way of deploying resources in remote locations , among other use cases.

"The proliferation of edge computing will be greater than anything we've seen in the past," Taylor says. "I almost can't think of a field where this won't matter."

Learn more about how solutions and services from CDW and APC can help your organization overcome its data center challenges.

Micro Data Centers Versus IT Closets

Think the micro data center is just a glorified update on the traditional IT closet? Think again.

"There are demonstrable differences," says Bruce A. Taylor, executive vice president at Datacenter Dynamics. "With micro data centers, there's a tremendous amount of computing capacity in a very small, contained space, and we just didn't have that capability previously ."

APC identifies three key differences between IT closets and micro data centers:

1. Difference #1: Uptime Expectations. APC notes that, of the nearly 3 million IT closets in the U.S., over 70 percent report outages directly related to human error. In an unprotected IT closet, problems can result from something as preventable as cleaning staff unwittingly disconnecting a cable. Micro data centers, by contrast, utilize remote monitoring, video surveillance and sensors to reduce downtime related to human error.
2. Difference #2: Cooling Configurations. The cooling of IT wiring closets is often approached both reactively and haphazardly, resulting in premature equipment failure. Micro data centers are specifically designed to assure cooling compatibility with anticipated loads.
3. Difference #3: Power Infrastructure. Unlike many IT closets, micro data centers incorporate uninterruptible power supplies, ensuring that infrastructure equipment has the power it needs to help avoid downtime.

Calvin Hennick is a freelance journalist who specializes in business and technology writing. He is a contributor to the CDW family of technology magazines.

[Nov 09, 2018] Solving Office 365 and SaaS Performance Issues with SD-WAN

Notable quotes:

"... most of the Office365 deployments face network related problems - typically manifesting as screen freezes. Limited WAN optimization capability further complicates the problems for most SaaS applications. ..."

"... Why enterprises overlook the importance of strategically placing cloud gateways ..."

Nov 09, 2018 | www.brighttalk.com

About this webinar Major research highlights that most of the Office365 deployments face network related problems - typically manifesting as screen freezes. Limited WAN optimization capability further complicates the problems for most SaaS applications. To compound the issue, different SaaS applications issue different guidelines for solving performance issues. We will investigate the major reasons for these problems.

SD-WAN provides an essential set of features that solves these networking issues related to Office 365 and SaaS applications. This session will cover the following major topics:

• How traditional networks impair O365 performance, and why?
  

• Why enterprises overlook the importance of strategically placing cloud gateways

• Understanding tradeoffs among reliability, performance and user-experience when architecting the WAN for your cloud

• Best practice architectures for different kinds of SaaS applications

[Nov 09, 2018] Make sense of edge computing vs. cloud computing

Notable quotes:

"... We already know that computing at the edge pushes most of the data processing out to the edge of the network, close to the source of the data. Then it's a matter of dividing the processing between the edge and the centralized system, meaning a public cloud such as Amazon Web Services, Google Cloud, or Microsoft Azure. ..."

"... The goal is to process near the device the data that it needs quickly, such as to act on. There are hundreds of use cases where reaction time is the key value of the IoT system, and consistently sending the data back to a centralized cloud prevents that value from happening. ..."

Nov 09, 2018 | www.infoworld.com

The internet of things is real, and it's a real part of the cloud. A key challenge is how you can get data processed from so many devices. Cisco Systems predicts that cloud traffic is likely to rise nearly fourfold by 2020, increasing 3.9 zettabytes (ZB) per year in 2015 (the latest full year for which data is available) to 14.1ZB per year by 2020.

As a result, we could have the cloud computing perfect storm from the growth of IoT. After all, IoT is about processing device-generated data that is meaningful, and cloud computing is about using data from centralized computing and storage. Growth rates of both can easily become unmanageable.

So what do we do? The answer is something called "edge computing." We already know that computing at the edge pushes most of the data processing out to the edge of the network, close to the source of the data. Then it's a matter of dividing the processing between the edge and the centralized system, meaning a public cloud such as Amazon Web Services, Google Cloud, or Microsoft Azure.

That may sound a like a client/server architecture, which also involved figuring out what to do at the client versus at the server. For IoT and any highly distributed applications, you've essentially got a client/network edge/server architecture going on, or -- if your devices can't do any processing themselves, a network edge/server architecture.

The goal is to process near the device the data that it needs quickly, such as to act on. There are hundreds of use cases where reaction time is the key value of the IoT system, and consistently sending the data back to a centralized cloud prevents that value from happening.

You would still use the cloud for processing that is either not as time-sensitive or is not needed by the device, such as for big data analytics on data from all your devices.

There's another dimension to this: edge computing and cloud computing are two very different things. One does not replace the other. But too many articles confuse IT pros by suggesting that edge computing will displace cloud computing. It's no more true than saying PCs would displace the datacenter.

It makes perfect sense to create purpose-built edge computing-based applications, such as an app that places data processing in a sensor to quickly process reactions to alarms. But you're not going to place your inventory-control data and applications at the edge -- moving all compute to the edge would result in a distributed, unsecured, and unmanageable mess.

All the public cloud providers have IoT strategies and technology stacks that include, or will include, edge computing. Edge and cloud computing can and do work well together, but edge computing is for purpose-built systems with special needs. Cloud computing is a more general-purpose platform that also can work with purpose-built systems in that old client/server model.

Related:

• Cloud Computing
• Software Development
• Data Management

David S. Linthicum is a chief cloud strategy officer at Deloitte Consulting, and an internationally recognized industry expert and thought leader. His views are his own.

[Nov 08, 2018] GT 6.0 GridFTP

Notable quotes:

"... GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks ..."

Nov 08, 2018 | toolkit.globus.org

The open source Globus® Toolkit is a fundamental enabling technology for the "Grid," letting people share computing power, databases, and other tools securely online across corporate, institutional, and geographic boundaries without sacrificing local autonomy. The toolkit includes software services and libraries for resource monitoring, discovery, and management, plus security and file management. In addition to being a central part of science and engineering projects that total nearly a half-billion dollars internationally, the Globus Toolkit is a substrate on which leading IT companies are building significant commercial Grid products.

The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications. Every organization has unique modes of operation, and collaboration between multiple organizations is hindered by incompatibility of resources such as data archives, computers, and networks. The Globus Toolkit was conceived to remove obstacles that prevent seamless collaboration. Its core services, interfaces and protocols allow users to access remote resources as if they were located within their own machine room while simultaneously preserving local control over who can use resources and when.

The Globus Toolkit has grown through an open-source strategy similar to the Linux operating system's, and distinct from proprietary attempts at resource-sharing software. This encourages broader, more rapid adoption and leads to greater technical innovation, as the open-source community provides continual enhancements to the product.

Essential background is contained in the papers " Anatomy of the Grid " by Foster, Kesselman and Tuecke and " Physiology of the Grid " by Foster, Kesselman, Nick and Tuecke.

Acclaim for the Globus Toolkit

From version 1.0 in 1998 to the 2.0 release in 2002 and now the latest 4.0 version based on new open-standard Grid services, the Globus Toolkit has evolved rapidly into what The New York Times called "the de facto standard" for Grid computing. In 2002 the project earned a prestigious R&D 100 award, given by R&D Magazine in a ceremony where the Globus Toolkit was named "Most Promising New Technology" among the year's top 100 innovations. Other honors include project leaders Ian Foster of Argonne National Laboratory and the University of Chicago, Carl Kesselman of the University of Southern California's Information Sciences Institute (ISI), and Steve Tuecke of Argonne being named among 2003's top ten innovators by InfoWorld magazine, and a similar honor from MIT Technology Review, which named Globus Toolkit-based Grid computing one of "Ten Technologies That Will Change the World." The Globus Toolkit also GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks . The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.

The following guides are available for this component:

Data Management Key Concepts	For important general concepts [ pdf ].
Admin Guide	For system administrators and those installing, building and deploying GT. You should already have read the Installation Guide and Quickstart [ pdf ]
User's Guide	Describes how end-users typically interact with this component. [ pdf ].
Developer's Guide	Reference and usage scenarios for developers. [ pdf ].
Other information available for this component are:
Release Notes	What's new with the 6.0 release for this component. [ pdf ]
Public Interface Guide	Information for all public interfaces (including APIs, commands, etc). Please note this is a subset of information in the Developer's Guide [ pdf ].
Quality Profile	Information about test coverage reports, etc. [ pdf ].
Migrating Guide	Information for migrating to this version if you were using a previous version of GT. [ pdf ]
All GridFTP Guides (PDF only)	Includes all GridFTP guides except Public Interfaces (which is a subset of the Developer's Guide)

[Nov 08, 2018] globus-gridftp-server-control-6.2-1.el7.x86_64.rpm

Nov 08, 2018 | centos.pkgs.org

6.2	x86_64	EPEL Testing
globus-gridftp-server-control	-	-	-

Requires

Name	Value
/sbin/ldconfig	-
globus-xio-gsi-driver(x86-64)	>= 2
globus-xio-pipe-driver(x86-64)	>= 2
libc.so.6(GLIBC_2.14)(64bit)	-
libglobus_common.so.0()(64bit)	-
libglobus_common.so.0(GLOBUS_COMMON_14)(64bit)	-
libglobus_gss_assist.so.3()(64bit)	-
libglobus_gssapi_error.so.2()(64bit)	-
libglobus_gssapi_gsi.so.4()(64bit)	-
libglobus_gssapi_gsi.so.4(globus_gssapi_gsi)(64bit)	-
libglobus_openssl_error.so.0()(64bit)	-
libglobus_xio.so.0()(64bit)	-
rtld(GNU_HASH)	-

See Also

Package	Description
globus-gridftp-server-control-devel-6.1-1.el7.x86_64.rpm	Globus Toolkit - Globus GridFTP Server Library Development Files
globus-gridftp-server-devel-12.5-1.el7.x86_64.rpm	Globus Toolkit - Globus GridFTP Server Development Files
globus-gridftp-server-progs-12.5-1.el7.x86_64.rpm	Globus Toolkit - Globus GridFTP Server Programs
globus-gridmap-callout-error-2.5-1.el7.x86_64.rpm	Globus Toolkit - Globus Gridmap Callout Errors
globus-gridmap-callout-error-devel-2.5-1.el7.x86_64.rpm	Globus Toolkit - Globus Gridmap Callout Errors Development Files
globus-gridmap-callout-error-doc-2.5-1.el7.noarch.rpm	Globus Toolkit - Globus Gridmap Callout Errors Documentation Files
globus-gridmap-eppn-callout-1.13-1.el7.x86_64.rpm	Globus Toolkit - Globus gridmap ePPN callout
globus-gridmap-verify-myproxy-callout-2.9-1.el7.x86_64.rpm	Globus Toolkit - Globus gridmap myproxy callout
globus-gsi-callback-5.13-1.el7.x86_64.rpm	Globus Toolkit - Globus GSI Callback Library
globus-gsi-callback-devel-5.13-1.el7.x86_64.rpm	Globus Toolkit - Globus GSI Callback Library Development Files
globus-gsi-callback-doc-5.13-1.el7.noarch.rpm	Globus Toolkit - Globus GSI Callback Library Documentation Files
globus-gsi-cert-utils-9.16-1.el7.x86_64.rpm	Globus Toolkit - Globus GSI Cert Utils Library
globus-gsi-cert-utils-devel-9.16-1.el7.x86_64.rpm	Globus Toolkit - Globus GSI Cert Utils Library Development Files
globus-gsi-cert-utils-doc-9.16-1.el7.noarch.rpm	Globus Toolkit - Globus GSI Cert Utils Library Documentation Files
globus-gsi-cert-utils-progs-9.16-1.el7.noarch.rpm	Globus Toolkit - Globus GSI Cert Utils Library Programs

Provides

Name	Value
globus-gridftp-server-control	= 6.1-1.el7
globus-gridftp-server-control(x86-64)	= 6.1-1.el7
libglobus_gridftp_server_control.so.0()(64bit)	-

Required By Show Packages Download

Type	URL
Binary Package	globus-gridftp-server-control-6.1-1.el7.x86_64.rpm
Source Package	globus-gridftp-server-control-6.1-1.el7.src.rpm

Install Howto

1. Download the latest epel-release rpm from

   http://dl.fedoraproject.org/pub/epel/7/x86_64/
   

2. Install epel-release rpm:

   # rpm -Uvh epel-release*rpm
   

3. Install globus-gridftp-server-control rpm package:

   # yum install globus-gridftp-server-control
   

Files

Path
/usr/lib64/libglobus_gridftp_server_control.so.0
/usr/lib64/libglobus_gridftp_server_control.so.0.6.1
/usr/share/doc/globus-gridftp-server-control-6.1/README
/usr/share/licenses/globus-gridftp-server-control-6.1/GLOBUS_LICENSE

Changelog

2018-04-07 - Mattias Ellert <[email protected]> - 6.1-1
- GT6 update: Don't error if acquire_cred fails when vhost env is set

[Nov 08, 2018] 9 Aspera Sync Alternatives Top Best Alternatives

Nov 08, 2018 | www.topbestalternatives.com

Aspera Sync is an elite, versatile, multi-directional no concurrent record replication and synchronization. It is intended to conquer the execution and versatility inadequacies of conventional synchronization instruments like Rsync. Aspera Sync can scale up and out for most extreme rate replication and synchronization over WANs. Prominent capacities are The FASP advantage, superior, smart trade for Rsync, underpins complex synchronization arrangements, propelled record taking care of, and so on. Aspera Sync is reason worked by Aspera for elite, versatile, multi-directional offbeat record replication and synchronization. Intended to beat the execution and adaptability deficiencies of conventional synchronization instruments like Rsync, Aspera Sync can scale up and out for greatest pace replication and synchronization over WANs, for now,'s biggest vast information record stores -- from a great many individual documents to the most significant document sizes. Hearty reinforcement and recuperation strategies secure business necessary information and frameworks so undertakings can rapidly recoup necessary documents, structures or a whole site in the occasion if a calamity. Be that as it may, these strategies can be undermined by average exchange speeds amongst essential and reinforcement locales, bringing about fragmented reinforcements and augmented recuperation times. With FASP – controlled transactions, replication fits inside the little operational window so you can meet your recuperation point objective (RPO) and recovery time objective (RTO).

1. Syncthing Syncthing replaces exclusive synchronize and cloud administrations with something open, reliable and decentralized. Your information is your information alone, and you should pick where it is put away if it is imparted to some outsider and how it's transmitted over the Internet. Syncthing is a record sharing application that permits you to share reports between various gadgets in an advantageous way. Its online Graphical User Interface (GUI) makes it conceivable Website 1 Recommend 2 Don't Recommend Syncthing Alternatives

[Nov 03, 2018] Technology is dominated by two types of people; those who understand what they don t manage; and those who manage what they don t understand – ARCHIBALD PUTT ( PUTTS LAW )

Notable quotes:

"... These C level guys see cloud services – applications, data, backup, service desk – as a great way to free up a blockage in how IT service is being delivered on premise. ..."

"... IMHO there is a big difference between management of IT and management of IT service. Rarely do you get people who can do both. ..."

Nov 03, 2018 | brummieruss.wordpress.com

...Cloud introduces a whole new ball game and will no doubt perpetuate Putts Law for ever more. Why?

Well unless 100% of IT infrastructure goes up into the clouds ( unlikely for any organization with a history ; likely for a new organization ( probably micro small ) that starts up in the next few years ) the 'art of IT management' will demand even more focus and understanding.

I always think a great acid test of Putts Law is to look at one of the two aspects of IT management

1. Show me a simple process that you follow each day that delivers an aspect of IT service i.e. how to buy a piece of IT stuff, or a way to report a fault
2. Show me how you manage a single entity on the network i.e. a file server, a PC, a network switch

Usually the answers ( which will be different from people on the same team, in the same room and from the same person on different days !) will give you an insight to Putts Law.

Childs play for most of course who are challenged with some real complex management situations such as data center virtualization projects, storage explosion control, edge device management, backend application upgrades, global messaging migrations and B2C identity integration. But of course if its evidenced that they seem to be managing (simple things ) without true understanding one could argue 'how the hell can they be expected to manage what they understand with the complex things?' Fair point?

Of course many C level people have an answer to Putts Law. Move the problem to people who do understand what they manage. Professionals who provide cloud versions of what the C level person struggles to get a professional service from. These C level guys see cloud services – applications, data, backup, service desk – as a great way to free up a blockage in how IT service is being delivered on premise. And they are right ( and wrong ).

... ... ...

( Quote attributed to Archibald Putt author of Putt's Law and the Successful Technocrat: How to Win in the Information Age )

rowan says: March 9, 2012 at 9:03 am

IMHO there is a big difference between management of IT and management of IT service. Rarely do you get people who can do both. Understanding inventory, disk space, security etc is one thing; but understanding the performance of apps and user impact is another ball game. Putts Law is alive and well in my organisation. TGIF.

Rowan in Belfast.

stephen777 says: March 31, 2012 at 7:32 am

Rowan is right I used to be an IT Manager but now my title is Service Delivery Manager. Why? Because we had a new CTO who changed how people saw what we did. I ve been doing this new role for 5 years and I really do understand what i don't manage. LOL

Stephen777

[Oct 30, 2018] Cloud has a rich future, but I didn't even know Red Hat had any presence there, let alone $35 billion worth.

Oct 30, 2018 | arstechnica.com

ControlledExperiments Ars Centurion reply 5 hours ago I am not your friend wrote: I just can't comprehend that price. Cloud has a rich future, but I didn't even know Red Hat had any presence there, let alone $35 billion worth.

Yeah the price is nuts, reminiscent of FB's WhatsApp purchase, and maybe their Instagram purchase.

If you actually look at the revenue Amazon gets from AWS or Microsoft from Azure, it's not that much, relatively speaking. For Microsoft, it's nothing compared to Windows and Office revenue, and I'm not sure where the growth is supposed to come from. It seems like most everyone who wants to be on the cloud is already there, and vulns like Spectre and Meltdown broke the sacred VM wall, so...

[Oct 27, 2018] We are effectively back to the centralized systems ("clouds" controlled by Amazon (CIA), Google (NSA)). The peripheral components of autonomous networked systems are akin to sensory devices with heavy computational lifting in the (local) regime controlled servers.

Oct 27, 2018 | www.moonofalabama.org

realist , Oct 27, 2018 10:27:04 AM | link

The first wave of the computer revolution created stand-alone systems. The current wave is their combination into new and much larger ones.
Posted by b on October 26, 2018 at 02:18 PM

Not strictly correct. It has come full circle in sense. We started with standalone machines; then we had central mainframes with dumb terminals; then
the first networks of these centralized servers (XeroxParc's efforts being more forward looking and an exception); then PCs; then internet; and now (per design and not by merit, imo)

We are effectively back to the centralized systems ("clouds" controlled by Amazon (CIA), Google (NSA)). The peripheral components of autonomous networked systems are akin to sensory devices with heavy computational lifting in the (local) regime controlled servers.

---

By this, I mean moral codes no longer have any objective basis but are evermore easily re-wired by the .002 to suit their purposes
Posted by: donkeytale | Oct 27, 2018 6:32:11 AM | 69

I question the implied "helpless to resist reprogramming" notion of your statement. You mentioned God. Please note that God has endowed each and every one of us with a moral compass. You mentioned Jesus. Didn't he say something about "you generation of vipers"? I suggest you consider that ours is a morally degenerate generation and is getting precisely what it wants and deserves.

[Sep 16, 2017] Google Drive Faces Outage, Users Report

Sep 16, 2017 | tech.slashdot.org

(google.com) 75

Posted by msmash on Thursday September 07, 2017

Numerous Slashdot readers are reporting that they are facing issues access Google Drive, the productivity suite from the Mountain View-based company. Google's dashboard confirms that Drive is facing outage .

Third-party web monitoring tool DownDetector also reports thousands of similar complaints from users. The company said, "Google Drive service has already been restored for some users, and we expect a resolution for all users in the near future.

Please note this time frame is an estimate and may change. Google Drive is not loading files and results in a failures for a subset of users."

[Jul 02, 2017] The Details About the CIAs Deal With Amazon by Frank Konkel

Jul 17, 2014 | www.theatlantic.com

The intelligence community is about to get the equivalent of an adrenaline shot to the chest. This summer, a $600 million computing cloud developed by Amazon Web Services for the Central Intelligence Agency over the past year will begin servicing all 17 agencies that make up the intelligence community. If the technology plays out as officials envision, it will usher in a new era of cooperation and coordination, allowing agencies to share information and services much more easily and avoid the kind of intelligence gaps that preceded the Sept. 11, 2001, terrorist attacks.

For the first time, agencies within the intelligence community will be able to order a variety of on-demand computing and analytic services from the CIA and National Security Agency. What's more, they'll only pay for what they use.

The vision was first outlined in the Intelligence Community Information Technology Enterprise plan championed by Director of National Intelligence James Clapper and IC Chief Information Officer Al Tarasiuk almost three years ago. Cloud computing is one of the core components of the strategy to help the IC discover, access and share critical information in an era of seemingly infinite data.

For the risk-averse intelligence community, the decision to go with a commercial cloud vendor is a radical departure from business as usual.

In 2011, while private companies were consolidating data centers in favor of the cloud and some civilian agencies began flirting with cloud variants like email as a service, a sometimes contentious debate among the intelligence community's leadership took place.

... ... ...

The government was spending more money on information technology within the IC than ever before. IT spending reached $8 billion in 2013, according to budget documents leaked by former NSA contractor Edward Snowden. The CIA and other agencies feasibly could have spent billions of dollars standing up their own cloud infrastructure without raising many eyebrows in Congress, but the decision to purchase a single commercial solution came down primarily to two factors.

"What we were really looking at was time to mission and innovation," the former intelligence official said. "The goal was, 'Can we act like a large enterprise in the corporate world and buy the thing that we don't have, can we catch up to the commercial cycle? Anybody can build a data center, but could we purchase something more?

"We decided we needed to buy innovation," the former intelligence official said.

A Groundbreaking Deal

... ... ...

The Amazon-built cloud will operate behind the IC's firewall, or more simply: It's a public cloud built on private premises.

Intelligence agencies will be able to host applications or order a variety of on-demand services like storage, computing and analytics. True to the National Institute of Standards and Technology definition of cloud computing, the IC cloud scales up or down to meet the need.

In that regard, customers will pay only for services they actually use, which is expected to generate massive savings for the IC.

"We see this as a tremendous opportunity to sharpen our focus and to be very efficient," Wolfe told an audience at AWS' annual nonprofit and government symposium in Washington. "We hope to get speed and scale out of the cloud, and a tremendous amount of efficiency in terms of folks traditionally using IT now using it in a cost-recovery way."

... ... ...

For several years there hasn't been even a close challenger to AWS. Gartner's 2014 quadrant shows that AWS captures 83 percent of the cloud computing infrastructure market.

In the combined cloud markets for infrastructure and platform services, hybrid and private clouds-worth a collective $131 billion at the end of 2013-Amazon's revenue grew 67 percent in the first quarter of 2014, according to Gartner.

While the public sector hasn't been as quick to capitalize on cloud computing as the private sector, government spending on cloud technologies is beginning to jump.

Researchers at IDC estimate federal private cloud spending will reach $1.7 billion in 2014, and $7.7 billion by 2017. In other industries, software services are considered the leading cloud technology, but in the government that honor goes to infrastructure services, which IDC expects to reach $5.4 billion in 2017.

In addition to its $600 million deal with the CIA, Amazon Web Services also does business with NASA, the Food and Drug Administration and the Centers for Disease Control and Prevention. Most recently, the Obama Administration tapped AWS to host portions of HealthCare.gov.

[Jun 09, 2017] Amazon's S3 web-based storage service is experiencing widespread issues on Feb 28 2017

Jun 09, 2017 | techcrunch.com

Amazon's S3 web-based storage service is experiencing widespread issues, leading to service that's either partially or fully broken on websites, apps and devices upon which it relies. The AWS offering provides hosting for images for a lot of sites, and also hosts entire websites, and app backends including Nest.

The S3 outage is due to "high error rates with S3 in US-EAST-1," according to Amazon's AWS service health dashboard , which is where the company also says it's working on "remediating the issue," without initially revealing any further details.

Affected websites and services include Quora, newsletter provider Sailthru, Business Insider, Giphy, image hosting at a number of publisher websites, filesharing in Slack, and many more. Connected lightbulbs, thermostats and other IoT hardware is also being impacted, with many unable to control these devices as a result of the outage.

Amazon S3 is used by around 148,213 websites, and 121,761 unique domains, according to data tracked by SimilarTech , and its popularity as a content host concentrates specifically in the U.S. It's used by 0.8 percent of the top 1 million websites, which is actually quite a bit smaller than CloudFlare, which is used by 6.2 percent of the top 1 million websites globally – and yet it's still having this much of an effect.

Amazingly, even the status indicators on the AWS service status page rely on S3 for storage of its health marker graphics, hence why the site is still showing all services green despite obvious evidence to the contrary. Update (11:40 AM PT): AWS has fixed the issues with its own dashboard at least – it'll now accurately reflect service status as it continues to attempt to fix the problem .

• Update (11:57 AM PT): AWS says it believes they new "understand root cause" of the S3 issues, and are "working hard at repairing." It has not shared specifics of that cause.
• Update (12:15 PM PT): Network intelligence software provider ThousandEyes notes that all the packet loss for the ongoing issue appears to be happening in the Ashburn, VA area. Amazon has an AWS data center in Ashburn, whose exact location was revealed in a news story last year due to a fire during its construction.
• Update (12:54 PM PT): AWS says it's seeing "recovery for S3 object retrievals, listing and deletions" which means you're probably seeing avatars and other visuals assets come back in some spots. The company also says it expects further improvements to error rates within the next hour.
• Update (1:20 PM PT): S3 is now fully recovered in terms of the retrieval, listing and deletion of existing objects, according to the AWS status page, and it's now working on restoring normal operation for the addition of new items to S3-based storage.
• Update (2:10 PM PT): AWS says that it's now fully recovered in terms of resolving the error rates it was seeing, and S3 service is now "operating normally."

[Apr 01, 2017] Amazon Web Services outage causes widespread internet problems

Apr 01, 2017 | www.cbsnews.com

Feb 28, 2017 6:03 PM EST NEW YORK -- Amazon's cloud-computing service, Amazon Web Services, experienced an outage in its eastern U.S. region Tuesday afternoon, causing unprecedented and widespread problems for thousands of websites and apps.

Amazon is the largest provider of cloud computing services in the U.S. Beginning around midday Tuesday on the East Coast, one region of its "S3" service based in Virginia began to experience what Amazon, on its service site, called "increased error rates."

In a statement, Amazon said as of 4 p.m. E.T. it was still experiencing "high error rates" that were "impacting various AWS services."

"We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue," the company said.

But less than an hour later, an update offered good news: "As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally," the company said.

Amazon's Simple Storage Service, or S3, stores files and data for companies on remote servers. It's used for everything from building websites and apps to storing images, customer data and customer transactions.

"Anything you can think about storing in the most cost-effective way possible," is how Rich Mogull, CEO of data security firm Securosis, puts it.

Amazon has a strong track record of stability with its cloud computing service, CNET senior editor Dan Ackerman told CBS News.

"AWS... is known for having really good 'up time,'" he said, using industry language.

Over time, cloud computing has become a major part of Amazon's empire.

"Very few people host their own web servers anymore, it's all been outsourced to these big providers , and Amazon is one of the major ones," Ackerman said.

The problem Tuesday affected both "front-end" operations -- meaning the websites and apps that users see -- and back-end data processing that takes place out of sight. Some smaller online services, such as Trello, Scribd and IFTTT, appeared to be down for a while, although all have since recovered.

Some affected websites had fun with the crash, treating it like a snow day:

[Apr 01, 2017] After Amazon outage, HealthExpense worries about cloud lock-in by Maria Korolov

Notable quotes:

"... "From a sustainability and availability standpoint, we definitely need to look at our strategy to not be vendor specific, including with Amazon," said Lee. "That's something that we're aware of and are working towards." ..."

"... "Elastic load balances and other services make it easy to set up. However, it's a double-edged sword, because these types of services will also make it harder to be vendor-agnostic. When other cloud platform don't offer the same services, how do you wean yourself off of them?" ..."

"... Multi-year commitments are another trap, he said. And sometimes there's an extra unpleasant twist -- minimum usage requirements that go up in the later years, like balloon payments on a mortgage. ..."

Apr 01, 2017 | www.networkworld.com

The Amazon outage reminds companies that having all their eggs in one cloud basket might be a risky strategy

"That is the elephant in the room these days," said Lee. "More and more companies are starting to move their services to the cloud providers. I see attackers trying to compromise the cloud provider to get to the information."

If attackers can get into the cloud systems, that's a lot of data they could have access to. But attackers can also go after availability.

"The DDoS attacks are getting larger in scale, and with more IoT systems coming online and being very hackable, a lot of attackers can utilize that as a way to do additional attacks," he said.

And, of course, there's always the possibility of a cloud service outage for other reasons.

The 11-hour outage that Amazon suffered in late February was due to a typo, and affected Netflix, Reddit, Adobe and Imgur, among other sites.

"From a sustainability and availability standpoint, we definitely need to look at our strategy to not be vendor specific, including with Amazon," said Lee. "That's something that we're aware of and are working towards."

The problem is that Amazon offers some very appealing features.

"Amazon has been very good at providing a lot of services that reduce the investment that needs to be made to build the infrastructure," he said. "Elastic load balances and other services make it easy to set up. However, it's a double-edged sword, because these types of services will also make it harder to be vendor-agnostic. When other cloud platform don't offer the same services, how do you wean yourself off of them?"

... ... ...

"If you have a containerized approach, you can run in Amazon's container services, or on Azure," said Tim Beerman, CTO at Ensono , a managed services provider that runs its own cloud data center, manages on-premises environments for customers, and also helps clients run in the public cloud.

"That gives you more portability, you can pick something up and move it," he said.

But that, too, requires advance planning.

"It starts with the application," he said. "And you have to write it a certain way."

But the biggest contributing factor to cloud lock-in is data, he said.

"They make it really easy to put the data in, and they're not as friendly about taking that data out," he said.

The lack of friendliness often shows up in the pricing details.

"Usually the price is lower for data transfers coming into a cloud service provider versus the price to move data out," said Thales' Radford.

Multi-year commitments are another trap, he said. And sometimes there's an extra unpleasant twist -- minimum usage requirements that go up in the later years, like balloon payments on a mortgage.

[Mar 03, 2017] Do You Replace Your Server Or Go To The Cloud The Answer May Surprise You

Mar 03, 2017 | www.forbes.com

Is your server or servers getting old? Have you pushed it to the end of its lifespan? Have you reached that stage where it's time to do something about it? Join the crowd. You're now at that decision point that so many other business people are finding themselves this year. And the decision is this: do you replace that old server with a new server or do you go to: the cloud.

Everyone's talking about the cloud nowadays so you've got to consider it, right? This could be a great new thing for your company! You've been told that the cloud enables companies like yours to be more flexible and save on their IT costs. It allows free and easy access to data for employees from wherever they are, using whatever devices they want to use. Maybe you've seen the recent survey by accounting software maker MYOB that found that small businesses that adopt cloud technologies enjoy higher revenues. Or perhaps you've stumbled on this analysis that said that small businesses are losing money as a result of ineffective IT management that could be much improved by the use of cloud based services. Or the poll of more than 1,200 small businesses by technology reseller CDW which discovered that " cloud users cite cost savings, increased efficiency and greater innovation as key benefits" and that " across all industries, storage and conferencing and collaboration are the top cloud services and applications."

So it's time to chuck that old piece of junk and take your company to the cloud, right? Well just hold on.

There's no question that if you're a startup or a very small company or a company that is virtual or whose employees are distributed around the world, a cloud based environment is the way to go. Or maybe you've got high internal IT costs or require more computing power. But maybe that's not you. Maybe your company sells pharmaceutical supplies, provides landscaping services, fixes roofs, ships industrial cleaning agents, manufactures packaging materials or distributes gaskets. You are not featured in Fast Company and you have not been invited to presenting at the next Disrupt conference. But you know you represent the very core of small business in America. I know this too. You are just like one of my company's 600 clients. And what are these companies doing this year when it comes time to replace their servers?

These very smart owners and managers of small and medium sized businesses who have existing applications running on old servers are not going to the cloud. Instead, they've been buying new servers.

Wait, buying new servers? What about the cloud?

At no less than six of my clients in the past 90 days it was time to replace servers. They had all waited as long as possible, conserving cash in a slow economy, hoping to get the most out of their existing machines. Sound familiar? But the servers were showing their age, applications were running slower and now as the companies found themselves growing their infrastructure their old machines were reaching their limit. Things were getting to a breaking point, and all six of my clients decided it was time for a change. So they all moved to cloud, right?

Nope. None of them did. None of them chose the cloud. Why? Because all six of these small business owners and managers came to the same conclusion: it was just too expensive. Sorry media. Sorry tech world. But this is the truth. This is what's happening in the world of established companies.

Consider the options. All of my clients' evaluated cloud based hosting services from Amazon , Microsoft and Rackspace . They also interviewed a handful of cloud based IT management firms who promised to move their existing applications (Office, accounting, CRM, databases) to their servers and manage them offsite. All of these popular options are viable and make sense, as evidenced by their growth in recent years. But when all the smoke cleared, all of these services came in at about the same price: approximately $100 per month per user. This is what it costs for an existing company to move their existing infrastructure to a cloud based infrastructure in 2013. We've got the proposals and we've done the analysis.

You're going through the same thought process, so now put yourself in their shoes. Suppose you have maybe 20 people in your company who need computer access. Suppose you are satisfied with your existing applications and don't want to go through the agony and enormous expense of migrating to a new cloud based application. Suppose you don't employ a full time IT guy, but have a service contract with a reliable local IT firm.

Now do the numbers: $100 per month x 20 users is $2,000 per month or $24,000 PER YEAR for a cloud based service. How many servers can you buy for that amount? Imagine putting that proposal out to an experienced, battle-hardened, profit generating small business owner who, like all the smart business owners I know, look hard at the return on investment decision before parting with their cash.

For all six of these clients the decision was a no-brainer: they all bought new servers and had their IT guy install them. But can't the cloud bring down their IT costs? All six of these guys use their IT guy for maybe half a day a month to support their servers (sure he could be doing more, but small business owners always try to get away with the minimum). His rate is $150 per hour. That's still way below using a cloud service.

No one could make the numbers work. No one could justify the return on investment. The cloud, at least for established businesses who don't want to change their existing applications, is still just too expensive.

Please know that these companies are, in fact, using some cloud-based applications. They all have virtual private networks setup and their people access their systems over the cloud using remote desktop technologies. Like the respondents in the above surveys, they subscribe to online backup services, share files on DropBox and Microsoft MSFT +1.45% 's file storage, make their calls over Skype, take advantage of Gmail and use collaboration tools like Google GOOG +1.45% Docs or Box. Many of their employees have iPhones and Droids and like to use mobile apps which rely on cloud data to make them more productive. These applications didn't exist a few years ago and their growth and benefits cannot be denied.

Paul-Henri Ferrand, President of Dell DELL +% North America, doesn't see this trend continuing. "Many smaller but growing businesses are looking and/or moving to the cloud," he told me. "There will be some (small businesses) that will continue to buy hardware but I see the trend is clearly toward the cloud. As more business applications become more available for the cloud, the more likely the trend will continue."

[Jan 12, 2017] Digitopoly Congestion on the Last Mile

Notable quotes:

"... By Shane Greenstein On Jan 11, 2017 · Add Comment · In Broadband , communication , Esssay , Net Neutrality ..."

"... The bottom line: evenings require far greater capacity than other times of the day. If capacity is not adequate, it can manifest as a bottleneck at many different points in a network-in its backbone, in its interconnection points, or in its last mile nodes. ..."

"... The use of tiers tends to grab attention in public discussion. ISPs segment their users. Higher tiers bring more bandwidth to a household. All else equal, households with higher tiers experience less congestion at peak moments. ..."

"... such firms (typically) find clever ways to pile on fees, and know how to stymie user complaints with a different type of phone tree that makes calls last 45 minutes. Even when users like the quality, the aggressive pricing practices tend to be quite irritating. ..."

"... Some observers have alleged that the biggest ISPs have created congestion issues at interconnection points for purposes of gaining negotiating leverage. These are serious charges, and a certain amount of skepticism is warranted for any broad charge that lacks specifics. ..."

"... Congestion is inevitable in a network with interlocking interests. When one part of the network has congestion, the rest of it catches a cold. ..."

"... More to the point, growth in demand for data should continue to stress network capacity into the foreseeable future. Since not all ISPs will invest aggressively in the presence of congestion, some amount of congestion is inevitable. So, too, is a certain amount of irritation. ..."

Jan 12, 2017 | www.digitopoly.org

Congestion on the Last Mile

By Shane Greenstein On Jan 11, 2017 · Add Comment · In Broadband , communication , Esssay , Net Neutrality It has long been recognized that networked services contain weak-link vulnerabilities. That is, the performance of any frontier device depends on the performance of every contributing component and service. This column focuses on one such phenomenon, which goes by the label "congestion." No, this is not a new type of allergy, but, as with a bacteria, many users want to avoid it, especially advanced users of frontier network services.

Congestion arises when network capacity does not provide adequate service during heavy use. Congestion slows down data delivery and erodes application performance, especially for time-sensitive apps such as movies, online videos, and interactive gaming.

Concerns about congestion are pervasive. Embarrassing reports about broadband networks with slow speeds highlight the role of congestion. Regulatory disputes about data caps and pricing tiers question whether these programs limit the use of data in a useful way. Investment analysts focus on the frequency of congestion as a measure of a broadband network's quality.

What economic factors produce congestion? Let's examine the root economic causes.

The Basics

Congestion arises when demand for data exceeds supply in a very specific sense.

Start with demand. To make this digestible, let's confine our attention to US households in an urban or suburban area, which produces the majority of data traffic.

No simple generalization can characterize all users and uses. The typical household today uses data for a wide variety of purposes-email, video, passive browsing, music videos, streaming of movies, and e-commerce. Networks also interact with a wide variety of end devices-PCs, tablets, smartphones on local Wi-Fi, streaming to television, home video alarm systems, remote temperature control systems, and plenty more.

It is complicated, but two facts should be foremost in this discussion. First, a high fraction of traffic is video-anywhere from 60 to 80 percent, depending on the estimate. Second, demand peaks at night. Most users want to do more things after dinner, far more than any other time during the day.

Every network operator knows that demand for data will peak (predictably) between approximately 7 p.m. and 11 p.m. Yes, it is predictable. Every day of the week looks like every other, albeit with steady growth over time and with some occasional fluctuations for holidays and weather. The weekends don't look any different, by the way, except that the daytime has a bit more demand than during the week.

The bottom line: evenings require far greater capacity than other times of the day. If capacity is not adequate, it can manifest as a bottleneck at many different points in a network-in its backbone, in its interconnection points, or in its last mile nodes.

This is where engineering and economics can become tricky to explain (and to manage). Consider this metaphor (with apologies to network engineers): Metaphorically speaking, network congestion can resemble a bathtub backed up with water. The water might fail to drain because something is interfering with the mouth of the drain or there is a clog far down the pipes. So, too, congestion in a data network can arise from inadequate capacity close to the household or inadequate capacity somewhere in the infrastructure supporting delivery of data.

Numerous features inside a network can be responsible for congestion, and that shapes which set of households experience congestion most severely. Accordingly, numerous different investments can alleviate the congestion in specific places. A network could require a "splitting of nodes" or a "larger pipe" to support a content delivery network (CDN) or could require "more ports at the point of interconnection" between a particular backbone provider and the network.

As it turns out, despite that complexity, we live in an era in which bottlenecks arise most often in the last mile, which ISPs build and operate. That simplifies the economics: Once an ISP builds and optimizes a network to meet maximum local demand at peak hours, then that same capacity will be able to meet lower demand the rest of the day. Similarly, high capacity can also address lower levels of peak demand on any other day.

Think of the economics this way. An awesome network, with extraordinary capacity optimized to its users, will alleviate congestion at most households on virtually every day of the week, except the most extraordinary. Accordingly, as the network becomes less than awesome with less capacity, it will generate a number of (predictable) days of peak demand with severe congestion throughout the entire peak time period at more households. The logic carries through: the less awesome the network, the greater the number of households who experience those moments of severe congestion, and the greater the frequency.

That provides a way to translate many network engineering benchmarks-such as the percentage of packet loss. More packet loss correlates with more congestion, and that corresponds with a larger number of moments when some household experiences poor service.

Tradeoffs and Externalities

Not all market participants react to congestion in the same way. Let's first focus on the gazillion Web firms that supply the content. They watch this situation with a wary eye, and it's no wonder. Many third-party services, such as those streaming video, deliver a higher-quality experience to users whose network suffers less congestion.

Many content providers invest to alleviate congestion. Some invest in compression software and superior webpage design, which loads in ways that speeds up the user experience. Some buy CDN services to speed delivery of their data. Some of the largest content firms, such as YouTube, Google, Netflix, and Facebook, build their own CDN services to improve delivery.

Next, focus on ISPs. They react with various investment and pricing strategies. At one extreme, some ISPs have chosen to save money by investing conservatively, and they suffer the complaints of users. At the other extreme, some ISPs build a premium network, then charge premium prices for the best services.

There are two good reasons for that variety. First, ISPs differ in their rates of capital investment. Partly this is due to investment costs, which vary greatly with density, topography, and local government relations. Rates of investment tend to be inherited from long histories, sometimes as a product of decisions made many years ago, which accumulated over time. These commitments can change, but generally don't, because investors watch capital commitments and react strongly to any departure from history.

The second reason is more subtle. ISPs take different approaches to raising revenue per household, and this results in (effectively) different relationships with banks and stockholders, and, de facto, different budgets for investment. Where does the difference in revenue come from? For one, competitive conditions and market power differ across neighborhoods. In addition, ISPs use different pricing strategies, taking substantially different approaches to discounts, tiered pricing structures, data cap policies, bundled contract offerings, and nuisance fees.

The use of tiers tends to grab attention in public discussion. ISPs segment their users. Higher tiers bring more bandwidth to a household. All else equal, households with higher tiers experience less congestion at peak moments.

Investors like tiers because they don't obligate ISPs to offer unlimited service and, in the long run, raise revenue without additional costs. Users have a more mixed reaction. Light users like the lower prices of lower tiers, and appreciate saving money for doing little other than email and static browsing.

In contrast, heavy users perceive that they pay extra to receive the bandwidth that the ISP used to supply as a default.

ISPs cannot win for losing. The archetypical conservative ISP invests adequately to relieve congestion some of the time, but not all of the time. Its management then must face the occasional phone calls of its users, which they stymie with phone trees that make service calls last 45 minutes. Even if users like the low prices, they find the service and reliability quite irritating.

The archetypical aggressive ISP, in contrast, achieves a high-quality network, which relieves severe congestion much of the time. Yet, such firms (typically) find clever ways to pile on fees, and know how to stymie user complaints with a different type of phone tree that makes calls last 45 minutes. Even when users like the quality, the aggressive pricing practices tend to be quite irritating.

One last note: It is a complicated situation where ISPs interconnect with content providers. Multiple parties must invest, and the situations involve many supplier interests and strategic contingencies.

Some observers have alleged that the biggest ISPs have created congestion issues at interconnection points for purposes of gaining negotiating leverage. These are serious charges, and a certain amount of skepticism is warranted for any broad charge that lacks specifics.

Somebody ought to do a sober and detailed investigation to confront those theories with evidence. (I am just saying.)

What does basic economics tell us about congestion? Congestion is inevitable in a network with interlocking interests. When one part of the network has congestion, the rest of it catches a cold.

More to the point, growth in demand for data should continue to stress network capacity into the foreseeable future. Since not all ISPs will invest aggressively in the presence of congestion, some amount of congestion is inevitable. So, too, is a certain amount of irritation.

Copyright held by IEEE.

To view the printed essay, click here.

[Nov 09, 2015] Thoughts on the Amazon outage

Notable quotes:

"... The 'Cloud' isn't magic, the 'Cloud' isn't fail-proof, the 'Cloud' requires hardware, software, networking, security, support and execution – just like anything else. ..."

"... Putting all of your eggs in one cloud, so to speak, no matter how much redundancy they say they have seems to be short-sighted in my opinion. ..."

"... you need to assume that all vendors will eventually have an issue like this that affects your overall uptime, brand and churn rate. ..."

"... Amazon's downtime is stratospherically high, and their prices are spectacularly inflated. Their ping times are terrible and they offer little that anyone else doesn't offer. Anyone holding them up as a good solution without an explanation has no idea what they're talking about. ..."

"... Nobody who has even a rudimentary best-practice hosting setup has been affected by the Amazon outage in any way other than a speed hit as their resources shift to a secondary center. ..."

"... Stop following the new-media goons around. They don't know what they're doing. There's a reason they're down twice a month and making excuses. ..."

"... Personally, I do not use a server for "mission critical" applications that I cannot physically kick. Failing that, a knowledgeable SysAdmin that I can kick. ..."

nickgeoghegan.net

Disaster Recovery needs to be a primary objective when planning and implementing any IT project, outsourced or not. The 'Cloud' isn't magic, the 'Cloud' isn't fail-proof, the 'Cloud' requires hardware, software, networking, security, support and execution – just like anything else.

All the fancy marketing speak, recommendations and free trials, can't replace the need to do obsessive due diligence before trusting any provider no matter how big and awesome they may seem or what their marketing department promise.

• Why do Data Centers have UPS and Diesel Generators on-site? They know electricity can and does fail.
• Why do we buy servers will dual power supplies? We know they can and do fail.
• Why do we implement RAID? We know hard drives can and do fail.

Prepare for the worst, period.

Putting all of your eggs in one cloud, so to speak, no matter how much redundancy they say they have seems to be short-sighted in my opinion. If you are utilizing an MSP, HSP, CSP, IAAS, SAAS, PAAS, et all to attract/increase/fulfill a large percentage of your revenue or all of your revenue like many companies are doing nowadays then you need to assume that all vendors will eventually have an issue like this that affects your overall uptime, brand and churn rate. A blip here and there is tolerable.

Amazon's downtime is stratospherically high, and their prices are spectacularly inflated. Their ping times are terrible and they offer little that anyone else doesn't offer. Anyone holding them up as a good solution without an explanation has no idea what they're talking about.

The same hosting platform, as always, is preferred: dedicated boxes at geographically disparate and redundant locations, managed by different companies. That way when host 1 shits the bed, hosts 2 and 3 keep churning.

Nobody who has even a rudimentary best-practice hosting setup has been affected by the Amazon outage in any way other than a speed hit as their resources shift to a secondary center.

Stop following the new-media goons around. They don't know what they're doing. There's a reason they're down twice a month and making excuses.

Personally, I do not use a server for "mission critical" applications that I cannot physically kick. Failing that, a knowledgeable SysAdmin that I can kick.

MAT Monitoring and Administration Tool

MAT is an easy to use network enabled UNIX configuration and monitoring tool. It provides an integrated tool for many common system administration tasks, including Backups and Replication It includes a warning system for potential system problems, and graphing of many common system parameters. Click here for more.

Coming soon in version 0.24 will be an embedded interpreter with it you will be able to monitor any parameter you can write a script to capture. It also will create the ability to have OS specific configuration tools.

Unix SysAdm Resources Automated Unix SysMgmt Software

• Master System is a public-domain Unix systems configuration tool written in Perl. The system is architecture and operating system independent, but it can handle architecture and operating system dependent configuration. It is designed to control the configuration of large groups of systems that are configured in the same style, but are not necessarily identical. From a group at Rutgers University.
  
• Webmin is a free web-based admin interface for Unix systems. Via a web browser, you can configure DNS, Apache, Samba, filesystems, startup scripts, inetd, crontabs and more. Written in Perl5 and easily extendable. Supports several Linux versions and Solaris.

Recommended Links

Softpanorama Recommended

Top articles

[Mar 29, 2020] Why Didn't We Test Our Trade's 'Antifragility' Before COVID-19 by Gene Callahan and Joe Norman Published on Mar 28, 2020 | www.theamericanconservative.com

Sites

Conveniently Administering Remote Servers

What is edge computing and how it's changing the network Network World

Edge computing - Wikipedia

What's the difference between edge computing and cloud computing - Quora

Bandwidth alone won't solve application performance problems Network World

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: February 28, 2021