We're entering our third year of Velocity, the Web Performance & Operations Conference.. Velocity 2010 will be June 22-24, 2010 in Santa Clara, CA. It's going to be another incredible year.

Steve & I have set a new theme this year, "Fast by Default".  We want the broader Velocity community & to adopt it as a shared mission & mantra. The reason for this is simple...

Fast isn't a Feature. Fast is a Requirement.

At Velocity earlier this year Marissa Meyer explained why performance mattered so much to Google. Then Eric Schurman (Bing & Velocity Program Committee member) and Jake Brutlag (Google Search) made history with a co-presentation on just how crucial performance is to revenue .

Phil Dixon of Shopzilla explained that a 5" second performance improvement increased their revenue by 7-12 percent":http://http://velocityconference.blip.tv/file/2290648/ while reducing hardware spend by 50%!!!

Fast means Client, Server, Infrastructure, Operations, & Organizations

Getting to Fast isn't just about any one part of the system. Browser & Client performance is crucial, and requires an equally fast server & infrastructure to support it. When load increases, infrastructure must scale quickly or performance suffers. The operational tools and processes for managing software & infrastructure must support rapid changes in a dynamic environment, and be backed by an organization & culture that embraces it.

We're Looking for Speakers - Submit your Proposal by January 11

See the full schedule after the jump...

At Velocity this year Microsoft, Google and Shopzilla each presented data on how web performance directly impacts revenue.

Their data showed that slow sites get fewer search queries per user, less revenue per visitor, fewer clicks, fewer searches, and lower search engine rankings. They found that in some cases even after site performance was improved users continued to interact as if it was slow. Bad experiences have a lasting influence on customer behavior.

What about smaller websites that aren't yet at this scale?

Alistair Croll and Sean Power, the authors of the new book Complete Web Monitoring, have continued this research for sites at smaller scale.

They used a Strangeloop Networks web acceleration appliance to optimize half the sessions to a smaller production website, tagging optimized and unoptimized visitors so they could be analyzed in Google Analytics. The Strangeloop device applies many of Steve Souders' performance rules to an existing site automatically (a kind of "Steve-in-a-Box" ;-).

The results of their analysis show how significant a reduction in page latency can be. In addition to reducing bounce rates, and increasing pages per visit & time on site, they found a 16.07% increase in conversion rates and a 5.50% increase in average order value.

Check out the full post on the Watching Websites blog.

Twitter is suffering outages today as they fend off a Denial of Service attack, and so I thought it would be helpful to post John Adams’ exceptional Velocity session about Operations at Twitter.

Good luck today John & team… I know it’s going to be a long day!

Update: Apparently Facebook & Livejournal have had similar attacks today. Rich Miller from Data Center Knowledge reminds us that this is just the latest in a series of major attacks.

We were honored to have Jonathan Heiliger, Facebook’s VP of Technology Operations, as our opening keynote speaker at Velocity. Jonathan is one of the most accomplished leaders in our field, and is a master of the craft.

Here is his keynote in its entirety:

Note: Other videos from Velocity are being posted to VelocityConference.blip.tv

Ignite! is coming to San Jose on Monday June 22, 2009 at 8:00 pm, attached to the Velocity Conference. Admission is free, open to all, and there will be a cash bar.

The deadline for talks is May 11th, so submit your talks now!

As with all Ignites each speaker will only get 20 slides that each auto-advance every 15 seconds for a total of five minutes. We'll be looking for fun geek topics like hacks, how-to's, and insights. (Talks don't have to be Velocity-related!) If you're not sure what an Ignite talk looks like check out the Ignite Show.

You can RSVP for the event on Upcoming or Facebook.

CrisisCamp is an unconference to bring together domain experts, hackers, makers, developers, and first responders to improve technology and practice for humanitarian crisis management and disaster relief. This is the first event in what I hope will become a movement, and it's happening on June 12 - 14, 2009 in Washington, DC.

Across the world, everyday people can find themselves in crisis. Whether for a day, a month or a continued state of social distress, citizens across the world have common needs for communication. We want to connect with our loved ones to let them know that we are okay (or that we need help) and we crave information by which we make decisions within that time or place of crisis. We want to let people know they they are not alone, that people across the world care and seek to act act altruistically to provide resources to aid in the crisis recovery.

CrisisCamp seeks to break down the bifurcation between international and domestic humanitarian relief agencies and unite their efforts to share lessons learned, response tools, and expertise to encourage citizen engagement and use of information communications technologies (ICTs) to aid in crisis recovery, wherever people need help. We have great hopes that with a successful CrisisCamp, we can inspire a global effort to mitigate the potential impact of times or places of crisis on the human condition.

Crisis Camp seeks participation by anyone who wishes to help. We are especially interested in the use of ICTs in developing countries, especially in the areas of access, usability, and innovation. We seek to learn from academic findings on citizen participation, needs and problem solving efforts. In addition, we seek to understand global information needs through a consumer approach, because people will use what is available and familiar if a crisis event occurs. And finally, we want to know how ICTs, in all their uses, can help citizens of all abilities, recover during a time or place of crisis.

CrisisCamp Ignite! Session Kick Off

Time: Friday, June 12, 2009 from 7:30-9PM
Location: The World Bank,1818 H St., NW Washington D.C.
Participate: Sign up at CrisisCampIgnite (separate registration required for World Bank entry)

CrisisCamp - Saturday, June 13 & Sunday, June 14th

Start Time: 9:00am both days
Location: The Institute for Politics Democracy & the Internet @ George Washington University Participate: Sign up at CrisisCamp

According to William Castleman: The time-lapse sequence was taken with the simplest equipment that I brought to the star party. I put the Canon EOS-5D (AA screen modified to record hydrogen alpha at 656 nm) with an EF 15mm f/2.8 lens on a weighted tripod. Exposures were 20 seconds at f/2.8 ISO 1600 followed by 40 second interval. Exposures were controlled by an interval timer shutter release (Canon TC80N3). Power was provided by a Hutech EOS203 12v power adapter run off a 12v deep cycle battery. Large jpg files shot in custom white balance were batch processed in Photoshop (levels, curves, contrast, Noise Ninja noise reduction, resize) and assembled in Quicktime Pro. Editing/assembly was with Sony Vegas Movie Studio 9.

[via the Primary Tentacle @ Laughing Squid]

In this tightly cropped image, the NASA space shuttle Atlantis is seen in silhouette during solar transit, Tuesday, May 12, 2009, from Florida. This image was made before Atlantis and the crew of STS-125 had grappled the Hubble Space Telescope. Photo Credit: (NASA/Thierry Legault)

Thierry made this image using a solar-filtered Takahashi 5-inch refracting telescope and a Canon 5D Mark II digital camera. Photo Credit: (NASA/Thierry Legault)

You can see more of Thierry's fine work at: www.astrophoto.fr

from nasa hq photostream [via slashdot]

While the actual process of serving and rendering a page takes seconds to complete, describing it in real detail can take an hour. A good answer spans every part of the Internet from the client browser & operating system, DNS, through the network, to load balancers, servers, services, storage, down to the operating system & hardware, and all the way back again to the browser. It requires an understanding of TCP/IP, HTTP, & SSL deep enough to describe how connections are managed, how load-balancers work, and how certificates are exchanged and validated... and that's just the first request!

Web Performance & Operations is an emerging discipline which requires incredible breadth, focusing less on specific technologies and more on how the entire system works together. While people often specialize on particular components, great engineers always think of that component in relation to the whole. The best engineers are able to fly to the 50,000 foot view and see the entire system in motion and then zoom in to microscopic levels and examine the tiny movements of an individual part.

John Allspaw recently described this interconnectedness on his blog:

With websites, the introduction of change (for example, a bad database query) can affect (in a bad way) the entire system, not just the component(s) that saw the change. Adding handfuls of milliseconds to a query that’s made often, and you’re now holding page requests up longer. The same thing applies to optimizations as well. Break that [bad] query into two small fast ones, and watch how usage can change all over the system pretty quickly. Databases respond a bit faster, pages get built quicker, which means users click on more links, etc. This second-order effect of optimization is probably pretty familiar to those of us running sites of decent scale.

Working with these systems requires an understanding not only of the way technology interacts, but the way that people do as well. The structure, operation, and development of a website mirrors the organization that creates it, which is why so many people in WebOps focus on understanding and improving management culture & process.

Organizing a conference like Velocity is a wonderful challenge because it requires the same sort of thinking. We focus on the big concepts that everyone needs to know and then go deep into the technologies that change our understanding of the system. We find ways to share the unique experience that can only be gained by operating at scale. We make it safe to share as much of the "Secret Sauce" as we can.

“Progress is not the mere correction of evils. Progress is the constant replacing of the best there is with something still better.” -Edward Filene

Two years ago, when we were organizing the first BarCampBank in the US, many people found it hard to believe that banks & credit unions could a place for meaningful grassroots innovation. Even crazier was the idea of organizing an unconference to begin bringing open source, transparency, identity, and community into the very closed world of banking & finance.

Since then the BarCampBank idea has turned into a movement. There have been over 14 events all over the world, and many of the ideas generated are beginning to turn into action.

To me, the global financial system is a platform must always create more value than it captures. Tim explained this in his Work on Stuff that Matters post, saying:

“A bank that loans money to a small business sees that business grow, perhaps borrow more money, hire employees who make deposits and take out loans, and so on. The power of this cycle to lift people out of poverty has been demonstrated by microfinance institutions like the Grameen Bank. Grameen is clearly focused on creating more value than they capture; not so the like of Fannie Mae and Freddy Mac, or WaMu, or many of the other failed financial institutions involved in the current financial meltdown.”

There has never been a more important time to bring meaningful innovation into the financial system, and there has never been more opportunity for our community to make it happen.

The next event is occurring this weekend (April 25-26, 2009) on Treasure Island in San Francisco.

After that, the following events are planned:

• BarCampBankVegas is set for May 2, 2009.
• BarCampBankCharleston2 is set for June 13, 2009
• BarCampBankGermany is set for October 23-25, 2009

James Hamilton is one of the smartest and most accomplished engineers I know. He now leads Microsoft's Data Center Futures Team, and has been pushing the opportunities in data center efficiency and internet scale services both inside & outside Microsoft. His most recent post explores misconceptions about the Cost of Power in Large-Scale Data Centers:

I’m not sure how many times I’ve read or been told that power is the number one cost in a modern mega-data center, but it has been a frequent refrain. And, like many stories that get told and retold, there is an element of truth to the it. Power is absolutely the fastest growing operational costs of a high-scale service. Except for server hardware costs, power and costs functionally related to power usually do dominate.

However, it turns out that power alone itself isn’t anywhere close to the most significant a cost. Let’s look at this more deeply. If you amortize power distribution and cooling systems infrastructure over 15 years and amortize server costs over 3 years, you can get a fair comparative picture of how server costs compare to infrastructure (power distribution and cooling). But how to compare the capital costs of server, and power and cooling infrastructure with that monthly bill for power?

The approach I took is to convert everything into a monthly charge. [...]

I'm excited to announce that joining Steve Souders & I on this year's program committee are John Allspaw, Artur Bergman, Scott Ruthfield, Eric Schurman, and Mandi Walls.  We've already started working on the program, and have just opened the Call for Participation.

We're especially interested in the following topics:

• How to tie web performance and operations to the bottom line
• Real-world incident management - getting “tight like a pit crew”
• Making websites as fast and reliable as desktop apps
• Networking, DNS, and load balancing
• Profiling everywhere: JavaScript, CSS, and the network
• Managing web services - flaming disasters you survived and lessons learned
• The intersection between performance and design
• Wicked cool (and actionable) metrics
• Ads, ads, ads - the performance killer?
• Troubleshooting in production
• How to scale and be fast on the social web
• Capacity planning and load testing
• Establishing performance and operations best practices within your organization
• Configuration management best (and worst) tools and practices
• Monitoring and instrumentation experiences: Open Source, as a service, commercially supported solutions
• Using multiple CDNs to improve customer experience and reduce cost

Last week marked an important milestone for the "Web as Platform" as the 1,000 API was added to the ProgrammableWeb registry. John Musser (see: Web2.0 Report) started tracking the first few web service API's back in 2005.

How do these 1000 APIs break down by type? The following chart, derived from our database, shows the the top 15 sectors or markets with the greatest number of competing API providers. As you can see there are already 71 mapping-related APIs alone"

Congratulations!

One of the most interesting DisasterTech projects I've been following is "Decisions for Heroes" led by developer and Irish Coast Guard volunteer Robin Blandford.

Decisions is like Basecamp for volunteer Search & Rescue teams. The focus is on providing "just enough" process to compliment the real-world workflow of a rescue team, without unnecessary complexity. One of Robin's design goals is that:

User requirements are nil. Nobody likes reading manuals - if we have to write one, we've gotten too complicated.

This is the winning approach for building systems that "serve those that serve others", and is echoed by InSTEDD's design philosophy and the Sahana disaster management system.

Teams begin by entering their responses to incidents and training exercises. They then tag them with things like the weather conditions, the tools and skills required, and who from the team was deployed.

As a team's incident database grows this information can be used to show heatmaps, and provide powerful insight on the locations, weather conditions, and times of year that various incidents occur. Over time this kind of data could be analyzed in aggregate across multiple teams and regions and create an incredibly powerful resource for Emergency Managers. This is very similar to what Wesabe does for consumers with financial transaction data today (disclosure: OATV investment).

Rescue team members enter training dates and levels. The system tracks certification expiration dates and prompts team members & leaders to plan classes and remain current. This is a huge issue for volunteers who have to manage professional-level training requirements with the demands of a regular career.

As more incidents are entered into the system, it compares the skills required for each of the rescues with the team training exercises. This allows teams to identify areas to focus, train, and develop new skills.

It appears that Sprint has stopped routing traffic (called "depeering") from Cogent as a result of some sort of legal dispute. Sprint customers cannot reach Cogent customers, and vice versa. The effect is similar to what would happen if Sprint were to block voice phonecalls to AT&T customers.

Here's a graph that shows the outage, courtesy of Keynote :

Rich Miller at DataCenterKnowledge has a great summary of the issues behind the incident, which has happened with Cogent before. Rich says:

At the heart of it, peering disputes are really loud business negotiations, and angry customers can be used as leverage by either side. This one will end as they always do, with one side agreeing to pay up or manage their traffic differently.

I think this is particularly Radar-worthy because it provides an example of the complex issues around Net Neutrality.In this case customers are harmed and most (especially Sprint wireless customers) will have no immediate recourse.

Todd Underwood of Renesys has posted an incredibly detailed explanation the scope and impact of this issue. Here is a summary:

Another way to look at the scope of this event is to identify the number, size and ownership of the network prefixes affected by the outage. [...] So, in total, at least 3500 networks on the Internet have less than full connectivity right now. [...]

One might suspect that these single-homed autonomous systems are simply incautious or insignificant networks. After all, given the history of Internet partitions, who would be rash enough to have important network services located on a single-homed prefix in this day and age?

The following prefixes are some of the more interesting networks single-homed behind Sprint:

• 208.95.96.0/21 Expedia, Inc.
• 164.62.0.0/16 Federal Trade Commission
• 204.108.8.0/24 Federal Aviation Administration
• 198.9.201.0/24 National Aeronautics and Space Administration
• 170.189.200.0/24 Occidental Petroleum Corporation
• 148.168.0.0/16 Pfizer Inc.
• 128.6.0.0/16 Rutgers University
• 173.100.0.0/16 Sprint PCS (lots of networks here, of course)
• 149.24.174.0/23 SUNGARD HIGHER EDUCATION INC.

And that is just a few.

The following prefixes are some of the more interesting networks single-homed behind Cogent:

• 89.251.2.0/24 Joost Production Benelux Network
• 72.5.224.0/24 Loopt, Inc.
• 198.185.178.0/23 National Aeronautics and Space Administration and many more)
• 204.201.48.0/21 NTT America, Inc. (and many more like it, from the T1 and hosting customers acquired from NTT/Verio)
• 204.9.56.0/24 Skynet Access (this might actually be good news, if the loss of connectivity to Skynet prevents or delays sentience).
• 142.155.0.0/16 St. Lawrence College
• 128.100.0.0/16 University of Toronto (and a bunch of other colleges and universities)

[...] The point here is that this is a big deal. There are lots of significant organizations that appear to have lost connectivity due to this dispute.

These same kinds of issues will likely happen with cloud service providers as well. As we've already learned from the evolution of VoIP, you become what you disrupt.

Amazon announced a new SLA for EC2, similar to the one for S3. This is a notable step for Amazon and cloud computing as a whole, as it establishes a new bar for utility computing services.

Amazon is committing to 99.95% availability for the EC2 service on a yearly basis, which corresponds to approximately four hours and twenty three minutes of downtime per year. It's important to remember that an SLA is just a contract that provides a commitment to a certain level of performance and some form of compensation when a provider fails to meet it.

Service Commitment AWS will use commercially reasonable efforts to make Amazon EC2 available with an Annual Uptime Percentage (defined below) of at least 99.95% during the Service Year. In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit as described below. [...]

• “Annual Uptime Percentage” is calculated by subtracting from 100% the percentage of 5 minute periods during the Service Year in which Amazon EC2 was in the state of “Region Unavailable.” If you have been using Amazon EC2 for less than 365 days, your Service Year is still the preceding 365 days but any days prior to your use of the service will be deemed to have had 100% Region Availability [...]
• “Unavailable” means that all of your running instances have no external connectivity during a five minute period and you are unable to launch replacement instances. [...]

To receive a Service Credit, you must submit a request by sending an e-mail message to aws-sla-request @ amazon.com. To be eligible, the credit request must [...] include your server request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks)

This new SLA does not appear to address the reliability of server instances individually or in aggregate. For example, if half of a customer's EC2 instances lose their connections or die every 6 minutes, EC2 would still be considered "available" even if it is essentially unusable.

If the entire EC2 service is down a cumulative four hours and twenty minutes, customers must furnish proof of the outage to Amazon to be eligible for the 10% credit. This seems like an onerous process for very little compensation, and isn't in-line with Amazon's famous "Relentless Customer Obsession". Amazon takes monitoring very seriously and should take the lead by tracking, reporting, and proactively compensating customers when it lets them down.

[link courtesy Barry Brumitt]

When Apple announced the iPhone SDK last year I said:

[...] Jobs makes it clear that the platform won't be completely open. While he says that this is to balance the benefits of an open platform with user security protection, it's unclear where Apple will draw those lines. Will there be a Skype client? Third-party media apps?

It would have been better if Apple had announced [the details] when it released the iPhone. I'm hopeful that Apple will now embrace the existing iPhone developer community, and won't use “security” as a way to keep potential competitors off its platform.

According to the developer, Apple blocked this application from the App store, saying:

Since Podcaster assists in the distribution of podcasts, it duplicates the functionality of the Podcast section of iTunes.

If you want to build a platform, you have to compete fairly with the developers on your platform (if you must to compete at all). By restricting developers, Apple is stifling innovation and their long-term growth. Frustrated customers and developers who "think different" are Jailbreaking their iPhones and getting excited about Google's Android.

Remember: Successful platforms create more value than they capture.

Dan Kaminsky has posted the details of the widespread DNS vulnerability. Clarified Networks created this visualization of DNS patch deployment over the past month:

Red = Unpatched
Yellow = Patched, "but NAT is screwing things up"
Green = OK

Theo Schlossnagle, author of Scalable Internet Architectures, gave a great explanation of how internet traffic spikes are shifting:

Lately, I see more sudden eyeballs and what used to be an established trend seems to fall into a more chaotic pattern that is the aggregate of different spike signatures around a smooth curve. This graph is from two consecutive days where we have a beautiful comparison of a relatively uneventful day followed by long-exposure spike (nytimes.com) compounded by a short-exposure spike (digg.com):

The disturbing part is that this occurs even on larger sites now due to the sheer magnitude of eyeballs looking at today's already popular sites. Long story short, this makes planning a real bitch.

[...]What isn't entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we've experienced from 100% to 1000% sudden increases in traffic across many of our clients.

[Link]