Isaac Cambron



Previous page

Scale and Google's Customer Support

Oct 5, 2011

This Hacker News item is a complaint about a website being delisted by Google without a good, easily-understood explanation. Matt Cutts, who runs the webspam team at Google, jumped into the comment thread and provided some explanations. While the reasons he gave for delisting do make sense, one of the main counterarguments was that Google’s customer support for webmasters is terrible, and the company is rather opaque and abrupt about the process. Why did the site owner have to find out how to fix it by getting his post at the top of Hacker News?

In the thread, Cutts says, in reference to a premium support option, “Normally when we consider it, we end up saying things like ‘Why don’t we just try to make it so that people don’t need that option?’” And that’s fine; not needing customer support is better than needing it. But until they figure that out, Google does need it; you can’t just dismiss it with a hand wave and a vague goal. Google has a huge transparency problem. Their rankings and delisting decisions are opaque and seemingly arbitrary. For good evidence of this, just take a look at how quacky the gigantic SEO industry is; witchcraft shouldn’t seem like a viable option here. More generally, Google just doesn’t have customer support worth much of anything, and they just don’t seem to care about it. Why?

Cutts has an answer there, although I suppose I should point out that he’s not acting as some sort of official spokesperson for Google. In responding to the opacity/abruptness/support complaint, he says this:

I know that Google can seem abrupt sometimes, and I dislike that, but part of the issue is also scale. See… that notes that if each Google user had a single 10 minute issue every three years, that would need 20,000+ support people to handle that load. Or consider that there’s 200M domain names, and all those webmasters want to talk to Google and ask questions.

For the record, I think this kind of direct engagement by important employees is great. But I don’t buy Cutt’s scale angle at all. Here’s the thing: 20,000 seems like a lot of people, but it’s not. Consider this statement I just made up:

I know McDonald’s can be impersonal sometimes, and I dislike that, but part of the issue is also scale. If each McDonald’s customer needed to be served a burger in person by a McDonald’s employee, that would require 400,000 employees to physically make and serve all those burgers.

And, of course, that’s exactly what McDonald’s does, and you’d be really annoyed if you walked into McDonald’s one day and your burger was sitting premade on a little heated tray that had been filled with foil-wrapped food hours earlier, and there was no one working there. McDonald’s is actually a somewhat smaller company than Google ($24 billion vs $29 billion) and has far lower profits ($5 billion vs $9 billion). So Google makes a lot more money per dollar it spends. Those are obviously very different companies, and there are a lot of reasons McDonald’s has lower profits than Google, but one of them is that Google doesn’t employ tens of thousands of people to do customer support. But maybe it should!

We could debate whether 20,000 support people is enough, or actually way more than they’d ever need, or whatever, but that’s not the point. The fact is that Google isn’t having difficulty scaling its customer support team. It’s just not trying to build one at all. Yet it could clearly afford one as big or bigger than the 20,000 people it thinks of as crazily huge.

No, it’s simpler: Google doesn’t have an army of support people because it can make more money by having poor customer service. It has a monopoly on search, and if you want your website listed there, you’ll play their game no matter how difficult it is. It’s just not under any meaningful pressure to improve in this area. So instead of employing a bunch of people in call centers to answer your questions and fix mistakes, it pockets more money. Corporations are greedy—maximizing profits is their job—but we shouldn’t be giving them a free pass on this sort of thing just because they’ve complained it’s expensive. Making billions of dollars is supposed to be hard work. We should all be demanding better customer support from Google, and maybe we should be taking a good hard look at their competitive practices in this area.


Introducing Spandex

Sep 29, 2011

So you’re a programmer, and you want to start a blog. You’ve got some pretty cool opinions about software construction and that wine from dinner has you thinking the entire world has just got to know all about them. And maybe some neat posts about the aardvarks you’ve seen lately. Sitting down with your digestif, you get to work. You go the standard route: you find one of the dozen crummy WordPress hosts, pick a theme, and start typing your brilliance into a form in the admin section of your site. And it sorta works.

But then you start thinking about it:

  • Why can’t you use your favorite editor? I mean, you spent all this time customizing the crap out of your Emacs experience, and now you’re typing tons of content into a form in your browser?

  • Man do these WISYWYG editors blow. Wouldn’t you rather be using a nice markup language, like maybe something similar to what StackOverflow uses?

  • WP’s admin interface is your content management system. But you’re already very proficient with a much fancier, more powerful CMS: your source control system. You can use it from the command line instead of a clunky interface, and you can do a lot more with it. Why can’t you use that?

  • Your posts live in a database somewhere that you don’t really have access to.

  • No, but really, did I mention your WP host totally sucks ass? I guess you could install WP on an EC2 instance, but that seems like overkill, right? Maybe you could host it on Heroku?

A different model

A better idea might be a site that reads your posts from files (like, on the filesystem!) and generates web pages from them. Now you can manage those files in Git and edit them however you want. And if they’re built on a cool stack, you’ll be able to host it somewhere free and nice. So, you look around, and there are some pretty cool options:

  • Jekyll: Generate static HTML pages from Textile or Markdown files and then host them wherever you want.

  • Nesta: Dynamically generate a site from markup files. Themes, plugins, and all that jazz.

  • Toto: Another dynamic site that generates the blog from static Markdown files.

I’m sure there are others. I lean more towards the Nesta/Toto model of generating the HTML dynamically. That narrows my hosting options (because my host will have to support whatever web framework the engine runs on), but means there’s less for me to actually do (I don’t have to generate the site before pushing it to the host). But whatevs.

What’s a blog, anyway?

But you’re feeling a bit greedy and you want to do even more with less. So you think, what, in a technical sense, is a blog? It’s two things: 1) a simple content management system that keeps track of your posts, figures out which ones to show your user, turns them into HTML, and generates an RSS feed; and 2) a web application to actually show those posts in. A blog engine tackles both things.

But here’s the thing: you’re good at building websites. You want to build your own website, where you control what the URLs mean, and how the framing markup gets generated, and all that jazz. Maybe you want to integrate the blog into an existing site. Maybe you want customize the framing HTML (you know the chrome stuff like the title and footer and sidebars), but you really hate Haml for some reason and Nesta uses it for just that purpose. And you don’t really want to think about “themes”; you want to have some HTML and write some CSS for it, and that’s that. The world has built you a bunch of awesome web frameworks to build sites with, and dammit you want to use them. It’s just the CMS part that’s annoying; there has to be some code somewhere that pulls the right files off of disk and formats them when the user, say, clicks on the links for posts tagged “aardvarks”. (You love aardvarks, it turns out. No one is sure why.)

I’m going to assume, for no real reason, that you’re the kind of programmer who likes Ruby. It’s a good thing you are, because I have some good news: I wrote you the piece that does all the content management. It’s called Spandex.


OK, so first of all, what kind of abject moron names a programming library “spandex”? When you hear it, all you can think about is this:

Yeah, I don’t know. The gem name wasn’t taken. Anyway, Spandex lets you do stuff like:

some_posts = spandex.find_articles(:tag => 'aardvarks')
some_posts.each do |post|
  puts "Title: #{post.title}"
  puts "Stuff about aardvarks: #{post.body}"

How do you make these awesome posts about aardvarks? Here’s

Title: Oh my God I saw some aardvarks!
Tags: aardvarks, things i saw today
Date: 2011/9/2011

I was just riding my bike down the street and **LO AND BEHOLD**, an aardvark!!!

Yeah, your aardvark obsession is a bit creepy; let’s more on. The top part of the file consists of key-value pairs that provide metadata to Spandex. Stuff like whether it’s in draft mode. The bottom is the content. You can write it in any markup language supported by Tilt; it’ll decide which to use based on the extension. And you can customize that through Tilt itself.

There are a lot more details, examples, and awkward non sequiturs on Spandex’s Github page.

Sounds easy, right? So finish your glass of port, get a sixpack of beer, and let’s build a blog.

Build ye blog

You’re going to want to start simple. A blog with some posts, an about page, something allows all of your avid fans to comment, and an atom feed.

Let’s get to work. We’ve got our spandex on, so now we just need to build a website. Let’s use Sinatra, Haml (which I guess you don’t hate after all), and the SCSS variety of Sass. OK, here’s our directory structure:

   +- app.rb              # Your Sinatra app
   +-           # Config file for making Sinatra sing. Get it?
   +-content/             # Bucket of your content
   | |
   | +-   # Your first post!
   | |
   | +-          # Some stuff about you
   +- views/              # HTML templates for the chrome
      +- index.haml       # Shows all of the posts in descending order
      +- layout.haml      # Title bars and such
      +- page.haml        # Display and individual post and its comments

So, app.rb handles the web requests:

require 'sinatra'
require 'spandex'
require 'haml'

content_dir = File.expand_path('content', File.dirname(__FILE__))
@@spandex =

#render the index
get '/' do
  posts = @@spandex.all_articles
  haml :index, :locals => {:posts => posts}

# render the atom feed
get '/feed.xml' do
  content_type :xml, :charset => 'utf-8'
  @@spandex.atom_feed(10, 'Tiny Blog of Doom', 'Your name here', '', '/feed.xml')

# render stylesheets
get '/css/:sheet.css' do
  content_type 'text/css', :charset => 'utf-8'
  scss params[:sheet].to_sym

#render individual posts
get '*' do
  post = @@spandex.get(params[:splat][0])
  haml :page, :locals => {:post =>  post, :show_comments => true}

The Haml files are also pretty simple. I won’t cover all of the details, but here’s page.haml:

  %a{:href => post.path}= post.title
%div= post.body
- if show_comments && ENV["DISQUS_NAME"]
  - if Sinatra::Application.environment == :development
    %script var disqus_developer = true;
  %script{:type => 'text/javascript', :src => "http://#{ENV['DISQUS_NAME']}", :async => true}

That shows a single post and pulls in comments from Disqus.

You can run the whole thing with shotgun or you can just check it into Git and push the whole thing to Heroku. The source for the entire tiny blog is here. Check it out! DO IT NOW.

You can also check out the code to this blog, which naturally uses Spandex, and does some other neato things.