One of the best speakers at the NY Web 2.0 Expo was Josh Schachter of delicious, who spoke on Lessons Learned in Scaling and Building Social Systems:
BACKGROUND
Built delicious in 2003, sold to Yahoo in 2005. I just left Yahoo a few months ago.
Billions of page views/month.
4m users at time of sale
3 kinds of scale: technological, personal, and software.
Technological scale
– partition users into multiple sets (sharding, clustering). I built delicious into one big database.
– caching. Avoid going into the database.
– replicas. Must have multiple copies of the data.
– mysql issue: we got a 60x performance speedup by making some changes
– autoincrement: will hurt you later. Dont do this.
– put proxy in front. Dont let the dialup user take up your whole server
– sloppiness. Use an offline process to decouple interactive processes from the rest of the system.
Social Scale
– different features at different scales
– at beginning, make it easy for users to find one another. Dont let them say ‘hi’ and see no one respond. Minimize barriers to entry and minimize transaction costs.
– as you grow, you have to design features to mitigate traffic. E.g., on delicious, the ability to follow your friends.
3 reasons a social app has value:
– utility, network effect, revenue
– Delicious initially focused on utility.
– have to provide value to the user *before* there are a lot of users online
– you have to provide these motivations in this order: 1) provide utility, 2) get a network effect, 3) monetize
For a long time, biggest apps on Facebook were Superpoke, etc. They have very little functionality, but a lot of users, because they are simple enough to spread rapidly.
App must be self-marketing without requiring sign-in.
Compare initial marketing vs. actual functionality. Initial marketing of delicious was: bookmarks for people with multiple PCs. But really, we became a search engine for bookmarks.
Half our traffic came from RSS.
Figure out drivers for infection. Firefox Extension was our biggest driver.
Choice of language matters: key to say ‘do not share’ instead of ‘private’, to discourage not sharing. ‘Not sharing’ sounds like something your mom would criticize you for.
Kids change their social network partly because it allows them to update the network to reflect their current status
Delicious doesn’t have a chat feature because I didn’t want the flame wars you see on wikipedia admin pages
Must prepare to deal with spam and abuse
Lengthen or destroy feedback loops.
If you kick off a spammer, you’ve taught them what they did wrong. But they’ll be back. So on delicious we let them use the system but didn’t let anyone see them.
When xdrive had problems with spammers, they just inserted a 20sec delay on downloading files. That frustrated them and they moved to a different system.
Make pretty URL’s; they’re easier to remember & forward.
We were one of the first sites with a public API. I used this as a tool to recruit people who might have built competitors otherwise
Scaling yourself. The first thing you do will be wrong. Iterate quickly.
Write down all your ideas. It’s good for future patent work.
If people in engineering have to read support tickets they’ll build around problems
Figure out user motivations
put the whole staff in the listening session. Measure and record everything.
Average user has 60 tags.
8% of users will use 12345 as password
Another few % will use 123456
And about 15% will use your domain plus their ID
His best observation: SMS is the next command line. The next 1b people online will only have SMS.
