RailsConf Day 3 Report
The day started out with the world premiere of the last of the Rails vs. PHP videos from the RailsEnvy guys. This was warmly received, as you might expect.
From there, we rolled right into Jamis Buck and Michael Koziarski from the Rails core team discussing The Rails Way . This was one of the best-loved keynotes of the whole conference, because they threw a bunch of code up on the screen; people were definitely hungry for more code in the presentations (a piece of feedback which I am sure the conference organizers have heard by now). A few of the points they raised:
They held a Q&A session at the end of the keynote via IRC. This was only moderately successful; as with the other conference IRC channels, there was a lot of pointless chatter going on. (I dipped into the main IRC channel a few times, and found it pretty much impossible to extract any value from it).
For my first session of the day, I elected to watch Britt Selvitelle and Alex Payne talk about "Scaling Twitter". This opened with Alex explaining the now-infamous interview; he maintains that he did not say that scaling on Rails is hard. Twitter was just a tiny project until SxSW and the follow-up media interest, at which point it took off.
Scaling has so far consisted mainly of throwing more boxes at the thing; they still just use a single database (with master-slave replication to a separate reporting database). Currently, Twitter is spread across 32 cores: 19 cores for Mongrel (120ish Mongrel instances), message processing and delivery across 16 cores, jabber across 2 cores, and MySQL on one big 8 core box. They're using 16GB or so for memcache.
Other Twitter statistics:
Caching is their biggest optimization. Memcache is their friend, and they do a bunch of event-driven cache invalidation (if Bob is a friend of Alice and Bob twitters, then Alice's friends timeline is invalid). They also denormalize the database as necessary. Another big help is Starling, a pure Ruby distributed queuing system that they wrote (and will probably open-source in the future).
Contrary to some reports on the Web, they've looked at ETags and it wouldn't be a huge optimization for them - they've actually had valid ETags in the past. HTTP delivery is not their big issue.
Another big help in scaling: the Rails community. "Act like it's a business plan. Get a board of advisors for your scaling plan, is our advice."
From there, I wandered over to hear Amy Hoy's talk "Rubber, Meet Road" about how developers and designers can work together. This was fun because Amy is an entertaining speaker, though I'm not sure how much I got out of it - I don't so much understand this fear that many developers seem to have of designers. A lot of the talk boiled down to reminding everyone that both fields are perfectly valid, and things will move along much better if we all collaborate instead of fighting. But people seemed to get a lot out of it, which was cool.
Amy did tie Rails back into the talk by looking at how you can get designers (with an HTML/CSS background) involved in actually working with Rails applications. As such things go, Rails is fairly friendly (though "all templating languages make things worse"). Developers can help by making it friendlier:
I found another table full of random folks to have lunch with (I tried to mix as much as possible during meals; networking is a large part of why I paid for this conference), and we had a wide-ranging discussion of everything from how to retread from Access to Rails, to search plugins, to open source. One of the topics that came up was what Microsoft is doing with Ruby on the DLR. Now, Scott Hanselman may have found interest in Microsoft spinning off the DLR code and developers to a separate non-profit organization, but all I can say is that he was talking to a different set of conference attendees than I was (I suspect Scott spent more time with the ThoughtWorkers and other enterprisey folks). I raised the question of the DLR in half a dozen groups, and the opinion was pretty well unanimous: no one trusted Microsoft in the least to keep their Ruby implementation pure, no one thought that it would be a safe component to depend on, and no one would even consider using it in a future application. The spectre of "embrace and extend (and extinguish)" still lives on.
I skipped one session and hung out and caught up on e-mail for a bit, and then caught the closing keynote with Dave Thomas. Dave started out with a satirical screenshot of rubymiddleagedwhiteguys.com (there had been some trouble with rubychicks.com during the conference) and went on to make the pitch that "I would like Rails to set the standard for the integration of people into the community." I don't have the energy for a long analysis of this stuff right now, but it's good to see that the Rails community is at least at the point where community leaders are recognizing that diversity and openness are useful core values, although there are some community dynamics that are still keeping the Rails community overwhelmingly white, male, young, and closed.
The next topic was the charitable donations drive that was spearheaded by the Pragmatic Programmers. As of the start of the keynote, about $26,000 had come in - which wasn't really an astounding amount for 1600 people, but not bad for 1600 people many of whom, I suspect, had never donated to anything. Dave suggested to other organizers that they keep the beg-a-thon up at future conferences: "Why don't we make this the standard for all technical conferences to show that the industry cares?"
The meat of the keynote started off with a discussion of the exponential growth of Rails and the number of Rails developers. Dave argues that we need to start putting in place mechanisms to allow people to learn better and to have conversations: better documentations, more blogs, help sites, who knows what else. The key question is how are we going to help new people join us?
Also, as the effort involved in launching a web application drops, more and more web applications become possible, and more and more people can launch them. We need to welcome in people who are not hardcore programmers. But this also means we get a whole new pool of people to play with: designers, writers, people with cool new ideas whoever they may be. We need to try to be an inclusive crowd, not an insular and geeky crowd.
From here, Dave moved into a discussion of cargo cult programming. He made a pitch that we're all doing it right now - broadly defined as doing things just because we've seen them work before. "There is no difference between the browser and the 3270 except that the browser can display better porn" (both are basically half-duplex devices). He urged the crowd to think beyond their habitual ways of doing things and to experiment.
Next rant: object-oriented programming. It's called object-oriented not *class-oriented" but we spend all of our time modeling the world with hierarchies of classes. But the world is chaotic, there are almost no inheritance hierarchies in nature. He sees the way we're using objects and classes as another cargo cult.
The bottom line: Don't just receive knowledge. (Of course, this is what all prophets say when they tire of being prophets. It rarely works. It's hard to get out of the business of being a prophet).
Dave ended, as one would expect of a closing keynote, with a challenge: "Go away from here, have fun, and do some good."
The conference wrapped with a closing reception with food and more chitchat. I ended up in a discussion of the Rails community (I suspect our little group was not the only one on that topic). My bet is that this RailsConf will be the last time that the majority of the Rails community ever gets together in one place - by next year, there will be too many developers working in this space for a single conference to accommodate them. So I suspect we're inevitably going to see fragmentation, not the sort of inclusivity that Dave Thomas was hoping for. Time will tell, though.
From there, we rolled right into Jamis Buck and Michael Koziarski from the Rails core team discussing The Rails Way . This was one of the best-loved keynotes of the whole conference, because they threw a bunch of code up on the screen; people were definitely hungry for more code in the presentations (a piece of feedback which I am sure the conference organizers have heard by now). A few of the points they raised:
- A model does not need to equate to an actual database table
- DRY is useful "but if the fix is worse than the original disease, don't do it!"
- After you declare associations in your models, you should actually use them
- Take advantage of the helper methods in ActiveSupport to make your code's intent clear
- Don't cargo cult.
They held a Q&A session at the end of the keynote via IRC. This was only moderately successful; as with the other conference IRC channels, there was a lot of pointless chatter going on. (I dipped into the main IRC channel a few times, and found it pretty much impossible to extract any value from it).
For my first session of the day, I elected to watch Britt Selvitelle and Alex Payne talk about "Scaling Twitter". This opened with Alex explaining the now-infamous interview; he maintains that he did not say that scaling on Rails is hard. Twitter was just a tiny project until SxSW and the follow-up media interest, at which point it took off.
Scaling has so far consisted mainly of throwing more boxes at the thing; they still just use a single database (with master-slave replication to a separate reporting database). Currently, Twitter is spread across 32 cores: 19 cores for Mongrel (120ish Mongrel instances), message processing and delivery across 16 cores, jabber across 2 cores, and MySQL on one big 8 core box. They're using 16GB or so for memcache.
Other Twitter statistics:
- 200-300 connections per second
- Spiking 800 connections per second
- Has been as high as 11,000 connections per second
- MySQL has spiked to 2400 queries per second
- API traffic is 20x web traffic, so Alexa underestimates by at least this factor
Caching is their biggest optimization. Memcache is their friend, and they do a bunch of event-driven cache invalidation (if Bob is a friend of Alice and Bob twitters, then Alice's friends timeline is invalid). They also denormalize the database as necessary. Another big help is Starling, a pure Ruby distributed queuing system that they wrote (and will probably open-source in the future).
Contrary to some reports on the Web, they've looked at ETags and it wouldn't be a huge optimization for them - they've actually had valid ETags in the past. HTTP delivery is not their big issue.
Another big help in scaling: the Rails community. "Act like it's a business plan. Get a board of advisors for your scaling plan, is our advice."
From there, I wandered over to hear Amy Hoy's talk "Rubber, Meet Road" about how developers and designers can work together. This was fun because Amy is an entertaining speaker, though I'm not sure how much I got out of it - I don't so much understand this fear that many developers seem to have of designers. A lot of the talk boiled down to reminding everyone that both fields are perfectly valid, and things will move along much better if we all collaborate instead of fighting. But people seemed to get a lot out of it, which was cool.
Amy did tie Rails back into the talk by looking at how you can get designers (with an HTML/CSS background) involved in actually working with Rails applications. As such things go, Rails is fairly friendly (though "all templating languages make things worse"). Developers can help by making it friendlier:
- Write the Ruby code for the designers
- Create cheat sheets explaining the key uses of ActiveRecord and ERb in your own project
- Write helper methods to hide the much
- Lots of comments
- And don't forget to teach them about Subversion
I found another table full of random folks to have lunch with (I tried to mix as much as possible during meals; networking is a large part of why I paid for this conference), and we had a wide-ranging discussion of everything from how to retread from Access to Rails, to search plugins, to open source. One of the topics that came up was what Microsoft is doing with Ruby on the DLR. Now, Scott Hanselman may have found interest in Microsoft spinning off the DLR code and developers to a separate non-profit organization, but all I can say is that he was talking to a different set of conference attendees than I was (I suspect Scott spent more time with the ThoughtWorkers and other enterprisey folks). I raised the question of the DLR in half a dozen groups, and the opinion was pretty well unanimous: no one trusted Microsoft in the least to keep their Ruby implementation pure, no one thought that it would be a safe component to depend on, and no one would even consider using it in a future application. The spectre of "embrace and extend (and extinguish)" still lives on.
I skipped one session and hung out and caught up on e-mail for a bit, and then caught the closing keynote with Dave Thomas. Dave started out with a satirical screenshot of rubymiddleagedwhiteguys.com (there had been some trouble with rubychicks.com during the conference) and went on to make the pitch that "I would like Rails to set the standard for the integration of people into the community." I don't have the energy for a long analysis of this stuff right now, but it's good to see that the Rails community is at least at the point where community leaders are recognizing that diversity and openness are useful core values, although there are some community dynamics that are still keeping the Rails community overwhelmingly white, male, young, and closed.
The next topic was the charitable donations drive that was spearheaded by the Pragmatic Programmers. As of the start of the keynote, about $26,000 had come in - which wasn't really an astounding amount for 1600 people, but not bad for 1600 people many of whom, I suspect, had never donated to anything. Dave suggested to other organizers that they keep the beg-a-thon up at future conferences: "Why don't we make this the standard for all technical conferences to show that the industry cares?"
The meat of the keynote started off with a discussion of the exponential growth of Rails and the number of Rails developers. Dave argues that we need to start putting in place mechanisms to allow people to learn better and to have conversations: better documentations, more blogs, help sites, who knows what else. The key question is how are we going to help new people join us?
Also, as the effort involved in launching a web application drops, more and more web applications become possible, and more and more people can launch them. We need to welcome in people who are not hardcore programmers. But this also means we get a whole new pool of people to play with: designers, writers, people with cool new ideas whoever they may be. We need to try to be an inclusive crowd, not an insular and geeky crowd.
From here, Dave moved into a discussion of cargo cult programming. He made a pitch that we're all doing it right now - broadly defined as doing things just because we've seen them work before. "There is no difference between the browser and the 3270 except that the browser can display better porn" (both are basically half-duplex devices). He urged the crowd to think beyond their habitual ways of doing things and to experiment.
Next rant: object-oriented programming. It's called object-oriented not *class-oriented" but we spend all of our time modeling the world with hierarchies of classes. But the world is chaotic, there are almost no inheritance hierarchies in nature. He sees the way we're using objects and classes as another cargo cult.
The bottom line: Don't just receive knowledge. (Of course, this is what all prophets say when they tire of being prophets. It rarely works. It's hard to get out of the business of being a prophet).
Dave ended, as one would expect of a closing keynote, with a challenge: "Go away from here, have fun, and do some good."
The conference wrapped with a closing reception with food and more chitchat. I ended up in a discussion of the Rails community (I suspect our little group was not the only one on that topic). My bet is that this RailsConf will be the last time that the majority of the Rails community ever gets together in one place - by next year, there will be too many developers working in this space for a single conference to accommodate them. So I suspect we're inevitably going to see fragmentation, not the sort of inclusivity that Dave Thomas was hoping for. Time will tell, though.