Double Shot #854

Had a fun weekend digging into the asset pipeline.

  • lograge - Minimal logging for Rails applications. Personally I like the verbose log but ymmv.
  • The future of MacRuby - Looks fairly dismal to me, actually. But they're asking people to get involved.
  • StackTrace - iPad application to put a nice face on Stack Overflow. I've played with this and it's pretty cool.
  • The Smallest Rails App - And it even puts its own source code out to the screen.
  • CoffeeConsole: A Chrome Extension - This one adds a new panel inside the Web Inspector to compile and run CoffeeScript in the context of the current window.
  • Dillinger - "Cloud-enabled HTML5 Markdown editor."

Notes on the Asset Pipeline

I've come to the conclusion that it's time to understand what's going on with the Rails 3.2 asset pipeline - primarily because I'm trying to upgrade a Rails 3.0 application and so far the process has been a complete disaster for me. So, I'm going back to first principles. Perhaps someone else will find these notes of use as well.

I started out by building up a Rails 3.2.3 application from scratch:


cd scratch
rvm use 1.9.3-p125
rvm gemset create ap_test
rvm gemset use ap_test
gem install rails
rails new ap_test
cd ap_test/
rails generate scaffold animal name:string legs:integer color:string
rake db:migrate

Remove public/index.html and it's off to the races - a tiny working Rails application. Run the rails server, browse to localhost:3000, and it's there. Looking at the page source shows a bunch of assets:


<link href="/assets/application.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/animals.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/scaffolds.css?body=1" media="all" rel="stylesheet" type="text/css" />
<script src="/assets/jquery.js?body=1" type="text/javascript"></script>
<script src="/assets/jquery_ujs.js?body=1" type="text/javascript"></script>
<script src="/assets/animals.js?body=1" type="text/javascript"></script>
<script src="/assets/application.js?body=1" type="text/javascript"></script>

This leads to a couple of immediate questions: where did these files come from and how did they get there? The css files come from files in app/assets/stylesheets:

  • animals.css.scss
  • application.css
  • scaffolds.css.scss

The two .scss files were generated by the scaffold generator; application.css is the CSS manifest provided originally by Rails.

For the JavaScript, things are more complicated. Two of the source files are in app/assets/javascripts:

  • animals.js.coffee
  • application.js

The CoffeeScript file was generated by the scaffold generator, and the JavaScript manifest came from running the rails command. But what about the two jQuery files? They're definitely being served (I can retrieve http://localhost:3000/assets/jquery.js in the browser) but I have no idea where they're coming from. A little poking around shows that they are in the jquery-rails gem, where the files live in vendor/assets/javascripts. This gem is actually a Rails engine, and a look at rails/railties/lib/rails/engine.rb in the Rails source shows that the vendor/assets, lib/assets and app/assets directories inside an engine are added to config.assets.paths search path when the engine is initialized.

Turning from file locations to process, this is what shows up on the first page access in the Rails log:


Started GET "/" for 127.0.0.1 at 2012-04-08 08:00:14 -0500
Processing by AnimalsController#index as HTML
  Animal Load (0.1ms)  SELECT "animals".* FROM "animals"
  Rendered animals/index.html.erb within layouts/application (1.3ms)
Compiled animals.css  (35ms)  (pid 91415)
Compiled scaffolds.css  (34ms)  (pid 91415)
Compiled application.css  (305ms)  (pid 91415)
Compiled jquery.js  (4ms)  (pid 91415)
Compiled jquery_ujs.js  (2ms)  (pid 91415)
Compiled animals.js  (502ms)  (pid 91415)
Compiled application.js  (670ms)  (pid 91415)
Completed 200 OK in 1194ms (Views: 1189.7ms | ActiveRecord: 1.0ms)

So it's apparent that there was a transformation from the sources to the actual served version when I loaded the page. A little poking around shows that the compilation step only happens when necessary - loading new pages doesn't trigger it, but changing a file does. This is in development mode, of course.

Time to look at configuration settings. These are scattered across several files. First, in config/application.rb:


# Enable the asset pipeline
config.assets.enabled = true

# Version of your assets, change this if you want to expire all your assets
config.assets.version = '1.0'

And in config/environments/development.rb:


# Do not compress assets
config.assets.compress = false

# Expands the lines which load the assets
config.assets.debug = true

For completeness, the corresponding section in config/environments/production.rb:


# Disable Rails's static asset server (Apache or nginx will already do this)
config.serve_static_assets = false

# Compress JavaScripts and CSS
config.assets.compress = true

# Don't fallback to assets pipeline if a precompiled asset is missed
config.assets.compile = false

# Generate digests for assets URLs
config.assets.digest = true

So it looks like the default for config.assets.compile is true, which would explain why the server compiled things on demand when I ran the page. Finally, there is a section in the Gemfile that defines the asset pipeline itself:


# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails',   '~> 3.2.3'
  gem 'coffee-rails', '~> 3.2.1'

  # See https://github.com/sstephenson/execjs#readme for more supported runtimes
  # gem 'therubyracer', :platform => :ruby

  gem 'uglifier', '>= 1.0.3'
end

So there's a gem that understands SASS/SCSS, one that understands CoffeeScript, and one that I suspect is concerned with compression.

As to what actually does the serving - that's the sprockets gem. There's a comment in the sprockets README file: "Under Rails 3.1 and later, your Sprockets environment is automatically mounted at /assets." Xavier Noria kindly added a comment to this post with some details on how this mounting is managed. In development, then, assets are handled something like this:

  1. Somehow Rails is calling sprockets to recompile any changed assets when a page request comes in. Sprockets is the source of the "Compiled animals.js (502ms) (pid 91415)" lines in the Rails log.
  2. A request comes in for a file somewhere in /assets
  3. This request gets routed to sprockets
  4. Sprockets uses its asset paths array to find the first matching file and serves it

The manifest files come into things as a shorthand way to tell Rails which assets to include with a particular request. The default layout only specifies one stylesheet and one javascript file:


<%= stylesheet_link_tag    "application", :media => "all" %>
<%= javascript_include_tag "application" %>

Note that we no longer include the :cache settings on these declarations - that's now done within the asset pipeline bits.

The two application files are no longer the default stylesheet and javascript files, but manifests that contain instructions to sprockets about what to include in the compiled files. You can still include css and javascript in these files, but I suspect that's a bad practice at this point; better to separate out the actual code to different files. Here's the defaultapplication.css file:


/*
 * This is a manifest file that'll be compiled into application.css, which will include all the files
 * listed below.
 *
 * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
 * or vendor/assets/stylesheets of plugins, if any, can be referenced here using a relative path.
 *
 * You're free to add application-wide styles to this file and they'll appear at the top of the
 * compiled file, but it's generally better to create a new file per style scope.
 *
 *= require_self
 *= require_tree .
*/

The lines starting with *= are directives to sprockets. These two say to include the current file, and to include every file in a tree starting in the current file's folder, i.e. the entire app/stylesheets hierarchy.

Here's the default application.js file:


// This is a manifest file that'll be compiled into application.js, which will include all the files
// listed below.
//
// Any JavaScript/Coffee file within this directory, lib/assets/javascripts, vendor/assets/javascripts,
// or vendor/assets/javascripts of plugins, if any, can be referenced here using a relative path.
//
// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
// the compiled file.
//
// WARNING: THE FIRST BLANK LINE MARKS THE END OF WHAT'S TO BE PROCESSED, ANY BLANK LINE SHOULD
// GO AFTER THE REQUIRES BELOW.
//
//= require jquery
//= require jquery_ujs
//= require_tree .

This file is explicitly including the jQuery bits, which are found in the jquery-rails gem because it added itself to the asset search paths when it was loaded as an engine. It then includes every file in a tree starting in the current file's folder, i.e. the entire app/javascripts hierarchy.

As we've seen, in development mode sprockets and the include tags include these files by bringing them in individually. In production mode things are a bit different. Just running the server in production mode and hitting the root will result in an error in the log:


Started GET "/" for 127.0.0.1 at 2012-04-08 09:45:05 -0500
Processing by AnimalsController#index as HTML
  Rendered animals/index.html.erb within layouts/application (3.4ms)
Completed 500 Internal Server Error in 77ms

ActionView::Template::Error (application.css isn't precompiled):

That's because of the config.assets.compile = false setting in production.rb. In production mode, Rails won't call sprockets to compile assets on the fly. Instead, you need to set them up manually by running rake assets:precompile first. This task puts copies of the compiled assets in public/assets. In production mode, the page source changes as well:


<link href="/assets/application-958f49f3752dcbeab5370a3aee0bec08.css" media="all" rel="stylesheet" type="text/css" />
<script src="/assets/application-130530c48b043db45fa6a7a03b179580.js" type="text/javascript"></script>

Note that sprockets has concatenated all the applicable files together and gone to cache-buster style naming. If you want to test this out using WEBrick, you'll need to set config.serve_static_assets = true in production.rb, otherwise Rails won't try to serve files out of public/assets

.

What about controller-specific assets? Let's generate a second scaffolded resource:


rails generate scaffold bird name:string color:string
rake db:migrate

This adds a bunch of files to the application, include app/assets/javascripts/bird.js.coffee and app/assets/stylesheets/birds.css.scss. Loading a page now loads both sets of assets:


<link href="/assets/application.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/animals.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/birds.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/scaffolds.css?body=1" media="all" rel="stylesheet" type="text/css" />
<script src="/assets/jquery.js?body=1" type="text/javascript"></script>
<script src="/assets/jquery_ujs.js?body=1" type="text/javascript"></script>
<script src="/assets/animals.js?body=1" type="text/javascript"></script>
<script src="/assets/birds.js?body=1" type="text/javascript"></script>
<script src="/assets/application.js?body=1" type="text/javascript"></script>

And precompiling only produces one set of application-level js and css files as well. This makes sense based on what we've seen in the manifest files, but it contradicts what you might guess from the comments in the controller-specific files. It turns out that by default the organization of JavaScript and CSS into controller-specific files is strictly to help you keep track of things; all of the code in all of these files is available on every page. To actually load things on a controller-by-controller basis, you'd need to modify the manifest files and the layouts. I've placed an example of how to do this on a separate gist: https://gist.github.com/2338096.

After working through the process to this point, I think I have a better idea what's going on. I still need to figure out a few things, chiefly: What's the deal with rake assets:precompile:primary and digested assets?

Hopefully this helped sort things out for other people as well. Additions and corrections are more than welcome.

Double Shot #853

End of the week? Hooray!

Double Shot #852

Sometimes being responsibility-driven is tough.

  • CanJS - Another client-side JavaScript framework designed to play well with a variety of UI libraries.
  • Heroku Add-ons Catalog - Listing and review site for all the things you can bolt on to your Heroku projects.
  • Pelusa - Static analysis a la lint for ruby.

Double Shot #851

Enjoying a blast of high-intensity light.

Double Shot #850

New goal: Inbox-don't-get-clients-so-angry-they-fire-me.

  • Startups, this is how design works - An introduction to "design" for developers. I'm somewhat skeptical of current design trends, but there are some useful ideas and resources here.
  • Novocaine - Open-source high-performance audio wrapper for iOS and OS X.
  • Figaro - Simple Rails application configuration using ENV variables.
  • TorqueBox 2.0 - New major release of this application platform that builds a Ruby and Rails environment on top of JBoss.

What's New in Edge Rails #15

Week of March 25 - March 31, 2012

Much of the work in edge Rails at the moment is concerned with cleaning up the code and making the Ruby better for the future. But there are still some new features coming down the pike.

  • 3e67e45dc327631e085cc67aaa6522b44324364c tweaks resource routing to be a bit more self-evident in the controllers through letting you define a custom parameter name to replace :id.
  • 4882271 cleans up Rails.initialized? to be simpler and more reliable.
  • The recent change to remote forms and authenticity tokens that was part of Rails 3.2.3 is also in master as 128cfbdf.
  • Active Record gets some new finders in 13b3c77e:
    • Post.find_by name: 'Spartacus', rating: 4
    • Post.find_by "published_at < ?", 2.weeks.ago
    • Post.find_by! name: 'Spartacus'

Double Shot #849

All worn out from weekend Scouting events. And wishing it was still the weekend, too.

Double Shot #848

Ending the week with a bang.

Double Shot #847

Took a couple of days off there for travel and then to be sick as a dog. Sorry about that.

What's New in Edge Rails #14

Week of March 18 - March 25, 2012

Thanks to Ryan Daigle, this content now lives at EdgeRails.info. If you have any ideas on other content that should be there, please drop me a line.

That said, it was a very quiet week in Rails master (or else I missed everything of significance. But here's the one commit that caught my eye.

  • Eager-evaluated scopes are deprecated as of 0a12a5f8, in the hopes of avoiding some common and hard-to-find errors when using scopes.

Double Shot #846

Posting from California. Imagine you can feel the desert heat.

  • TypeButter - Kerning plugin for jQuery, just in case you want to twiddle the fonts on your website.
  • Polishing Rubies (Part 3): Tools for Testing - Michael Bleigh continues his series on contributing to open-source software.
  • Disconnect - Tools to stop sites from tracking you around the web.
  • Factory Girl Hits 3.0 - A big step forward that breaks the ties with Rails 2 and Ruby 1.8.
  • Cinch 2.0 - "The IRC Bot Building Framework" focused on a clean Ruby API and defining things in terms of plugins and rules.

Double Shot #845

Had a fine day in the woods yesterday, so not too many links today.

Double Shot #844

Today it's off to the woods to play in the mud for the Boy Scouts.

  • Continuous Cache Warming for Rails - If I said I'd never indulged in this sort of hackery I'd be lying.
  • strong_parameters - Official Rails plugin to support taint and required checking in Action Pack with enforcement in Active Model. But why are they bringing out new plugins when plugins are deprecated now?
  • ZeroNinetyNine - 99-cent deals on Mac App Store apps, valid for the next 18 hours as I write this. At this price at least LiveReload is worth picking up.
  • Home Internet with Anonymity Built In - Looks like you might be able to pick up a wireless router with Tor preinstalled soon enough.

Double Shot #843

I'd feel more heroic about staying up till midnight fixing a bug if the bug were not my fault.

Double Shot #842

Packing bits for a trip takes at least as much time as packing atoms.

  • Punch - The latest offering for building brochure sites, this one using Mustache templates with JSON.
  • Hosted NoSQL - A rundown of some of the available commercial offerings.
  • Two New AWS Getting Started Guides - One each for Linux and Windows showing how the load balancing and scaling pieces fit together.
  • Fubar - New error-tracking and logging service taking beta signups now.
  • Mobile HTML5 - Big feature tables for a bunch of different mobile browsers.
  • Learning Puppet VM - Pre-configured virtual machine with Puppet Enterprise.
  • rbfu - Simple ruby version management, an alternative to rvm or rbenv.
  • Git project seeks discussion on "push" change - If you're a git power user this is worth a read.

What's New in Edge Rails #13

Week of March 11-March 17, 2012

The big news this week: Active Resource has been removed from Rails, and "API only" Rails applications put in a brief appearance before going back to the drawing board.

  • Text fields and text areas no longer have default size, rows, or cols as of 3384ee24. It's 2012, if you want a default set it in your CSS.
  • eee32af45 adds a dynamic find_or_create_by_{attribute}! method to Active Record.
  • With f1637bf2 Active Resource has been removed from Rails. It will be available as a separate gem if anyone wants to maintain it, though I suspect most people who need that sort of functionality are using one of the alternatives already.
  • c1f397f88 merges a chunk of work to remove sprockets integration from Action Pack and depend on the sprockets-rails gem instead.
  • dde3058c7 removes the Active Record IdentityMap feature, which never worked right anyhow (see 302c912b for some details of the problems).
  • A bunch of commits starting at 4c16791f added "API only" Rails applications. All of them were reverted in 6db930cb because the feature is still half-baked.

Double Shot #841

Open Source Report #7

Confession time: my string of daily contributions to open source crashed and burned shortly after the last Open Source Report. Things heated up at two of my paying clients, to the point where I've been stumbling out of bed, having breakfast, and diving right into paid code. Time to give back to the community simply vanished.

So in lieu of reporting on my own accomplishments, a couple of potentially useful links to get you thinking about contributing back yourself:

With those links out of the way, what's next for me? Pretty simple: it's time to climb back on the horse that threw me. There's some code I want to tweak this morning, in fact. So perhaps next week I'll have more positive actions to report.

Double Shot #840

This week has been long enough. I'd be happy to donate the remaining hours to someone else.

subscribe via RSS