I've come to the conclusion that it's time to understand what's going on with the Rails 3.2 asset pipeline - primarily because I'm trying to upgrade a Rails 3.0 application and so far the process has been a complete disaster for me. So, I'm going back to first principles. Perhaps someone else will find these notes of use as well.

I started out by building up a Rails 3.2.3 application from scratch:


cd scratch
rvm use 1.9.3-p125
rvm gemset create ap_test
rvm gemset use ap_test
gem install rails
rails new ap_test
cd ap_test/
rails generate scaffold animal name:string legs:integer color:string
rake db:migrate

Remove public/index.html and it's off to the races - a tiny working Rails application. Run the rails server, browse to localhost:3000, and it's there. Looking at the page source shows a bunch of assets:


<link href="/assets/application.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/animals.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/scaffolds.css?body=1" media="all" rel="stylesheet" type="text/css" />
<script src="/assets/jquery.js?body=1" type="text/javascript"></script>
<script src="/assets/jquery_ujs.js?body=1" type="text/javascript"></script>
<script src="/assets/animals.js?body=1" type="text/javascript"></script>
<script src="/assets/application.js?body=1" type="text/javascript"></script>

This leads to a couple of immediate questions: where did these files come from and how did they get there? The css files come from files in app/assets/stylesheets:

  • animals.css.scss
  • application.css
  • scaffolds.css.scss

The two .scss files were generated by the scaffold generator; application.css is the CSS manifest provided originally by Rails.

For the JavaScript, things are more complicated. Two of the source files are in app/assets/javascripts:

  • animals.js.coffee
  • application.js

The CoffeeScript file was generated by the scaffold generator, and the JavaScript manifest came from running the rails command. But what about the two jQuery files? They're definitely being served (I can retrieve http://localhost:3000/assets/jquery.js in the browser) but I have no idea where they're coming from. A little poking around shows that they are in the jquery-rails gem, where the files live in vendor/assets/javascripts. This gem is actually a Rails engine, and a look at rails/railties/lib/rails/engine.rb in the Rails source shows that the vendor/assets, lib/assets and app/assets directories inside an engine are added to config.assets.paths search path when the engine is initialized.

Turning from file locations to process, this is what shows up on the first page access in the Rails log:


Started GET "/" for 127.0.0.1 at 2012-04-08 08:00:14 -0500
Processing by AnimalsController#index as HTML
  Animal Load (0.1ms)  SELECT "animals".* FROM "animals"
  Rendered animals/index.html.erb within layouts/application (1.3ms)
Compiled animals.css  (35ms)  (pid 91415)
Compiled scaffolds.css  (34ms)  (pid 91415)
Compiled application.css  (305ms)  (pid 91415)
Compiled jquery.js  (4ms)  (pid 91415)
Compiled jquery_ujs.js  (2ms)  (pid 91415)
Compiled animals.js  (502ms)  (pid 91415)
Compiled application.js  (670ms)  (pid 91415)
Completed 200 OK in 1194ms (Views: 1189.7ms | ActiveRecord: 1.0ms)

So it's apparent that there was a transformation from the sources to the actual served version when I loaded the page. A little poking around shows that the compilation step only happens when necessary - loading new pages doesn't trigger it, but changing a file does. This is in development mode, of course.

Time to look at configuration settings. These are scattered across several files. First, in config/application.rb:


# Enable the asset pipeline
config.assets.enabled = true

# Version of your assets, change this if you want to expire all your assets
config.assets.version = '1.0'

And in config/environments/development.rb:


# Do not compress assets
config.assets.compress = false

# Expands the lines which load the assets
config.assets.debug = true

For completeness, the corresponding section in config/environments/production.rb:


# Disable Rails's static asset server (Apache or nginx will already do this)
config.serve_static_assets = false

# Compress JavaScripts and CSS
config.assets.compress = true

# Don't fallback to assets pipeline if a precompiled asset is missed
config.assets.compile = false

# Generate digests for assets URLs
config.assets.digest = true

So it looks like the default for config.assets.compile is true, which would explain why the server compiled things on demand when I ran the page. Finally, there is a section in the Gemfile that defines the asset pipeline itself:


# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails',   '~> 3.2.3'
  gem 'coffee-rails', '~> 3.2.1'

  # See https://github.com/sstephenson/execjs#readme for more supported runtimes
  # gem 'therubyracer', :platform => :ruby

  gem 'uglifier', '>= 1.0.3'
end

So there's a gem that understands SASS/SCSS, one that understands CoffeeScript, and one that I suspect is concerned with compression.

As to what actually does the serving - that's the sprockets gem. There's a comment in the sprockets README file: "Under Rails 3.1 and later, your Sprockets environment is automatically mounted at /assets." Xavier Noria kindly added a comment to this post with some details on how this mounting is managed. In development, then, assets are handled something like this:

  1. Somehow Rails is calling sprockets to recompile any changed assets when a page request comes in. Sprockets is the source of the "Compiled animals.js (502ms) (pid 91415)" lines in the Rails log.
  2. A request comes in for a file somewhere in /assets
  3. This request gets routed to sprockets
  4. Sprockets uses its asset paths array to find the first matching file and serves it

The manifest files come into things as a shorthand way to tell Rails which assets to include with a particular request. The default layout only specifies one stylesheet and one javascript file:


<%= stylesheet_link_tag    "application", :media => "all" %>
<%= javascript_include_tag "application" %>

Note that we no longer include the :cache settings on these declarations - that's now done within the asset pipeline bits.

The two application files are no longer the default stylesheet and javascript files, but manifests that contain instructions to sprockets about what to include in the compiled files. You can still include css and javascript in these files, but I suspect that's a bad practice at this point; better to separate out the actual code to different files. Here's the defaultapplication.css file:


/*
 * This is a manifest file that'll be compiled into application.css, which will include all the files
 * listed below.
 *
 * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
 * or vendor/assets/stylesheets of plugins, if any, can be referenced here using a relative path.
 *
 * You're free to add application-wide styles to this file and they'll appear at the top of the
 * compiled file, but it's generally better to create a new file per style scope.
 *
 *= require_self
 *= require_tree .
*/

The lines starting with *= are directives to sprockets. These two say to include the current file, and to include every file in a tree starting in the current file's folder, i.e. the entire app/stylesheets hierarchy.

Here's the default application.js file:


// This is a manifest file that'll be compiled into application.js, which will include all the files
// listed below.
//
// Any JavaScript/Coffee file within this directory, lib/assets/javascripts, vendor/assets/javascripts,
// or vendor/assets/javascripts of plugins, if any, can be referenced here using a relative path.
//
// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
// the compiled file.
//
// WARNING: THE FIRST BLANK LINE MARKS THE END OF WHAT'S TO BE PROCESSED, ANY BLANK LINE SHOULD
// GO AFTER THE REQUIRES BELOW.
//
//= require jquery
//= require jquery_ujs
//= require_tree .

This file is explicitly including the jQuery bits, which are found in the jquery-rails gem because it added itself to the asset search paths when it was loaded as an engine. It then includes every file in a tree starting in the current file's folder, i.e. the entire app/javascripts hierarchy.

As we've seen, in development mode sprockets and the include tags include these files by bringing them in individually. In production mode things are a bit different. Just running the server in production mode and hitting the root will result in an error in the log:


Started GET "/" for 127.0.0.1 at 2012-04-08 09:45:05 -0500
Processing by AnimalsController#index as HTML
  Rendered animals/index.html.erb within layouts/application (3.4ms)
Completed 500 Internal Server Error in 77ms

ActionView::Template::Error (application.css isn't precompiled):

That's because of the config.assets.compile = false setting in production.rb. In production mode, Rails won't call sprockets to compile assets on the fly. Instead, you need to set them up manually by running rake assets:precompile first. This task puts copies of the compiled assets in public/assets. In production mode, the page source changes as well:


<link href="/assets/application-958f49f3752dcbeab5370a3aee0bec08.css" media="all" rel="stylesheet" type="text/css" />
<script src="/assets/application-130530c48b043db45fa6a7a03b179580.js" type="text/javascript"></script>

Note that sprockets has concatenated all the applicable files together and gone to cache-buster style naming. If you want to test this out using WEBrick, you'll need to set config.serve_static_assets = true in production.rb, otherwise Rails won't try to serve files out of public/assets

.

What about controller-specific assets? Let's generate a second scaffolded resource:


rails generate scaffold bird name:string color:string
rake db:migrate

This adds a bunch of files to the application, include app/assets/javascripts/bird.js.coffee and app/assets/stylesheets/birds.css.scss. Loading a page now loads both sets of assets:


<link href="/assets/application.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/animals.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/birds.css?body=1" media="all" rel="stylesheet" type="text/css" />
<link href="/assets/scaffolds.css?body=1" media="all" rel="stylesheet" type="text/css" />
<script src="/assets/jquery.js?body=1" type="text/javascript"></script>
<script src="/assets/jquery_ujs.js?body=1" type="text/javascript"></script>
<script src="/assets/animals.js?body=1" type="text/javascript"></script>
<script src="/assets/birds.js?body=1" type="text/javascript"></script>
<script src="/assets/application.js?body=1" type="text/javascript"></script>

And precompiling only produces one set of application-level js and css files as well. This makes sense based on what we've seen in the manifest files, but it contradicts what you might guess from the comments in the controller-specific files. It turns out that by default the organization of JavaScript and CSS into controller-specific files is strictly to help you keep track of things; all of the code in all of these files is available on every page. To actually load things on a controller-by-controller basis, you'd need to modify the manifest files and the layouts. I've placed an example of how to do this on a separate gist: https://gist.github.com/2338096.

After working through the process to this point, I think I have a better idea what's going on. I still need to figure out a few things, chiefly: What's the deal with rake assets:precompile:primary and digested assets?

Hopefully this helped sort things out for other people as well. Additions and corrections are more than welcome.