April 24, 2009

Rails gems unpack native

Suppose you need to work on an already existing Rails project. The ideal situation would be that right after the check-out you should be able to ./script/server and pointing at http://locahost:3000 the app is up and running. Most of the time it’s easy as this adding just the rake db:migrate step for development on sqlite. The next common complication is to setup mysql or another external database adding the rake db:create step. As a dependency you may have to edit config/database.yml with local mysql credentials. Only three steps.

But there is another common complication which is the Rails version and gems dependencies. You fix the first by freezing Rails with:

reborg:test reborg$ rake rails:freeze:gems
(in /Users/reborg/tmp/test)
Freezing to the gems for Rails 2.3.2
rm -rf vendor/rails
mkdir -p vendor/rails
cd vendor/rails
Unpacked gem: '/Users/reborg/tmp/test/vendor/rails/activesupport-2.3.2'
mv activesupport-2.3.2 activesupport
Unpacked gem: '/Users/reborg/tmp/test/vendor/rails/activerecord-2.3.2'
mv activerecord-2.3.2 activerecord
Unpacked gem: '/Users/reborg/tmp/test/vendor/rails/actionpack-2.3.2'
mv actionpack-2.3.2 actionpack
Unpacked gem: '/Users/reborg/tmp/test/vendor/rails/actionmailer-2.3.2'
mv actionmailer-2.3.2 actionmailer
Unpacked gem: '/Users/reborg/tmp/test/vendor/rails/activeresource-2.3.2'
mv activeresource-2.3.2 activeresource
Unpacked gem: '/Users/reborg/tmp/test/vendor/rails/rails-2.3.2'
cd -

In the old days and with my basic Rails knowledge I used to resolve gem dependencies by ./script/server manual testing until a stack trace with “missing library blah” was found. A time waster. The solution to this by “sudo gem install blah” is not good. First because of the pollution generated in local Ruby installation second because a new developer will face exactly the same problems over and over again.

Since Rails 2.x we can finally “rake gems:unpack” and have Rails load dependencies from vendor/gems. Gems are configured in environment.rb:

RAILS_GEM_VERSION = '2.3.2' unless defined? RAILS_GEM_VERSION
require File.join(File.dirname(__FILE__), 'boot')

Rails::Initializer.run do |config|
  config.gem "webrat"
  config.time_zone = 'UTC'

In the example we want webrat. The developer who introduces the dependency in the project needs to local install webrat:

reborg:test2 reborg$ sudo rake gems:install
(in /Users/reborg/tmp/test2)
gem install webrat
Successfully installed webrat-0.4.4
1 gem installed
Installing ri documentation for webrat-0.4.4...
Installing RDoc documentation for webrat-0.4.4...

and then she needs to unpack the gem plus the dependencies into the rails vendor/gems directory with:

reborg:test2 reborg$ rake gems:unpack:dependencies
(in /Users/reborg/tmp/test2)
Unpacked gem: '/Users/reborg/tmp/test2/vendor/gems/webrat-0.4.4'
Unpacked gem: '/Users/reborg/tmp/test2/vendor/gems/nokogiri-1.2.3'

The next developer who needs the application just need to check-out from source control and only think about migrations and the database without polluting their own local ruby gems installation. But there is a problem with native gems that needs the additional compilation step. The unpack operation doesn’t copy post-build artifacts into vendor/gems but only a clean copy of the gem. Nokogiri needs native build before use and we are in trouble if we just commit this project:

reborg:test2 reborg$ ./script/server
=> Booting Mongrel
=> Rails 2.3.2 application starting on
no such file to load -- nokogiri/native
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
Missing these required gems:

You're running:
  ruby at /usr/local/bin/ruby
  rubygems 1.3.2 at /Users/reborg/.gem/ruby/1.8, /usr/local/lib/ruby/gems/1.8

Run `rake gems:install` to install the missing gems.

The long stack trace (cut for brevity) asserts that webrat is missing at the end, while the correct message is at the beginning “no such file to load — nokogiri/native” which tells us we need an additional build step:

reborg:test2 reborg$ rake gems:build
(in /Users/reborg/tmp/test2)
Built gem: '/Users/reborg/tmp/test2/vendor/gems/webrat-0.4.4'
Built gem: '/Users/reborg/tmp/test2/vendor/gems/nokogiri-1.2.3'
reborg:test2 reborg$ 

and now the server starts up completely. What the build step does is clear if you have the project under SVN or GIT:

reborg:test2 reborg$ git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/Makefile
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/conftest.dSYM/
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/html_document.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/html_entity_lookup.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/html_sax_parser.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/mkmf.log
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/native.bundle
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/native.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_attr.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_cdata.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_comment.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_document.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_document_fragment.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_dtd.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_entity_reference.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_io.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_node.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_node_set.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_processing_instruction.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_reader.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_sax_parser.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_sax_push_parser.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_syntax_error.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_text.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_xpath.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xml_xpath_context.o
# vendor/gems/nokogiri-1.2.3/ext/nokogiri/xslt_stylesheet.o
# vendor/gems/nokogiri-1.2.3/lib/nokogiri/native.bundle
nothing added to commit but untracked files present (use "git add" to track)

you shouldn’t be tempted to “git add .” all of the files. The problem is that what is produced is platform dependent and should not be part of someone else’s installation. Essentially you need to svn/git ignore these files and remember to add at the end of “gems:build” an additional step which is not optional at this point. Fortunately is only required once by the developer adding the native gem. You can for example create a template like this one:

rake 'gems:build', :sudo => true

if File.exist?(".svn")
  run "svn propset svn:ignore '*.o' vendor/gems/nokogiri-1.2.3/ext/nokogiri"
  run "svn propset svn:ignore 'Makefile' vendor/gems/nokogiri-1.2.3/ext/nokogiri"
  run "svn propset svn:ignore '*' vendor/gems/nokogiri-1.2.3/ext/nokogiri/conftest.dSYM"
  run "svn propset svn:ignore 'native.bundle' vendor/gems/nokogiri-1.2.3/lib/nokogiri"
  run "svn ci -m 'Ignored native files'"

elsif File.exist?(".git")
  run("find . \\( -type d -empty \\) -and \\( -not "+
      "-regex ./\\.git.* \\) -exec touch {}/.gitignore \\;")
  file '.gitignore', <<-CODE
  git :add => "."
  git :commit => "-a -m 'Ignored native files'"

which produces the expected:

reborg:test2 reborg$ rake rails:template LOCATION=./on_gems_build.rb 
(in /Users/reborg/tmp/test2)
    applying  template: ./on_gems_build.rb
        rake  gems:build
   executing  find . \( -type d -empty \) -and \( -not -regex ./\.git.* \) 
   -exec touch {}/.gitignore \; from /Users/reborg/tmp/test2
        file  .gitignore
     running  git add .
     running  git commit -a -m 'Ignored native files'
     applied  ./on_gems_build.rb
reborg:test2 reborg$ git status
# On branch master
nothing to commit (working directory clean)
reborg:test2 reborg$ 

To summarize

  • Working on a Rails app you have the need to add gems dependency but some of the gems are native (developer adding gem dependency)
  • Edit config/environment.rb with wanted gems
  • run “rake gems:install”
  • run “rake gems:unpack:dependencies”
  • Git add everything or svn add everything, no need to commit, just useful to see what the build steps will add
  • run “rake gems:build”
  • git ignore or svn ignore all created files and dirs
  • commit

If you are just checking out the app with all gems installed, then you only need to “rake gems:build” after check-out.

Happy coding.

Comments (View)
blog comments powered by Disqus