Setting up integration testing is complex.  We have several technologies that we want to work together to test our application and they don't always cooperate.  Getting these technologies working requires a good mixture of Ops and Development knowledge.  There is also a lot of very important information spread out across several gems that you really only find in bug threads after hours of hunting.  Fortunately for you and unfortunately for me, I have put the time in to figure this out once and for all and put it into a gem I call Selenikit.

>

At Cloudpace, we are big on automation and uniformity.  We use Vagrant to set up or virtual boxes, and then we use Chef to configure our virtual boxes to the needs of the specific project.  The setup I am about to go over will work on Ubuntu 12.04 running Ruby 2.0.0 and Rails 3.2 with Selenium-Webdriver 2.33.0, Firefox 22, Capybara 2.1.0, Capybara-Webkit 1.0.0.  If you've ever tried to set up integration tests before, you know why I just listed all of those versions.  In short, having wrong versions of any of those pieces of software will make your tests not work with very cryptic and uninformative errors.  Versions matter!

Why use Selenium-Webdriver?

The reason we still prefer Selenium is because it provides us with a way to see what's going on in the browser in real-time.  Using gems like Debugger and Pry, we can use Capybara commands while the tests are in scope and see the effect in real-time right in front of us.  The days of guessing with screenshots and HTML outputs are over.  In my experience, using Selenium with Pry more than doubled my speed writing integration tests.

If Selenium is so great, why use Capybara-Webkit as well?

Selenium is slow… Very slow!  It has to actually render the browser which takes time, and waaaaay more time than you might think.  Here is a comparison of the two drivers with a  very small test suite where each step signs into the application and and tries to interact with a specific form:

With Capybara-Webkit and Xvfb:

Finished in 24.25 seconds
12 examples, 2 failures

With Selenium-Webdriver and VNC:

Finished in 1 minute 55.27 seconds
12 examples, 2 failures

The reason we use Xvfb with Capybara-Webkit is because we've noticed it's faster than VNC on our setup.  The maintainers of Capybara-Webkit also recommend Xvfb, so who am I to argue?

If you're not currently writing the test and need the GUI, going headless is almost 5 times faster!  A word of warning; Just because something works in Selenium-Webdriver that does not mean it will work in Capybara-Webkit.  Unfortunately at the time of writing this, we have run into a situation where elements which are display: inline-block aren't able to be clicked with Webkit.  To be fair, it isn't a problem with Webkit as much as a problem with the libraries that Webkit depends on.  I have conveniently provided a solution for this in the Selenikit gem which I will get to later on.

Introducing Selenikit

I created a gem called Selenikit which sets up our integration test environment for us!  I won't be going over how to use the gem (See the gem Readme for that), I will be going over how the gem works.  In short there are two Rake tasks, one to run the integration tests under Selenium-Webdriver, and one to run them under Capybara-Webkit.  There is also a helper method to properly set the Capybara Javascript Driver inside of spec_helper.rb.

Selenikit: Rake Tasks

Lets start with the Rake tasks which boots our VNC or Xvfb servers.  Admittedly, my Rake-Fu isn't what it should be, but it works!

require 'rspec/core/rake_task'

namespace :spec do
  RSpec::Core::RakeTask.new(:selenium) do |t|
    ENV["RAILS_ENV"] = "test"

    # This will be used in spec_helper to determine
    # which JS driver to use in capybara
    ENV["SELENIUM"] = "true"

    # Kill any previously running servers
    Rake::Task["vnc:kill"].reenable
    Rake::Task["vnc:kill"].invoke
    Rake::Task["xvfb:kill"].reenable
    Rake::Task["xvfb:kill"].invoke

    # Start the vnc server
    Rake::Task["vnc:start"].reenable
    Rake::Task["vnc:start"].invoke

    # Start firefox
    Rake::Task["vnc:firefox"].reenable
    Rake::Task["vnc:firefox"].invoke

    # Run all features or one file
    t.pattern = ENV["FILE"].blank? ? ["spec/features/*.rb","spec/features/**/*.rb"] : ENV["FILE"]

    # Make rspec pretty
    t.rspec_opts = ['-f documentation', '--color']

  end

  RSpec::Core::RakeTask.new(:webkit) do |t|
    ENV["RAILS_ENV"] = "test"

    # Kill any previously running server
    Rake::Task["vnc:kill"].reenable
    Rake::Task["vnc:kill"].invoke
    Rake::Task["xvfb:kill"].reenable
    Rake::Task["xvfb:kill"].invoke

    # Start the Xvfb server
    # We could use headless gem for this, but I know how it works so why not
    Rake::Task["xvfb:start"].reenable
    Rake::Task["xvfb:start"].invoke

    # Run all features or one file
    t.pattern = ENV["FILE"].blank? ? ["spec/features/*.rb","spec/features/**/*.rb"] : ENV["FILE"]

    # Make rspec pretty
    t.rspec_opts = ['-f documentation', '--color']

  end
end

namespace :xvfb do

  desc "Xvfb break down"
  task :kill do
    # System call kill all Xvfb processes
    %x{killall Xvfb}
  end

  desc "Xvfb setup"
  task :start do
    # This is what links the server to the test
    ENV["DISPLAY"] = ":99"
    # System call to start the server on display :99
    %x{Xvfb :99 2>/dev/null >/dev/null &}
  end
end

namespace :vnc do

  desc "vnc4server break down"
  task :kill do
    # System call kill vnc4server on display :99
    %x{vnc4server -kill :99}
  end

  desc "vnc4server js setup"
  task :start do
    # This is what links the server to the test
    ENV["DISPLAY"] = ":99"
    # System call to start the server on display :99
    %x{vnc4server :99 2>/dev/null >/dev/null &}
  end

  task :firefox do
    # System call to start firefox on display :99
    %x{DISPLAY=:99 firefox 2>/dev/null >/dev/null &}
  end

end

I've created some extra spec tasks spec:selenium and spec:webkit.  They each do what you think they do.  spec:selenium will run the tests on display :99 using a vnc server so it can be viewed.  spec:webkit will heedlessly run the tests.  vnc4server and Xvfb are system packages.  The key to linking these servers to the rspec tests is where we set ENV["DISPLAY"] to the display we started the server on.

An assumption that is made is that your integration tests are in spec/featuers and end in .rb. You can also run just one file by passing in the FILE environment variable.

Selenikit: Capybara Configuration

Now for the other half of the puzzle, lets take a look at how Selenikit configures Capybara:

module Selenikit
  module Rspec
    module Configure
      def self.set_driver
        Capybara.run_server = false
        Capybara.app_host = "http://0.0.0.0:80"
        Capybara.server_port = 80

        Capybara.register_driver :selenium_firefox_driver do |app|
          profile = Selenium::WebDriver::Firefox::Profile.new
          Capybara::Selenium::Driver.new(app, :browser => :firefox, :profile => profile)
        end

        if ENV["SELENIUM"] == "true"
          Capybara.javascript_driver = :selenium_firefox_driver
        else
          Capybara.javascript_driver = :webkit
        end
      end
    end
  end
end

The first thing that might pop out at you is that I told Capybara not to run a server.  I like to see the server output in real time, so I run my own server on port 80 in the test environment.  The other strange thing I've done is make my own Firefox driver when using Selenium.  Selenium will not take control of the Firefox browser we started on display :99 unless we set up our own web driver.

Selenikit: Switching Between Drivers

As I said before, sometimes you just won't be able to get something to work in Capybara-Webkit, so what do you do? You don't want to always run all of your tests with Selenium, that's too slow! What would be nice is if we could flag a test to always run with Selenium. Luckily, Selenikit provides a way to do this with the :selenium => true tag:

module Selenikit
  module Rspec
    module Helpers
      RSpec.configure do |config|
        config.before(:each) do
          # If the test is tagged as selenium or if we're in the selenium Rake task, set the driver to selenium
          if self.example.metadata[:selenium].present? || ENV["SELENIUM"] == "true"
            Capybara.current_driver = :selenium_firefox_driver
          end
        end
      end
    end
  end
end

If the Selenium tag exists or if we're running the Selenium Rake task, we want to use the Selenium driver. So if we run our tests with Webkit, any test with no :js => true tag will use Rack Test, any test with the :js => true tag will use Webkit, and any test with the :selenium => true tag will use Selenium.

Hopefully this saves someone a few hours of headaches and hair pulling.  I would have appreciated all of this information being in one spot before I ventured down the rabbit hole.

Feel free to leave any questions or suggestions you might have in the comments section.  I didn't put much thought into this beyond getting it working, so any suggestions for improvement are more than welcome!