Douglas F Shearer

Threadsafe File Consistency in Ruby

A large part of the work in the 0.7.0 release of Acts As Indexed was in guaranteeing the consistency of the index files which may be written to by many processes. I shall split this into two halfs: atomic writes, and locking writes.

I talk mostly of processes here, since most Rails hosting implementations at the moment employ multiple processes, though the same methodologies can be applied to threads.

Atomic Writes

An example: Say we have one process which writes to a file, and many processes which may be reading from that same file. If we do a simple write, it is possible that one of the reading processes may see a half-written file. While digging through the Rails source I discovered a monkey-patch on the Ruby File class which added a method called atomic_write.

The basic operation of this as is follows:

1. Write to a temporary file.
2. Move that temporary file to be the actual file we want to write to.

Since we are delegating the move operation to a system call, we can almost guarantee that any process reading the file will only see a fully written one, since all that is being changed during the move is a pointer to the file’s physical location on disk. A simple implementation of this would be thus:

require 'fileutils'

def atomic_write(path, temp_path, content)
  File.open(temp_path, 'w+') do |f|
    f.write(content)
  end

  FileUtils.mv(temp_path, path)
end

The Rails implementation goes a lot further than this, creating a tempfile in the OS mandated location, and making sure the newly written file has the same permissions as the original file.

Locking Writes

Another example: We have many processes, all of which can write to the same file. Our processes first read the file, and then make some change to it. A race condition for this looks as follows.

1. Process A reads the file.
2. Process B reads the file.
3. A makes changes and writes these.
4. B makes changes and writes these.

In this example, changes made by A are lost. The solution to this is to use locks, which are provided by the Ruby File class via the flock method.

def lock(path)
  # We need to check the file exists before we lock it.
  if File.exist?(path)
    File.open(path).flock(File::LOCK_EX)
  end

  # Carry out the operations.
  yield

  # Unlock the file.
  File.open(path).flock(File::LOCK_UN)
end

We can combine this with the atomic_write method as follows:

lock('my_file') do
  atomic_write('my_file', 'my_file.tmp', 'Hello, World!')
end

Rails’ file store has a great implementation of this pattern, which automatically unlocks the file again in case of an exception while the lock is applied.

Simple Rails Config

There are a lot of Rails config options around, but despite this, I thought I’d share mine.

My design goals for this were:

1. Once the configuration was loaded, it would be stored in memory for the next time it was required.
2. Nice usage syntax, like config.authentication.password rather than the oft-used config[:authentication_password].
3. Zero dependencies outside what is already included in Rails.

config/app.yml

# Rails Config.
# Copyrighted(c) Douglas F Shearer 2009
# Licensed under The MIT License.
site:
  url: http://example.com

authentication:
  username: bob@example.com
  password: myhackproofpassword

flickr:
  api_key: d3c3576398a4876c920553b714bc177f
  username: flickrusername

# Akismet.
wordpress:
  api_key: 876c920553b7

config/initializers/config.rb

# Rails Config Loader.
# Copyrighted(c) Douglas F Shearer 2009
# Licensed under The MIT License.
module Config

  class ConfigStore

    # Takes a hash as an argument.
    # If this hash contains other hashes, these too will turned into
    # ConfigStore objects.
    def initialize(contents)
      @contents = contents

      @contents.each do |k, v|
        if v.is_a?(Hash)
          @contents[k] = self.class.new(v)
        end
      end

    end

    def method_missing(sym, *args)    
      @contents[sym.to_s] || super
    end

  end

  def config
    @@config ||= ConfigStore.new(YAML.load_file("#{RAILS_ROOT}/config/app.yml"))
  end
end

include Config

Usage

With both these files in place, we can now call the config from anywhere in our app:

>> config.authentication.username
=> "bob@example.com"
>> config.authentication.password
=> "myhackproofpassword"

I’ll probably add overrides for environment specific configuration in the future, but this covers most of my needs for now.

Acts_As_indexed v0.5.0 Released

My Acts_as_indexed plugin has been updated to version 0.5.0.

New in this version is:

• Ruby 1.9 and Rails 2.3 compatibility.
• Index location can now be set. Provides Heroku compatibility.
• Better errors on bad options.
• ActiveRecord order argument overrides ranking returned by find_by_index.
• Various test environment improvements
• Various Bugfixes

Get it on Github or view the RDoc.

Rails Plugin: Acts As Follower

Followers are the new friends. Apps such as Twitter and Github have effectively enabled virtual stalking within their apps. So how can you do this easily without wasting time with your own join tables? Acts_As_Follower to the rescue!

Install

Usage

A service like Twitter would have a user model, which would be both a follower and followable…

  class User < ActiveRecord::Base
    acts_as_follower
    acts_as_followable
  end
  
  # In the console...
  user1.follow(user2)
  
  # Get the users following user2
  user2.followers # => # Array containing user1
  
  # Get the users user1 is following
  user1.following # => Array containing user2
  
  # Check following
  user1.following?(user2) # => true
  user2.following?(user1) # => false
  
  # Stop following
  user1.stop_following(user2)
  

Acts_As_Follower allows heterogeneous follow relationships too, so a user could follow repositories and users, say, in an application like Github.

This plugin saved me a lot of time during the start-up of a recent project, and it hopefully should do the same for you. For more usage examples, see the README.

Acts_As_Indexed now on GitHub

A few people had been requesting that I moved my Acts_as_indexed plugin to GitHub.

See the original post for the new install details.

Pagination

The pagination for this plugin has now been rolled into the release version, no more separate plugin.

JQuery on Rails

Now, I know I am totally late to the party on this one, but how much does JQuery rock??

For several years now I have done all of my JS stuff using the Prototype and Scriptaculous libraries. I think they’re pretty awesome tools, and certainly being part of the standard Ruby on Rails package has helped them gain a lot of respect and usage.

Today I had to do a fairly complex piece of UI prototyping that involved a lot of complex JS. It took me about an hour to do it with Prototype, but I wasn’t completely satisfied with the results. Some of m fellow ZoeCity devs had been talking about JQuery for a new project we’ve just started, so I thought I’d give it a bash. 20 minutes later I had a functional better implementation, with less, neater looking code.

What About Rails?

As I’ve already mentioned, Rails ships with Prototype, so how do you go about using JQuery in these places? Enter JRails by aaronchi. This is a Rails plugin which replaces all the regular Prototype utilizing helpers with JQuery. Sorted!

JQuery is stupidly easy to learn, especially for anyone from an OOP background, so you should definitely give it a try on your next project.

Mod_Rails aka Passenger Migration and Review

Deployment of Ruby on Rails apps to a production environment has always been a huge pain in the backside. The most common setup seen these days is to run a proxy in front of a cluster of “Mongrels”:, which leaves you with a lot of processes to look after. Not ideal. Recently I got excited about using Jruby and the Glassfish Gem to run apps, but this is something I’m going to come back to at a later date now.

So the new kid on the block is mod_rails (otherwise known as ‘Passenger’) for Apache. So how can this be any easier?

• Install is a case of installing a gem, running a command, and following the on-screen instructions.
• Rails now runs within the realm of Apache, so we can start and stop it as if it were one process.
• This spawns and manages a number of Ruby processes up to a maximum, as demand requires.
• Restarting the Ruby processes for a specific app is simply a case of modifying a single temp file.
• Multiple Rails apps can share the pool of Ruby Processes, so no more Mongrels hogging resources that could otherwise be used by other apps.

I run two apps on my Slicehost VPS; this blog, and my training diary. Based on their traffic I was running three and one Mongrels respectively, maybe too many for a 256MB VPS. For Mod_Rails, I decided to follow the recommendation in the documentation of running a maximum of two Ruby workers on the memory I had available. I was running Apache anyway for my public SVN, so the transition was a case of moving my virtualhosts over from Nginx, stopping Nginx and the Mongrels, and restarting Apache. Ludicrously easy.

Some basic benchmarks showed that performance was on a Par with Mongrel, which is certainly pleasing. One small gotcha that I encountered is that the user which the Ruby processes are run under is the same as the owner of environment.rb in your Rails deploy. So check this is the user you want.

Capistrano Restart

If you’re deploying with Capistrano (and why shouldn’t you be?), the add this to your deploy.rb to get your restart rolling:

	namespace :deploy do
	  task :restart do
	    run "cd /path/to/app/current; touch tmp/restart.txt"
	  end
	end

Barring typos, that should be good to go. I look forward to seeing the reception of this amazing improvement in Rails deployment by the community. Shout out in the comments with your thoughts and/or problems.

Update

I had some issues with static files not being served in one of my projects. My solution was to remove the public/.htaccess file that the rails command generates. Hope this helps other’s out.

Scotland On Rails 2008 - Day Two

Today was the second day of Scotland On Rails which I was visiting courtesy of a free ticket from EngineYard.

Update: They had tea, maybe all conferences should do this!

Here’s a run down of the talks I went to:

Keynote – David A Black

Despite having been ill the day before, David provided a talk on the similarities between programming and music. I’m sure other people will provide more on this, but one difference is that musicians don’t hate those who play different instruments, unlike coders ripping on other coders due to their choice of tools.

Rails Patterns – Jonathan Weiss

Jonathan gave a talk on some common rails patterns, here’s a rundown:

• Long running tasks, video/audio processing etc: Break these off into separate jobs, and run them later.
• Lots of data: Use a background process to push to a service such as “Amazon S3”:, and use your local disk as a cache in case of downtime or for popular content.
• Plugins: Don’t blindly use them, always look over the code. A lot of plugins are poorly or infrequently maintained.
• Dependencies: Vendor everything, Gems, Rails. Saves your developers a lot of time in getting up and running when they join the team.
• DOS avoidance: Split up your Mongrels into groups, that way multiple slow downloads or uploads don’t consume all your resource. Don’t use sendfile, use xsendfile, free’s up the Mongrels faster. Maybe use another tool to deal with uploads, Merb or CGI.

Collective Intelligence: Leveraging User Data to Create Intelligent Rails Applications – Paul Dix

Paul of Mint Digital gave a great talk on how you can mine your user data to provide them with recommendations. Using the “Netflix challenge” data, he showed an example using Ruby and some C libraries. Very clever stuff, I’ll probably be digging into this more later.

Advanced Ruby Class Design – Jim Weirich

Jim gave a great talk on designing Ruby classes in a Ruby way. He highlighted some cool tools, including the BlankState class, which undefines all the Object/Class methods normally attributed to a Ruby class. In 1.9 there will be a version of this called BasicObject, from which Object inherits, thus creating fewer problems for classes doing clever things like DSLs where namespaces may collide.

Domain Specific Languages: Molding Ruby – Joe O’Brien

Today Joe presented a talk on DSLs. So what is a DSL? Well, you know the absurd nomenclature Starbucks use for their coffees? Venti, Grande, etc. Well, that is a DSL, designed so that when a customer makes an order, the chance of the barista getting the order wrong is pretty wrong, as everything is ordered so that nothing can be missed out. I essence, as someone pointed out, the barista is a compiler.

A DSL is a mini language, if you will, with “Rake” being one of those that Rails developers will be most familiar with. When designed well, a DSL will bring methods calls and operations up to the level of natural language. Image the benefits of a well designed DSL in showing implementations and getting approval from clients. So, bring your code up to the client, not the client down to the code.

JRuby on Rails: Up And Running – Charles Oliver Nutter and Thomas Enebo

Charles and Thomas (THE JRuby guys) presented what was for me, the best talk of the entire conference. Not only in it’s revelations, but the enthusiasm they brought to the subject of JRuby.

The last time I played with JRuby was a year ago, when it was pre 1.0, and a bit buggy. During the talk the guys officially released 1.1, despite it being available for over a week. To save me a lot of effort I’m just going to list some of the facts regarding it:

• JRuby is just Ruby.
• Has a Java port of the Oniguruma regular expressions library, like Ruby 1.9.
• Compatibility is key in it’s development, Rails, Rake, RSpec etc all run. Full ruby 1.8.6 compatibility. 42k assertions in tests.
• Java 6 over 30% faster than 5, all good for JRuby.
• JRuby 1.1 is 2-5x faster than Ruby 1.8.
• Dynamic optimization makes programs run faster as they run longer.
• JRuby almost twice as fast as Ruby on a single thread. Threads in JRuby are real, so near-linear scaling across cores!
• Can take advantage of different profilers available to Java, the guys showed a live profiler demo inside NetBeans using fractal benchmark.
• Can call Java APIs/classes from within Ruby. Showed off a Swing example where the interface was build through Ruby methods, much nicer than the horrible Java methods.
• Can dynamically add methods to Java classes, Java won’t see them though.
• NetBeans supports JRuby and Ruby. Command completion, method completion, method jumping, regexp completion, % completion. Shows RDoc for methods as you type them. Auto complete require classes and subclass completion.
• Example of Processing , run Ruby algorithms, through the Java lib, hardware accelerated on OpenGL. Games, Nick? These guys reckon someone will do it soon.
• JRuby database support through activerecord.
• Package your Rails app as a WAR file, everything in 1 process. Throw into GlassFish, done, instant scaling across multiple processors.
• JRuby runs in asynchronous mode, can spin off new processes.
• Ruby 1.9 support on the way, maybe in the next major release.

Awesome stuff, expect an article on getting up and running with Rails on GlassFish here soon.

Wrapup

All in all, it has been a fantastic few days. I’ve learnt lots of stuff and met some great people. Roll on next year.

Thanks very much to the SoR Team for all their organisation.

Scotland On Rails 2008 - Day One

Today was the first (official) day of Scotland On Rails which I was visiting courtesy of a free ticket from EngineYard.

Have met lots of great people, as expected. Am rather surprised at the number of non-rubyists, there are a fair number of people who currently work full-time on Java or .Net and are pursuing Rails as a sideline with a view to either going freelance, or ‘sneaking’ Rails in the back door at their employer. They were all impressed that I was a full time Rails developer, and have been since I left University. Maybe it was jealousy, but I can see why it is a lifestyle that many would like.

Here’s a brief rundown of the talks I attended, and what I took away.

Keynote – Michael Koziarski

Many Rails followers will know Michael as Koz, the man on the Rails core you hassle until your patch gets accepted. His keynote wasn’t the inspirational piece I would usually expect as a keynote, but did cover some interesting material as far as code beauty is concerned. In Rails there is some nice code, and some not so nice code, as expected, but it was nice to be shown some of it, and the reasons for it’s existence.

Pushing Rails onto our Stack: A System Integration Study – Richie McMahon & Maria Gutierrez

Richie and Maria (was very amused by her accent, is Spanish, with random bits of Edinburgh mixed in) put on a talk about their adventures integrating Rails apps with an existing Java infrastructure. They covered:

As well as covering XMLRPC in the days pre-ActiveRecord, they also covered working with legacy database schemas. One interesting thing that was brought up is the lack of support in Rails for legacy STI keys. A quick patch seems to have solved it though.

Handling Long-Running Tasks in Rails – Andrew Stewart

Andrew gave a great overview of the solutions out there for running background and scheduled tasks in Rails. BackgroundJob seems to come out the best, through ease of installation, setup, breadth of use cases and persistance. Lot of options out there though.

Edgecase Dialog – Jim Weirich and Joe O’Brien

Due to some rescheduling, this was earlier than planned. Jim (Author of Rake) and Joe of EdgeCase gave us an amusing but informative three scene play on the benefits of using mocks and stubs in your tests. The examples were presented using Jim’s FlexMock library, which I shall be checking out as an alternative to Mocha over the next few weeks.

Code Generation: The Safety Scissors of Metaprogramming – Giles Bowkett

“Giles”: provided the most amusing top of the day, complete with pictures of Jessica Alba and Darth Vader in a sombrero. Apart from the comedy, his talk was a great overview of what can be done when code is used to generate code. In short, code == data && data == code! He highly recommends Jack Herrinton’s Code Generation in Action, which was originally supposed to be a Ruby book, but due to the publishers fears about a Ruby book (this was 2003) the second half of the book is filled with examples in various other languages.

Migrating to Ruby 1.9 – Bruce Williams

Bruce presented a great talk on some of the new features and gotchas in Ruby 1.9. The highlights for me were the new hash symbol syntax: { a: 1, b: 2, :c => 3}, and the persistent ordering of hashes. The new lambda syntax is somewhat less verbose than previously, and seems like a step backwards. Be sure to check out the app Bruce used to run code in both languages for comparison: Compare-1-9.

Bonus Erlang Session – Gordon Guthrie

Gordon is a (relative) local, having only travelled from Linlithgow. He works for a small two-man Erlang development team. He gave a great overview of why Erlang deserves the attention it has been receiving. Despite not showing many code samples, he certainly managed to whet my whistle for having another go at learning it.

All in all, an awesome day, despite the early start. Thoroughly looking forward to tomorrow.

Rails: Fix has_many through polymorphic has_many Relationships

Ok, so the title is a bit of a mouthful, but stick with this, the answer to a longstanding Ruby on Rails problem is at hand.

The Setup

Imagine you have a setup similar to the following:

class User < ActiveRecord::Base

  has_many :albums, :as => :albumable

end

class Album < ActiveRecord::Base

  belongs_to :albumable, :polymorphic => true
  has_many :images

end

class Image < ActiveRecord::Base
  
  belongs_to :album
  
end

So a User has many albums, which have many images. But, albums have a polymorphic relationship, so they can belong to other objects if required.

The Problem

The problem comes when we want to fetch all the images that belong to our user.

Thinking about it, we should just add has_many :images, :through => :albums to the user model, but unfortunately this breaks in Rails 2.0.2 and prior versions.

The Solution

Easy, just do one of the following:

1. Upgrade to Edge Rails, which may carry risks depending on your app setup, but does fix this issue along with adding lots of shiny new features.
2. Wait for Rails 2.1.0 to be released, if you’re patient!

Enjoy!