Douglas F Shearer

Threadsafe File Consistency in Ruby

A large part of the work in the 0.7.0 release of Acts As Indexed was in guaranteeing the consistency of the index files which may be written to by many processes. I shall split this into two halfs: atomic writes, and locking writes.

I talk mostly of processes here, since most Rails hosting implementations at the moment employ multiple processes, though the same methodologies can be applied to threads.

Atomic Writes

An example: Say we have one process which writes to a file, and many processes which may be reading from that same file. If we do a simple write, it is possible that one of the reading processes may see a half-written file. While digging through the Rails source I discovered a monkey-patch on the Ruby File class which added a method called atomic_write.

The basic operation of this as is follows:

1. Write to a temporary file.
2. Move that temporary file to be the actual file we want to write to.

Since we are delegating the move operation to a system call, we can almost guarantee that any process reading the file will only see a fully written one, since all that is being changed during the move is a pointer to the file’s physical location on disk. A simple implementation of this would be thus:

require 'fileutils'

def atomic_write(path, temp_path, content)
  File.open(temp_path, 'w+') do |f|
    f.write(content)
  end

  FileUtils.mv(temp_path, path)
end

The Rails implementation goes a lot further than this, creating a tempfile in the OS mandated location, and making sure the newly written file has the same permissions as the original file.

Locking Writes

Another example: We have many processes, all of which can write to the same file. Our processes first read the file, and then make some change to it. A race condition for this looks as follows.

1. Process A reads the file.
2. Process B reads the file.
3. A makes changes and writes these.
4. B makes changes and writes these.

In this example, changes made by A are lost. The solution to this is to use locks, which are provided by the Ruby File class via the flock method.

def lock(path)
  # We need to check the file exists before we lock it.
  if File.exist?(path)
    File.open(path).flock(File::LOCK_EX)
  end

  # Carry out the operations.
  yield

  # Unlock the file.
  File.open(path).flock(File::LOCK_UN)
end

We can combine this with the atomic_write method as follows:

lock('my_file') do
  atomic_write('my_file', 'my_file.tmp', 'Hello, World!')
end

Rails’ file store has a great implementation of this pattern, which automatically unlocks the file again in case of an exception while the lock is applied.

Integrity Git post-receive Hook

I host a few of my repositories myself, rather than on GitHub. As a result I can’t take advantage of their marvelous post-commit hook support. Instead, when it comes to informing Integrity of new commits, I have to use a custom Git post-commit hook.

Use the following in /path/to/repo/.git/hooks/post-receive changing the login credentials, URL, and project name as necessary. Make sure the script is executable.

#!/bin/sh

# Username and password not needed if publicly accessible.
curl -d '' http://USERNAME:PASSWORD@example.com/PROJECT_NAME/builds

echo "\n\n=========="
echo "  Received push and alerted Integrity"
echo "==========\n\n"

Now whenever I push to the remote repo, Integrity is informed and grabs all the new commits to run the test suite against them. This could also work as a post-commit hook, if your Integrity CI server pulls from your local working repository.

Flic.kr - Flickr Short URLs Explained

Flic.kr links recently began appearing on Twitter and around the web, so I did a little bit of digging. Twitter has brought about a wild storm of URl shortening services, and some issues surrounding them. James Duncan Davidson has a good summing up of these, and some of the solutions.

One of these solutions is to use a link tag on a page to give an alternative short URL. Flickr has started to support this, as so…

On a photo page, say http://www.flickr.com/photos/douglasfshearer/3447346323/, we find in the source:

<link rev="canonical" type="text/html" href="http://flic.kr/p/6fCxXz" >

Twitter clients can now take a pasted Flickr link, and go look up this short url on the Flickr page, without making use of a third party service such as tr.im. Good stuff.

This also works for user accounts, as an example, mine would be http://flic.kr/douglasfshearer.

Tools

The short photo URL is the photo ID converted to Base58, so you need to turn one into the other.

The item that really brought this to my attention was Fraser Speirs Base58Encode Objective-C class, ideal for those of you making Twitter clients.

I’ve released a RubyGem for this, Base58, and a CLI script that takes a flickr URL and gives you the short version.

Ruby Google Charts API Data Encoding

On Friday, Google released their Charts API. You pass your data in a URL, google give you a nice graph back.

The hard part is encoding you’re raw data, into one of Google’s three formats. So I thought I’d help all out!

All of the methods take an array of integers and the maximum value of those integers. It then returns the data in the relevant format.

my_data = [1,2,3,4,5]
my_max_value = my_data.sort.last
simple_encode(my_data, my_max_value)
text_encode(my_data, my_max_value)
extended_encode(my_data, my_max_value)

Simple Encoding

Documentation

    def simple_encode(data_arr, max_value)
      simple_encoding = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'
      # Douglas F Shearer 2007
      # http://douglasfshearer.com/blog/ruby-google-charts-api-data-encoding
      data = ''
      data_arr.each do |value|
        if value.is_a?(Integer) && value >= 0
          data << simple_encoding[((simple_encoding.length-1).to_f * value.to_f / max_value.to_f).to_i]
        else
          data << '_'
        end
      end
      data
    end

Text Encoding

Documentation

    def text_encode(data_arr, max_value)
      # Douglas F Shearer 2007
      # http://douglasfshearer.com/blog/ruby-google-charts-api-data-encoding
      data = ''
      data_arr.each do |value|
        if value.is_a?(Integer) && value >= 0
          data << ((999.0 * value.to_f / max_value.to_f).round/10.to_f).to_s
        else
          data << '-1'
        end
        data << ','
      end
      data.chop
    end

Extended Encoding

Documentation

    def extended_encode(data_arr, max_value)
      # Douglas F Shearer 2007
      # http://douglasfshearer.com/blog/ruby-google-charts-api-data-encoding
      characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-.'.split(//)
      data = ''
      data_arr.each do |value|
        if value.is_a?(Integer) && value >= 0
          new_value = ((4095 * value.to_f / max_value.to_f).to_i)
          sixtyfours = new_value/64
          units = new_value%64
          p '64:' + sixtyfours.to_s
          p 'units: ' + units.to_s
          data << characters[sixtyfours] + characters[units]
        else
          data << '__'
        end
      end
      data
    end

A Plugin

I may at some point release a ruby wrapper for this as a plugin. Currently I have a fairly good, tested prototype that covers most of the features for line and bar charts. More will be added as I need it, but release may not be imminent.

Any errata or comments for the above should be emailed to my usual address, or just leave a comment below.

Leopard

I installed Leopard this morning before work. From a day’s work I can say it’s faster (better multi-core support) is general use. A bonus too is having most of the development tools I use day to day build in. No more struggling to get the Ruby version changed and Gems installed.

One issue though, ScreenCapture (Command + Shift + 3 | 4) won’t capture screnshots, and instead crashes.

Process:         screencapture [98636]
Path:            /usr/sbin/screencapture
Identifier:      screencapture
Version:         ??? (???)
Code Type:       X86 (Native)
Parent Process:  launchd [1]

Date/Time:       2007-10-29 20:26:40.211 +0000
OS Version:      Mac OS X 10.5 (9A581)
Report Version:  6

Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Crashed Thread:  0

Application Specific Information:
*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[NSCFString _getValue:forType:]: unrecognized selector sent to instance 0xa0752a70

*snip*

Why? I have no idea at the moment, but it does leave me without a feature I use several times a day.

A few sites mention this if you do a Google search, but none suggest a fix. I’ll post a fix here once I discover one.

Update. There is now a thread on the Apple Support Forums about this. It seems to be caused by importing a user and their files from a previous install. If you create a new user on the same machine, the problem does not exist.

The Fix. The thread I linked to above has the solution. Basically if you delete the file ~/Library/Preferences/.GlobalPreferences.plist, then log out and in, the problem should be resolved. Enjoy.

Gravatar For Ruby And Ruby On Rails

Gravatar is a web based avatar service. You sign up, upload your avatar, and associate it with your email address(es). Now when you visit a site leveraging Gravatar and enter your email address (like comments on a blog), your avatar will automatically appear!

So today I decided to implement Gravatar on this blog, and here’s what I came up with.

If you are using this with Ruby on Rails, then add the code to you Application Helper (application_helper.rb). You may ommit the require line at the top, as this is already included in your Rails environment.

Generate A Gravatar URL

# This require is not needed for Rails.
require 'digest/md5'

# Returns a Gravatar URL associated with the email parameter.
#
# Gravatar Options:
# - rating: Can be one of G, PG, R or X. Default is X.
# - size: Size of the image. Default is 80px.
# - default: URL for fallback image if none is found or image exceeds rating.
def gravatar_url(email,gravatar_options={})
  grav_url = 'http://www.gravatar.com/avatar.php?'
  grav_url << "gravatar_id=#{Digest::MD5.new.update(email)}"
  grav_url << "&rating=#{gravatar_options[:rating]}" if gravatar_options[:rating]
  grav_url << "&size=#{gravatar_options[:size]}" if gravatar_options[:size]
  grav_url << "&default=#{gravatar_options[:default]}" if gravatar_options[:default]
end

All we have to do to get a Gravatar URL is:

gravatar_url('test@gravatar.com')
=> "http://www.gravatar.com/avatar.php?gravatar_id=df3d4780faaf2446a65ce39eafdfe1c0"

Use the @gravatar_options hash when you want to adjust the size, rating, or add a default URL for when there is no Gravatar.

gravatar_url('test@gravatar.com',{ :size => 20 })
=> "http://www.gravatar.com/avatar.php?gravatar_id=df3d4780faaf2446a65ce39eafdfe1c0&size=20"

For more detail on the particulars of the options, see the online documentation.

Generate Gravatar HTML

If you want to generate an img tag from this, something like the following is what you want.

# Returns a Gravatar image tag associated with the email parameter.
def gravatar(email,gravatar_options={})
  
  # Set the img alt text.
  alt_text = 'Gravatar'
  
  # Sets the image sixe based on the gravatar_options.
  img_size = gravatar_options.include?(:size) ? gravatar_options[:size] : '80'
  
  "<img src=\"#{gravatar_url(email, gravatar_options)}\" alt=\"#{alt_text}\" height=\"#{img_size}\" width=\"#{img_size}\" />"
  
end

Now if we run our examples from above…

gravatar('test@gravatar.com')
=> "<img src="http://www.gravatar.com/avatar.php?gravatar_id=df3d4780faaf2446a65ce39eafdfe1c0&rating=PG" alt="Gravatar" height="80" width="80" />"

And with options…

gravatar('test@gravatar.com',{ :size => 20 })
=> "<img src="http://www.gravatar.com/avatar.php?gravatar_id=df3d4780faaf2446a65ce39eafdfe1c0&rating=PG&size=20" alt="Gravatar" height="20" width="20" />"

Enjoy!

Taking it further.

A good enhancement to this would be to cache the Gravatars locally, to speed up their load times. Look out for a future blog post on this.

Fxyomatosis RSS Feed

A little while ago I created an RSS feed for Fyxomatosis, which can be found at the following address:

http://feeds.feedburner.com/douglasfshearer/fyxomatosis

It features:

• Text and images from the front page articles.
• Links to the full article if there is more to read.

Made with:

• Ruby (What else?)
• Hpricot
• Lots of fancy parsing and patience.

Update

In the course of adding the above feed to feedburner, I found that a Fyxomatosis RSS feed already exists, albeit without the pretty pictures and formatting that mine has. Strange Andy doesn’t publicise this.

Mocking User_Agent In Rails Tests

Sometimes you may want to check the user agent of a request in your Rails controller, be this for checking for spam, or only allowing services such as Feedburner to access your RSS feeds. Something like this:

def feed
  if request.user_agent !~"Feedburner"
    # redirect to feedburner
  else
    # display feed
  end
end

So what happens when you try to test this using the standard Rails functional tests? You should see an error something like the following:

ActionView::TemplateError: undefined method `user_agent' for
 #<ActionController::TestRequest:0xb69d9b88>

Not so good. The solution is to add the following module to your app. I chose to add it in the same file as the appropriate controller, after the end statement for the class:

module ActionController
  class TestRequest < AbstractRequest
    attr_accessor :user_agent
  end
end

Now your tests should run fine, and you can even set the user agent as you wish in your tests:

def test_access_allowed_to_feedburner
  @request.user_agent = 'Feedburner'
  get :feed
  assert_response :success
end

Hope this helps you out.

Site Search Using Google In Ruby On Rails

Enter Google. They index all of your content, no matter how it is generate. So why not use them for the search?

Prerequisites

Make sure you have the Hpricot gem installed –

gem update hpricot —source \

http://code.whytheluckystiff.net

Make sure you include both of them in your environment.rb as follows…

require 'hpricot'
require 'open-uri'

Controller

Generating the query string and getting the results is simple enough to do in a controller method, like so…

	
Search site using google

  def google

@query = params[:id]
@start = params[:start] if params[:start]
@start ||= “0”

	
Site url as well as any other conditions you’d like.
	
I chose to ignore all of my tag cloud pages, pagination pages and date pages.

    site = ‘douglasfshearer.com -“tagged with” -“posts by date” -page’

uri=“http://www.google.com/search?q=#{URI.escape(‘site:’+site+’ ‘@query’&start=’+@page.to_s)}”
html_result = open(uri)
parsed = Hpricot(html_result)

	
parse out the number of results.

    @no_results = parsed.to_s[/<\/b> of about (\d*)<\/b> from/,1]


@results = (parsed/“div.g”).map do |ele|
{:title => ele.at(“a”).inner_text,
:link => ele.at(“a”)[‘href’],

	
Huge fat hack alert. Use gsub to get rid of the weird stuff around the bold statements.

        :description => (ele/(“font”..“font/br”)).to_s.gsub(/\221/,‘’).gsub(/\222/,’’)}

    end

<%- if @results -%>
<h3> <%= @start.to_i+1 -%> - <%= @start.to_i+10 -%> of about <%= @no_results -%> results.</h3>

<%- @results.each do |r| -%>
	<h4><%= link_to r[:title], r[:link] -%></h4>
	<p><%= r[:description] -%></p>
<%- end -%>

<%= link_to 'Prev', :start => @start.to_i - 10 if @start.to_i >= 10 -%> | 
<%= link_to 'Next', :start => @start.to_i + 10 if @start.to_i < @no_results.to_i - 10 -%>
<%- end -%>