Douglas F Shearer

Arch Arm Linux on PogoPlug - Keeping Time

I recently procured a PogoPlug V2 from Ebay, hoping to use it as a small linux server for various tasks. A PogoPlug is a small internet-enabled device that allows attached external hard drives to be accessed from anywhere, whether on the local network or on the wider internet, with minimal hassle.

It’s architecture is open, and a flick of a checkbox allows you to SSH in. I decided to install Arch Linux for Arm, and found this to be pretty easy to do following their installation guide.

Unfortunately the PogoPlug does not have a hardware clock, and will consistently think it is 1st of January 1970 after reboots. Not very handy, especially as I was planning to use it to report measurements to external services, and this was time-sensitive.

So, to set the time on each reboot, I installed and enabled the Network Time Protocol client, ntp. Here’s how to do that, and how to set your timezone correctly.

First up, update the package manager to have the latest packages available. You may be asked if you want to update pacman itself, and a bunch of other stuff. In my case it grabbed the latest linux kernel version, and took a fair amount of time. Run this command a few times to make sure everything is the latest it can be.

pacman -Syu

Next up, install ntp.

pacman -S ntp

To enable ntp on startup, edit /edit/rc.conf, adding ntpd to the DAEMONS array, and removing hwclock.

# before: DAEMONS=(hwclock syslog-ng network netfs crond sshd)
DAEMONS=(syslog-ng network netfs crond sshd ntpd)

Next you should configure ntp by editing /etc/ntp.conf. A list of local ntp servers can be found at http://www.pool.ntp.org/. I chose to use the UK servers.

Comment out any existing lines that begin with ‘server’, and add in those you want to use. Append iburst to the end, this specifies how the servers are polled.

server 0.uk.pool.ntp.org iburst
server 1.uk.pool.ntp.org iburst
server 2.uk.pool.ntp.org iburst
server 3.uk.pool.ntp.org iburst

Next, set your timezone. A complete list of zones can be found in /usr/share/zoneinfo. I went with Europe/London. Edit the TIMEZONE variable in /etc/rc.conf, changing it to your chosen zone.

TIMEZONE="Europe/London"

Now restart.

reboot

That’s it! You should now have the correct time and timezone.

# date
Fri Sep 16 19:58:09 BST 2011

Threadsafe File Consistency in Ruby

A large part of the work in the 0.7.0 release of Acts As Indexed was in guaranteeing the consistency of the index files which may be written to by many processes. I shall split this into two halfs: atomic writes, and locking writes.

I talk mostly of processes here, since most Rails hosting implementations at the moment employ multiple processes, though the same methodologies can be applied to threads.

Atomic Writes

An example: Say we have one process which writes to a file, and many processes which may be reading from that same file. If we do a simple write, it is possible that one of the reading processes may see a half-written file. While digging through the Rails source I discovered a monkey-patch on the Ruby File class which added a method called atomic_write.

The basic operation of this as is follows:

1. Write to a temporary file.
2. Move that temporary file to be the actual file we want to write to.

Since we are delegating the move operation to a system call, we can almost guarantee that any process reading the file will only see a fully written one, since all that is being changed during the move is a pointer to the file’s physical location on disk. A simple implementation of this would be thus:

require 'fileutils'

def atomic_write(path, temp_path, content)
  File.open(temp_path, 'w+') do |f|
    f.write(content)
  end

  FileUtils.mv(temp_path, path)
end

The Rails implementation goes a lot further than this, creating a tempfile in the OS mandated location, and making sure the newly written file has the same permissions as the original file.

Locking Writes

Another example: We have many processes, all of which can write to the same file. Our processes first read the file, and then make some change to it. A race condition for this looks as follows.

1. Process A reads the file.
2. Process B reads the file.
3. A makes changes and writes these.
4. B makes changes and writes these.

In this example, changes made by A are lost. The solution to this is to use locks, which are provided by the Ruby File class via the flock method.

def lock(path)
  # We need to check the file exists before we lock it.
  if File.exist?(path)
    File.open(path).flock(File::LOCK_EX)
  end

  # Carry out the operations.
  yield

  # Unlock the file.
  File.open(path).flock(File::LOCK_UN)
end

We can combine this with the atomic_write method as follows:

lock('my_file') do
  atomic_write('my_file', 'my_file.tmp', 'Hello, World!')
end

Rails’ file store has a great implementation of this pattern, which automatically unlocks the file again in case of an exception while the lock is applied.

Using Bundler With Rails 1.2

Bundler has truly been a revolution in how dependencies for Ruby projects are managed. The Bundler site provides a guide for using it with Rails 2.3 and Rails 3, but not Rails 1.2.

This won’t matter for most people, but if you have clients on older Rails versions, it can be nice to manage dependencies in the same manner across all installations.

So here’s how you do it:

1. Create a new Rails 2.x app, and copy it’s config/boot.rb across to your 1.2 app.

2. Make the VendorBoot#load_initializer() method look like the following:

class VendorBoot < Boot
  def load_initializer
    require "#{RAILS_ROOT}/vendor/rails/railties/lib/initializer"
    
    # Commented out as not in Rails 1.2.x.
    # Rails::Initializer.run(:install_gem_spec_stubs)
    # Rails::GemDependency.add_frozen_gem_path
  end
end

3. Follow the Rails 2.3 Bundler guide for the rest.

I’ve only tested this setup with Webrick, Mongrel and Passenger, using Rails 1.2.6, so your mileage may vary.

Prevent EC2 Instance Termination

Ever accidentally deleted an EC2 instance? Save yourself a lot of trouble and use the following API tools command:

ec2-modify-instance-attribute --disable-api-termination true i-01d3e23b

Swap in your own instance ID for best effect.

Run the same command again with the boolean switched to false to allow deletion.

Integrity Git post-receive Hook

I host a few of my repositories myself, rather than on GitHub. As a result I can’t take advantage of their marvelous post-commit hook support. Instead, when it comes to informing Integrity of new commits, I have to use a custom Git post-commit hook.

Use the following in /path/to/repo/.git/hooks/post-receive changing the login credentials, URL, and project name as necessary. Make sure the script is executable.

#!/bin/sh

# Username and password not needed if publicly accessible.
curl -d '' http://USERNAME:PASSWORD@example.com/PROJECT_NAME/builds

echo "\n\n=========="
echo "  Received push and alerted Integrity"
echo "==========\n\n"

Now whenever I push to the remote repo, Integrity is informed and grabs all the new commits to run the test suite against them. This could also work as a post-commit hook, if your Integrity CI server pulls from your local working repository.

Clearing a File Input Field Using JQuery

While working on some file uploads today, I was in need to a way to clear a file input field if a user changed their mind after selecting a file. I came up with the following simple solution.

In my example I place a ‘clear’ link next to the field, but a modified method could be called from anywhere.

HTML

Wrap the field you intend to clear in a span, with a descriptive ID so we can find it.

We’ll also want our ‘clear’ link to be descriptively IDed.

<span id="myfileinput"><input type="file" name="myfile" /></span>
<a href="" id="clearmyfile">clear</a>

JavaScript

Now in our JavaScript (JQuery) we want to add a click event to our clear link which will clear the file input field.

$('#clearmyfile').click(function(){
  var fieldSpan = $('#myfileinput');
  fieldSpan.html(fieldSpan.html());
});

Explanation

Since we can’t write to the value of a file input field due to security concerns, instead we overwrite the contents of the span, with the contents of the span (itself), thus creating a new file input DOM object which doesn’t have a value set.

A crude but working example can be found in this Gist.

Simple Rails Config

There are a lot of Rails config options around, but despite this, I thought I’d share mine.

My design goals for this were:

1. Once the configuration was loaded, it would be stored in memory for the next time it was required.
2. Nice usage syntax, like config.authentication.password rather than the oft-used config[:authentication_password].
3. Zero dependencies outside what is already included in Rails.

config/app.yml

# Rails Config.
# Copyrighted(c) Douglas F Shearer 2009
# Licensed under The MIT License.
site:
  url: http://example.com

authentication:
  username: bob@example.com
  password: myhackproofpassword

flickr:
  api_key: d3c3576398a4876c920553b714bc177f
  username: flickrusername

# Akismet.
wordpress:
  api_key: 876c920553b7

config/initializers/config.rb

# Rails Config Loader.
# Copyrighted(c) Douglas F Shearer 2009
# Licensed under The MIT License.
module Config

  class ConfigStore

    # Takes a hash as an argument.
    # If this hash contains other hashes, these too will turned into
    # ConfigStore objects.
    def initialize(contents)
      @contents = contents

      @contents.each do |k, v|
        if v.is_a?(Hash)
          @contents[k] = self.class.new(v)
        end
      end

    end

    def method_missing(sym, *args)    
      @contents[sym.to_s] || super
    end

  end

  def config
    @@config ||= ConfigStore.new(YAML.load_file("#{RAILS_ROOT}/config/app.yml"))
  end
end

include Config

Usage

With both these files in place, we can now call the config from anywhere in our app:

>> config.authentication.username
=> "bob@example.com"
>> config.authentication.password
=> "myhackproofpassword"

I’ll probably add overrides for environment specific configuration in the future, but this covers most of my needs for now.

Easy_Install Leopard bug: No Eggs Found

Today I came across Zed Shaw latest awesome project, Lamson, an Python mail server that isn’t stuck in the past. The Register has a pretty good rundown on it if you want to know more.

So anyway, on installing it, one of it’s dependences, the jinja2 templating engine, exposed a bug in easy_install that seems to be quite prevelant on OS X 10.5 Leopard. The error message will be something like No eggs found in /foo/bar/etc (setup script problem?).

The solution is to upgrade to the latest version of easy_install as so:

sudo easy_install http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg

Best to check the project page to see if a newer version has be released.

Credits: A similar older fix by Sean Lynch with a non-working patched upgrade.

Flic.kr - Flickr Short URLs Explained

Flic.kr links recently began appearing on Twitter and around the web, so I did a little bit of digging. Twitter has brought about a wild storm of URl shortening services, and some issues surrounding them. James Duncan Davidson has a good summing up of these, and some of the solutions.

One of these solutions is to use a link tag on a page to give an alternative short URL. Flickr has started to support this, as so…

On a photo page, say http://www.flickr.com/photos/douglasfshearer/3447346323/, we find in the source:

<link rev="canonical" type="text/html" href="http://flic.kr/p/6fCxXz" >

Twitter clients can now take a pasted Flickr link, and go look up this short url on the Flickr page, without making use of a third party service such as tr.im. Good stuff.

This also works for user accounts, as an example, mine would be http://flic.kr/douglasfshearer.

Tools

The short photo URL is the photo ID converted to Base58, so you need to turn one into the other.

The item that really brought this to my attention was Fraser Speirs Base58Encode Objective-C class, ideal for those of you making Twitter clients.

I’ve released a RubyGem for this, Base58, and a CLI script that takes a flickr URL and gives you the short version.

Acts_As_indexed v0.5.0 Released

My Acts_as_indexed plugin has been updated to version 0.5.0.

New in this version is:

• Ruby 1.9 and Rails 2.3 compatibility.
• Index location can now be set. Provides Heroku compatibility.
• Better errors on bad options.
• ActiveRecord order argument overrides ranking returned by find_by_index.
• Various test environment improvements
• Various Bugfixes

Get it on Github or view the RDoc.