Elevated Rails RSS Feed

Painless monitoring with Scout

Wed, 06 Aug 2008 10:43:34 GMT

I write about monitoring from time to time. We host about 15 applications for our clients, so monitoring our servers is a high priority for us. We've experimented with a large number of tools including monit and god. Both work great as process monitors, but they aren't built for some of our most important tasks. Enter Scout. In less then a week I've been able to port all of our ad-hoc monitors to Scout. See inside for more information.

What makes Scout great is its configuration and plugin architecture. When a host runs it's monitors, it talks to Scout to download an executiong plan. This plan includes a list of plugins to execute. If a plugin has been added, the client will automatically download the required code. This makes it trivial to change monitors via the web and have your clients update automatically.

A Scout plugin is just a simple Ruby class that either returns data, sends and alert or raises an error. It provides a very simple architecture for adding new plugins. This means that unlike with monit, you can create monitors that are tailored to your environment.

Let's look at an example. Weare heavy users of Starling. We want to be able to monitor certain queues to make sure that our processors are running. Doing this under monit was a hack. I wrote a ruby program to read the queues and touch a watchdog file when the queue depth is below a threshold. I configured monit to raise an error when a file's timestamp was older than a set interval. This worked in practice, but it was complicated to set up and required modifying multiple pieces any time a change was required.

I was able to replace my cobbled together monitor in just a few minutes with Scout. Here is the full code to my new plugin:

class MissingLibrary < StandardError; end
class StarlingMonitor < Scout::Plugin


  attr_accessor :connection


  def setup_starling
    begin
      require 'starling'
    rescue LoadError
      begin
        require "rubygems"
        require 'starling'
      rescue LoadError
        raise MissingLibrary
      end
    end
    self.connection=Starling.new("#{option(:host)}:#{option(:port)}")
  end



  def build_report
    begin
      setup_starling
      connection.sizeof(:all).each do |queue_name,item_count|
        check_queue(queue_name,item_count) if should_check_queue?(queue_name)
      end
      report(@report)
    rescue  MissingLibrary=>e
      error("Could not load all required libraries",
            "I failed to load the starling library. Please make sure it is installed.")
    rescue Exception=>e
      error("Got unexpected error: #{e} #{e.class}")
    end
  end

  def should_check_queue?(name)
    option(:queue_re).nil? or /#{option(:queue_re)}/ =~ name
  end

  def check_queue(name,depth)
    q_depth = (depth||0).to_i
    @report ||= {}
    @report[name] = q_depth
    if q_depth > option(:max_depth).to_i
      alert("Max Queue Depth for #{name} exceeded","#{q_depth} items is more than the max allowed #{option(:max_depth)}")
    end
  end

end

There's a bit of code, but it is pretty simple. Scout starts our plugin by calling the build_report method. In build_report, I do a little setup to make sure can load all of the required libraries. After requiring in Starling, I make a connection. The plugin uses the option method to pull configuration information from the environment. Scout provides a simple configuration interface that allows plugins to be configured via the web.

After loading up the environment, I walk the list of matching queues and send any necessary alerts. After looking at all queues, I return a report that maps the name of each queue to its depth. Along with providing an alert infrastructure, Scout also stores your reported data and can plot it on a graph. For example, here is a graph of the depth of our starling queues on one server:

This simple interface to reporting and alerting is incredibly powerful. What makes it even more powerful is the ease at which you can install plugins. If you want to install my starling plugin, you can simply enter the URL to the Ruby file from github (http://github.com/mmangino/er-scout-plugins/tree/master%2Fstarling_monitor%2Fstarling_monitor.rb?raw=true)

As I said, I've moved all of my ad-hoc monitoring to Scout. I had to create a few plugins to make that work. They're available on Github at http://github.com/mmangino/er-scout-plugins/tree/master. Feel free to fork them and submit enhancements!

So far, using Scout has been a real joy. That's something I never thought I would say about a monitoring tool.

Memcached sessions and Facebook

Fri, 25 Jul 2008 09:41:24 GMT

Memcached makes an excellent session store for Facebook applications. Unfortunately, it doesn't work out of the box with Facebooker. See how to fix that inside.

By default, the Memcached session store doesn't allow session keys with dashes in them. Since Facebook session identifiers include a dash, this causes an immediate error. Luckily, there is a simple fix. Add the following code as either an initializer or to your environment.rb file to get your Facebooker sessions running with memcached.

# add - as an okay key
class CGI
  class Session
    class MemCacheStore
      def check_id(id) #:nodoc:#
        /[^0-9a-zA-Z-]+/ =~ id.to_s ? false : true
      end
    end
  end
end

Don't you love simple solutions to annoying problems?

We're looking for a few people to test a new product

Fri, 18 Jul 2008 06:56:57 GMT

We've been taking all of our payments from clients via ACH or electronic check for the last few months. We've really enjoyed the traceability and dependability that these payments give us. Along the way, we built a very simple payment solution that our clients use to pay our invoices. We're looking for 3 or 4 other people who are interested in trying it out. Our solution will be available within the next month and will be free for our testers. We can help you get the necessary merchant accounts if you need them.

Are you interested? Send me an email at mmangino@elevatedrails.com or give me a call at 773.259.0145 if you are interested.

Major Facebooker Updates

Wed, 16 Jul 2008 08:42:13 GMT

About a month ago I moved Facebooker development to GitHub. What a great decision that was. I've applied a flurry of patches to fix some outstanding issues and add some additional support to the platform. If you haven't done any Facebook development, Facebooker is a Ruby library for creating Facebook Applications. It is the library that I used in Developing Facebook Platform Applications with Rails. Details on the recent changes inside.

In the last few days, we've added a lot of new features. The biggest of which is support for Bebo and other social platforms. David Clements forked the repository and made massive internal changes to support multiple backends. With his change I was able to get a client's application running on Bebo in just a few minutes. Great work David! Chris Redinger submitted several changes including support for forgery protection in Facebook forms. Brian Lee added support for friend lists as well.

On top of those changes, I added support for the new action templates to both Facebooker and the Publisher. The Facebooker Publisher is an ActionMailer like interface to Facebook messaging. Check out the documentation (http://facebooker.rubyforge.org/ and http://facebooker.rubyforge.org/classes/Facebooker/Rails/Publisher.html in particular)

If you've already created a publisher, you will need to re-run the script/generate command to create a new migration. By default, Facebooker will now keep track of your template ids. To send a message using the new API, you'll need to create two methods. The first returns the templates. For example:

class MyPublisher < Facebooker::Rails::Publisher
  def publish_action_template
    one_line_story_template "{*actor*} did stuff with {*friend*}"
    one_line_story_template "{*actor*} did stuff"
    short_story_template "{*actor*} has a title {*friend*}", render(:partial=>"short_body")
    short_story_template "{*actor*} has a title", render(:partial=>"short_body")
    full_story_template "{*actor*} has a title {*friend*}", render(:partial=>"full_body")
  end

The name of the method needs to end in _template.

To register this template, you can call

MyPublisher.register_publish_action

or, to register all templates in a publisher, you can call

MyPublisher.register_all_templates

Facebooker will store the template id and the template name in the facebook_templates table.

To send a message from a stored template, you'll need to create another method that registers the data:


 def publish_action(f)
   from f
   data :friend=>"Mike"
 end

Then, you can call

MyPublisher.deliver_publish_action

Facebooker will look up the template id automatically.

This will all be covered in a forthcoming Beta of my book.

Our schedule is opening

Tue, 08 Jul 2008 12:28:06 GMT

We've been working pretty hard around Elevated Rails. We've launched a few new client projects this year and worked on some pretty massive applications. We've written a book and talked at RailsConf. We're even getting ready to launch a simple application to make Facebook development easier. We're finally starting to see some openings appear in our schedule. What does that mean? It's time to fill them!

If you are looking for a top notch development team to work on your application, let us know. We've built sites that handle serious load. We've built applications for Facebook and applications that integrate with telephone systems. We're test driven and ready to build your next application.

Give me (Mike) a call at 773.259.0145 to discuss your project. We'll talk about your goals and our abilities and see if we're a good match. If we're not, we'll try to recommend somebody who can do a great job for you.

Facebook Podcast

Thu, 19 Jun 2008 06:25:46 GMT

It's been a hectic week around here. After moving to Philadelphia on Sunday, I participated in a Podcast with Susannah Pflazer, my editor for Developing Facebook Platform Applications with Rails. You can listen to the interview at Pragmatic Bookshelf Podcasts. On top of that, I'm also hard at work updating the facebooker library for the upcoming changes. Hopefully things will slow down shortly to give Jen and I some time to enjoy our new city!

I'm speaking at RailsConf

Fri, 30 May 2008 10:33:41 GMT

My talk about Facebook Development is at 1:50 on Friday.

You can download the slides.

You can also view them online

Stop by and say Hi!

My Recipe Featured in a Video

Tue, 13 May 2008 16:15:46 GMT

I was really excited to have several recipes included in Advanced Rails Recipes. It's even cooler to have one of them shown in the video. Thanks to Mike Clark for pulling together a really great reference for Rails developers.

Sorry about the lag between posts. A vacation to Hawaii interfered with my schedule :)

Building a plugin (form_for and overriding methods continued)

Mon, 28 Apr 2008 19:42:39 GMT

In my last post, we looked at why Rails doesn't use getter methods in form_for. We also came up with a relatively ugly fix. In this post, we'll clean that code up and turn it into a plugin.

Previously, we figured out that we could override respond_to? to make form_for use getter methods on our Rails models. We ended up with the following code to do this for our pending_email method.

def respond_to?(*args)
  if args.first.to_s == "pending_email_before_type_cast"
    false
  else
    super
  end
end

We will make our code more modular in a couple of steps. First, let's change our respond_to? method to deal with an array of column names instead of a hard coded value. We could easily do something like:

def respond_to?(*args)
  if @use_getters.include?(args.first.to_s)
    false
  else
    super
  end
end

That code would work, but there is a problem. Notice that @use_getters is an instance variable. We probably don't want to create a variable to hold the list of columns to use getters for on every model object. Instead, we could store that variable in a class variable and create accessors for it. We probably won't ever remove an item from this list of attributes, let's create two methods add_method_to_use_getter and methods_to_use_getters. There's nothing tricky in these methods as you can see:

 
  def self.add_method_to_use_getter(method)
    @methods_to_use_getters ||= []
    @methods_to_use_getters << method.to_s+"_before_type_cast"
  end

  def self.methods_to_use_getters
    @methods_to_use_getters || []
  end

In our code, we first make sure that our array exists. If it does, we add the name of the method to hide to our array. In the next method, we return either the array of method names or an empty array. Next, we can modify our respond_to? method.

def respond_to?(*args)
  if self.class.methods_to_use_getters.include?(args.first.to_s)
    false
  else
    super
  end
end

With that done, we have just a couple of remaining steps to make this look nice. First, let's create make it so that the following code will work:

class User < ActiveRecord::Base
  use_getters_for :pending_email, :other_attribute
end

Doing that is easier than you might think. All we need is a use_getters_for method that takes a variable number of arguments. That is simply:

  def self.use_getters_for(*args)
    args.each {|a| add_method_to_use_getter a}
  end

Next, let's turn this into a plugin so that we don't have to constantly define all of those methods on our classes. To create a plugin, we'll start by running script/generate plugin uses_getters_for_forms to create a skeleton plugin. Next, we'll create a couple of modules inside the generated lib/uses_getters_for_forms.rb. We will need to create two modules. One for instance methods and one for class methods. (Including a module only adds instance methods. This is explained brilliantly in Patrick Farley's Mountain West Ruby Conf talk.) To avoid naming conflicts, we will put both of these modules into the UsesGettersForForms module.

# UsesGettersForForms
module UsesGettersForForms
  module ClassMethods
    def use_getters_for(*args)
      args.each {|a| add_method_to_use_getter a}
    end
    def add_method_to_use_getter(method)
      @methods_to_use_getters ||= []
      @methods_to_use_getters << method.to_s+"_before_type_cast"
    end

    def methods_to_use_getters
      @methods_to_use_getters || []
    end
  end
  module InstanceMethods
    def respond_to?(*args)
      if self.class.methods_to_use_getters.include?(args.first.to_s)
        false
      else
        super
      end
    end

  end
end

With that done, we just need to make that method available to all of our models. We customarily do that by including it in ActiveRecord::Base. We can do that for our class methods, but not for our instance methods. As Patrick Farley explains, Our included respond_to? method won't be called until after the Rails method is called. Tow work around this, we'll dynamicly include our respond_to? method only when we need to. Here is the code to put into init.rb to do this.

# Include hook code here
require 'uses_getters_for_forms'
ActiveRecord::Base.send(:extend,UsesGettersForForms::ClassMethods)

Finally, we need to modify our use_getters_for method to add the respond_to? method the first time this is called.

# UsesGettersForForms
module UsesGettersForForms
  module ClassMethods
    def use_getters_for(*args)
      unless @ugff_loaded
        include ::UsesGettersForForms::InstanceMethods
        @ugff_loaded=true
      end
      args.each {|a| add_method_to_use_getter a}
    end
    def add_method_to_use_getter(method)
      @methods_to_use_getters ||= []
      @methods_to_use_getters << method.to_s+"_before_type_cast"
    end

    def methods_to_use_getters
      @methods_to_use_getters || []
    end
  end
  module InstanceMethods
    def respond_to?(*args)
      if self.class.methods_to_use_getters.include?(args.first.to_s)
        false
      else
        super
      end
    end

  end
end

There's a lot of code there, and a lot of it is somewhat tricky. Don't just copy and paste this code. Understand why it works. Watch the video by Patrick Farley and play with the code. Next, we'll look at why this breaks for STI models and how we can fix it.

form_for and overriding methods

Thu, 24 Apr 2008 08:49:43 GMT

From time to time, we override ActiveRecord attribute methods to add some additional behavior. Yesterday, I was working on some code to require a user to approve a new email address before the change was made and ran into some problems using my new methods in forms. I'll show you how to fix this inside

First, let's take a look at the code in question. I have a User model with an email attribute. When a user wants to change their email address, I want to send an email to their new address and delay the actual change until they click a confirmation link.

To implement this, I added a pending_email attribute on the user mode. I then use this method in all of my forms for users. By default, the pending_email attribute is blank. When a user views their profile edit form, I want them to see their existing email address. It should be easy to make the pending_email attribute return either itself if it is set, or the email attribute if it is blank. That method looks like:

  def pending_email
    read_attribute(:pending_email) || email
  end

That does exactly what we want. Calling pending_email gives us the email address by default. Unfortunately, things break down when we try to use this method in a form. When we do something like:

  <% form_for @user do |f| %>
    <%=f.text_field :pending_email %>
  <% end %>

Our pending email never displays the existing email address. Once again, to understand what was happening, I had to dig in to the Rails source. I started by looking at the text_field method in form_helper.rb. That pointed me to the InstanceTag class. That finally led me to the following code:


class << self

        def value_before_type_cast(object, method_name)
          unless object.nil?
            object.respond_to?(method_name + "_before_type_cast") ?
            object.send(method_name + "_before_type_cast") :
            object.send(method_name)
          end
        end
end



  That code looks at the object in our form, and sees whether it has a pending_email_before_type_cast method. If it does, it calls it to get the value. If it doesn't, it calls pending_email. By default, ActiveRecord creates the _before_type_cast methods so that code can access the textual value of a field before it is turned into a ruby object. For example, calling created_at_before_type_cast will give you the textual representation of a date inside the database. There are good reasons for this behavior, but they don't really matter to us. All we need to know is that our pending email method doesn't every get called.


 So how do we fix this? I'll show you a hack to make it work today, and then tomorrow, or sometime in the next few days, I'll show you how to turn this into a plugin. We're going to exploit the trick that the value_before_type_cast method checks to see if our object responds to a method before it calles the _before_type_cast method. We can easily override respond_to? to make it lie about the existence of our method. Here's the code that I used:

  # fake out the helpers to use the method and not the raw attribute
  def respond_to?(*args)
    if args.first.to_s == "pending_email_before_type_cast"
      false
    else
      super
    end
  end



  That's all it takes! Our code checks to see if the caller is asking about our pending_email method. If they are, it lies and says the _before_type_cast method doesn't exist. For anything else, we fall back to the default behavior. Now, our users see their existing email address when they view their profile edit form.

You can see why we want to clean up this code a little. It's not obvious what that code is doing, and it's really easy to get wrong. (I spent 20 minutes debugging because I spelled type_cast wrong. I don't want to have to ever do that again)


  Expect to see more frequent posting here. My book, Developing Facebook Platform Applications with Rails is almost done and I will have a little bit more time to write.