Abusing HTTParty for a Tender client

We all know how great Tender is and had a chance to play with their great API while writing a tool to send out an email every morning with any pending issues.

They’ve done some really great stuff with it, especially with their use of URI Templates.

If you haven’t played with URI Templates before, an example of their JSON with a template in it looks like:

{"site_href": "http://api.tenderapp.com/{site_permalink}"}

which, with the Addressable gem would allow us to say:

>> require 'addressable/template'
>> tmpl = Addressable::Template.new('http://api.tenderapp.com/{site_permalink}')
>> tmpl.expand('site_permalink' => 'help').to_s
=> http://api.tenderapp.com/help

To make things nice and strait forward, Tender follows the convention of using _href as the suffix for any key that contains a template. I was able to use this along with a little magic I snuck into Hash to make my code that much prettier.

All that was required to expand the templates was a little module:

module JsonHelpers
  def href(key, opts = {})

which I was able to mix in to any Hash that was returned:

# Make the JSON a litte bit more fun to work with
def self.add_json_helpers(data)
  case data
  when Hash
    data.send(:extend, TenderSummary::JsonHelpers)
    data.each { |_, value| add_json_helpers(value) }
  when Array
    data.each { |elem| add_json_helpers(elem) }

and then tell HTTParty to use it by specifying a custom parser:

module TenderSummary
  class TenderApi
    include HTTParty
    headers  'Accept' => 'application/vnd.tender-v1+json'
    format   :json
    parser   Proc.new { |b| add_json_helpers(Crack::JSON.parse(b)) }

    # ...

With all this done, it made it very simple to generate the URLs to the resources I wanted access to.

# assuming `site` holds the JSON returned from
# 'http://api.tenderapp.com/{site_permalink}'
discussion_url = site.href(:discussions, :state => :pending)

You can see the entire class of TenderApi on GitHub.

Posted Wednesday, January 6 2010 (∞).

Tracking initial memory usage by file in Ruby

I’ve found that as a project progresses, the initial memory usage of a Rails application seems to grow more and more.

The more time I spend trying to track down memory leaks (or just pieces of code that use more memory than they should) the more I realize that it’s a fairly imprecise science. I’ve had the best luck using tools to give me a good idea of where to start poking around. From there it’s just a matter of looking at the code and finding what silly things people are doing.

If I wanted to see what was contributing to the large memory footprint of an application on startup, tracking how much memory was allocated during each require would give me a good place to start.

The code to do this is amazingly strait forward:

module RequireTracking
  def require(*args)
    start_size = GC.allocated_size
    $require_stats[args.first] += (GC.allocated_size - start_size)
Object.send(:include, RequireTracking)
Kernel.send(:include, RequireTracking)

The entire implementation is available as a gist on GitHub.

The GC.allocated_size method is included in the RailsBench GC patch which is part of the Ruby Enterprise Edition interpreter.

One thing to understand is this is only tracking how much memory was allocated but not how much was freed. This will cause these statistics to include memory that was temporarily allocated and then no longer referenced. This can be useful because even temporarily using lots of memory can negatively impact startup time.

Another aspect to understand is the numbers we are tracking are what are normally called “self + children” in profilers. This means that all memory allocated by a file as well as anything allocated by files that are required from it are included in the statistics. This results in the same memory being counted multiple times, but is useful in understanding the total memory implications of requiring a file.

Running this on one of the projects I was working on found this little gem:

class Webster
  DICTIONARY = File.open(File.join(File.dirname(__FILE__), 'words')) do |file|
    file.readlines.collect {|each| each.chomp}
  def random_word

You can find the source on GitHub here.

This would be a prime candidate for refactoring if you are worried about your memory usage.

Posted Wednesday, December 30 2009 (∞).

Add NewRelic instrumentation for ThinkingSphinx

NewRelic provides a really great mechanism in their plugin to instrument just about anything.

One of the things I found when analyzing actions in NewRelic was that all of the time that was being spent in the ThinkingSphinx methods were being attributed to the template instead of the model. As we all know, mis-attribution of time spent can make tracking down trouble spots in your code much more difficult.

It ends up that all that is required to start tracking the time you are searching in ThinkingSphinx is a couple calls to add_method_tracer:

add_method_tracer :search, 'ActiveRecord/#{self.name}/search'
add_method_tracer :search, 'ActiveRecord/search', :push_scope => false
add_method_tracer :search, 'ActiveRecord/all', :push_scope => false

You can see the full code here.

Once you’ve required the code, you’ll start to see the #search and #search_count methods show up in your Performance Breakdowns:

Performance Breakdown

Isn’t that sweet?

Update: NewRelic has some great documentation for Custom Metric Collection if you want to do more.

Posted Tuesday, December 29 2009 (∞).

written by Eric Lindvall

I also appear on the internet on GitHub and Twitter as @lindvall and work hard to make Papertrail awesome.

themed by Adam Lloyd.