Answers About Web App Architecture For Small Teams

Brent Simmons asked some questions recently about Web App Architecture.

At Papertrail I’ve been living a lot of this stuff on a daily basis for some years now and wanted talk about it from the perspective of being on a small team.

In thinking about the questions, I realized that a lot of these architectural questions are much more to do with business decisions than people realize. Cliff Moon recently had a great post about microservices that is worth reading for a very grounded perspective on why microservices can be great.

There is a sentence in particular that I think is very valuable to think about:

An important corollary of Conway’s law, in my experience, has been that teams will tend to scale about as well as the software that they create.

One of the big benefits of a distributed architecture is providing a structure for communication and collaboration with large numbers of people. The fundamental problem that everyone is trying to solve is reducing friction. The funny thing about friction is that it is caused by different things depending on the size and makeup of the organization.

So, stepping back for a moment, one of the prerequisites of making these decisions is understanding what sort of service and what sort of company you’re trying to build.

What sort of growth do you expect, in terms of users and traffic, but more importantly in terms of your engineering team?

Are you on a rocket ship where you’re going to be going crazy from the very beginning and never stop with 100 developers before you know it?

Are you going to be growing organically and plan on keeping your team small for a few years?

These different paths will give you very different requirements. If you are planning on staying a small team (say, less than 3-5 engineers) for a few years, the friction you feel is going to be coming from some place different than communication and coordination between engineers.

When you’re starting out, and when you’re small, the speed at which you can make changes and improvements makes all the difference in the world. Having a bunch of separate services with interfaces and contracts just means that you have to make the same change in more places and have to do busywork to share code.

What can you do to reduce the friction required to push out that new feature or fix that bug? How can you reduce the number of steps that it takes to get a change into the hands of your users? Having code in a single repository, using an established web framework like Rails or Django can help a lot in reducing those steps. Don’t be scared of monolithic web apps when you’re small. Being small can be an advantage. Use it.

Predictions of the future are seldom accurate. Be mentally prepared for your architecture and your needs to be very different a year from now.

Never stop reevaluating your past decisions. It’s likely you’ll reach a point where you need to start breaking out your codebase into smaller components. Don’t consider that a failure of your past self.

A fundamental part of scaling a growing service is that you’re never done. If you are continuing to grow, there will always be more and different bottlenecks that crop up. There will never be a piece of technology or a style of development or architecture that will mitigate that.

Don’t try to be too clever. Assume you’ll have to change everything at some point in ways you never expected and don’t let that scare you. It’s a lot of fun.

Posted Tuesday, September 16 2014 (∞).

Introducing Metriks

I was very inspired by Coda Hale’s Metrics Metrics Everywhere talk at CodeConf 2011 and have spent a lot of time over the past year thinking about it. After seeing rack-statsd and how it kept important process stats in the proctitle, I wanted the same thing for the background tasks that run Papertrail.

I hadn’t been able to find a metrics library for ruby that provided the calculations I was looking for, so I decided to experiment with creating one myself.

The end result was an API for measurement that I’m really proud of. For example, to calculate how much work a process is doing, use a meter to measure how many times an event has happened by calling the mark() method on the meter:

def perform(job)
  # the work

The meter provides methods to give a 1, 5 and 15 minute average rate-per-second. The process title can be updated in with the rate returned by one_minute_average() in another thread:

meter = Metriks.meter('tasks')
loop do
  $0 = "worker: #{meter.one_minute_average.to_i} tasks/sec"
  sleep 5

…which will give you the output from ps ax that looks like this:

22665 ?        S     17:09 worker: 273 tasks/sec

Once you are tracking those metrics in-process it becomes very easy to start sending them to remote services in all sorts of ways.

The library is called Metriks, an experiment in creating a ruby metrics library with a simple interface and the ability to send the metrics to a number of services.

What sort of stuff are we tracking?

Error rates, database insertion times, cache hits vs misses, messages-per-second processed by workers.


The main components of the library are the metrics and reporters.

  • Metrics are responsible for doing a specific kind of measurement
  • Reporters are responsible for sending metrics to a specific destination

Today you can send to Graphite, Librato Metrics and a log file.

Here’s a quick review of what it’s like to use it and what the important pieces of the library are. It isn’t a huge amount of code, so please check it out.

If you have any questions or comments, feel free to say hi on twitter.

What does it look like to use it?

Using a meter to track web requests

To track the number of requests per second a rack app was doing but didn’t care about timing info, use a Metriks::Meter which can be created by calling Metriks.meter('name').

Here’s a very simple example of what it would look like to track the number of requests per second a rack app was doing.

class MetriksMiddleware < Rack::Middleware
  def initialize(app)
    @app = app
  def call(env)

Sending that metric to Librato Metrics could get you a pretty graph looking like this:

Librato Metrics Meter

Using a timer to measure how long a method takes to run

To track how long it took to run a method, use a Metriks::Timer which can be created with Metriks.timer('name').

To time how long it took to run a fib() method, all it takes is:

def fib(n)
  n < 2 ? n : fib(n-1) + fib(n-2)

Metriks.timer('fib.time').time do
  puts fib(10)

Sending that metric to Librato Metrics could get you a pretty graph looking like this:

Librato Metrics Timer

Try it

To install the gem just add this to your Gemfile:

gem 'metriks'

The source is available on GitHub.

Metrics supported

  • Meter: used to measure the rate that something is happening (number of times per second a method is called).
  • Timer: used to measure how long it takes to perform something. Also contains a meter to track how many times it’s happening.
  • Counter: used to keep track of how many times something has happened since the process started. This is mostly used by the other metrics and isn’t often used directly.


Meters are used to keep track of a rate of an action (how many times per second it happens).

To mark when an action is performed:



Timers are used to keep track of how long it takes for an action to take. It also contains a Meter in it to track how often it happens.

To measure how long an action takes:

Metriks.timer('fib.duration').time do

It’s also possible to use it without making it a block:

timer = Metriks.timer('fib.duration').time


Counters are used to keep track of an absolute number. They can be incremented and decremented. This metric is generally used as the basis for other metrics instead of being one that would be used directly.

To increment:



Reporters take a Registry and report the metrics to a remote store.

For a detailed overview of the reporter API, it’s available on the wiki.


Sends metrics to graphite on a set interval. It takes a host and port of the carbon agent as required arguments. Example:'localhost', 3309).start


Send metrics to Librato Metrics on a specified interval. It takes the API credentials as two required arguments: email and token. Example:'user@metriks.local', '186dbe1cf215').start


Sends metrics to a logger on a specified interval. Example:

logger ='log/metrics.log') => logger).start

The main reason behind this reporter was that I wanted to be able to aggregate the metrics collected by multiple processes on the same system before they were sent to Librato Metrics similarly to how StatsD does. To facilitate that, I created a Papertrail webhook receiver that took the logs, parsed the metrics, and submitted them to Librato Metrics. The work resulted in a Sinatra app I’ve posted on GitHub at metriks_log_webhook.


Being inspired by rack-statsd I realized there are many metrics that are deep in my processes that would be very interesting to keep track of how my workers are running.

This reporter isn’t really like the others. It reports metrics by updating your proctitle so you can see select metrics when you run ps aux.

Because space in the process title is limited, it requires configuration to specify what metrics are reported.

Example usage:

reporter =
reporter.add 'reqs', 'sec' do

It would allow you to see the metric when you run ps ax:

22665 ?        S     17:09 thin reqs: 273.3/sec

Implementation Details

Metriks is thread-safe. It uses a combination of mutexes and the atomic gem. Using atomic reduces the need for mutexes without sacrificing thread safety.

It uses the hitimes gem to get high granularity timing data without having to call frequently.

Metriks doesn’t tie the gathering of metrics to how they are reported. This allows for swapping out where metrics are reported to without having to change any of the instrumentation.

The metrics classes themselves are mostly ports of the metrics from Coda Hale’s Java metrics library.


Thanks to Troy Davis, Joe Ruscio and Mathias Meyer for their help reviewing drafts of this post.

Posted Tuesday, March 6 2012 (∞).

Make your library a better citizen

I’ve recently ran into a case where I wanted the Rails logger to be at :debug level, but really didn’t care about the debug output from the Hoptoad plugin. It kept giving a big XML dump every time it reported an exception which just added noise to the log file that wasn’t relevant.

I’ve found many plugins including Dash (may it rest in peace) and Hoptoad who would steal the logger from Rails and reference that internally.

Here’s the logger in hoptoad:

# Look for the Rails logger currently defined
def logger
  if defined?(Rails.logger)
  elsif defined?(RAILS_DEFAULT_LOGGER)

Many times it’s useful to run your staging environment (or even production) with debug logging enabled, but the downside was that now all of these plugins would be logging at that level as well.

One little change that plugin authors can start to do to help the rest of us out is make sure you #dup the logger before you grab a copy.

def logger
  @logger ||= if defined?(Rails.logger)
  elsif defined?(RAILS_DEFAULT_LOGGER)

This would allow us to then say:

Hoptoad.logger.level = Logger::WARN

and escape all of the noise from plugins that we don’t care about.

Much thanks to Bruce Williams for first proposing this trick.

Posted Saturday, March 27 2010 (∞).

written by Eric Lindvall

I also appear on the internet on GitHub and Twitter as @lindvall and work hard to make Papertrail awesome.

themed by Adam Lloyd.