John Hawthorn

Squeezing performance out of CUDA

2012-03-11T08:00:00Z

GPU code is known for being capable of extremely fast computation, but such performance only comes with well crafted code.

The following are the iterations I went through to squeeze performance out of a CUDA kernel for matrix multiplication in CSR format. Testing was done on a Quadro FX 5800 with CUDA 3.2 and involved a matrix multiplication of a large 52000x52000 sparse matrix with about 1% non-zero entries. All timings listed here are imprecise, and only meant to give an idea of relative performance.

Initial (naive) implementation

__global__ void cuda_mat_mul_kernel(const int *rows,
                                    const int *cols,
                                    const scalar_t *data,
                                    const scalar_t *v,
                                    scalar_t *dest,
                                    int N){
  int row = threadIdx.x + blockIdx.x * blockDim.x;
  if(row >= N)
     return;
  int row_start = rows[row];
  int row_end   = rows[row+1];
  float sum = 0.0f;
  for(int i = row_start; i < row_end; i++)
     sum += data[i] * v[cols[i]];
  dest[row] = sum;
}

This is a pretty straightforward implementation, similar to what one would write as a serial CPU implementation. It uses one thread per row, each of which simply computes the dot product and writes it back to global. This kernel takes about 100 milliseconds to run, several times slower than the equivalent CPU code.

Memory coalescing

Perhaps the most important factor in optimizing CUDA performance is to allow memory coalescing when accessing global memory. This is possible when sequential words of memory are accessed by sequential threads in a warp (thread 0 reads word 0, thread 1 reads word 1, etc.). This implementation achieves this by having multiple threads per row, which access words at an offset equal to their thread number within the block. The partial sum of each thread is then stored in per-block shared memory, which is then summed and stored as the final result.

#define THREADS_PER_BLOCK 32
__global__ void cuda_mat_mul_kernel(const int *rows,
                                    const int *cols,
                                    const scalar_t *data,
                                    const scalar_t *v,
                                    scalar_t *dest,
                                    int N){

  __shared__ volatile scalar_t sums[THREADS_PER_BLOCK * 2];
  const int row_start = rows[blockIdx.x];
  const int row_end   = rows[blockIdx.x+1];

  scalar_t sum = 0.0f;
  for(int i = row_start + threadIdx.x; i < row_end; i += THREADS_PER_BLOCK){
     sum += data[i] * v[cols[i]];
  }

  sums[threadIdx.x] = sum;
  __syncthreads();

  /* hardcoded reduction for 32 threads */
  sums[threadIdx.x] += sums[threadIdx.x + 16];
  sums[threadIdx.x] += sums[threadIdx.x +  8];
  sums[threadIdx.x] += sums[threadIdx.x +  4];
  sums[threadIdx.x] += sums[threadIdx.x +  2];
  sums[threadIdx.x] += sums[threadIdx.x +  1];

  if(threadIdx.x == 0)
     dest[row] = sums[0];
}

This kernel allows reads from the data and cols vectors to be coalesced. The reduction works without __syncthreads() calls between each addition since the threads will be executing them simultaneously. This is similar to the reduction example in the NVIDIA GPU Computing SDK. This kernel improved perforamnce to about 5 milliseconds.

Texture Memory

Although reads to data and cols are not coalesced, accesses to the v vector is still non-coalesced. By caching v in texture memory, the random reads will be faster.

texture<scalar_t> texRef;
#define THREADS_PER_BLOCK 32
__global__ void cuda_mat_mul_kernel(const int *rows,
                                    const int *cols,
                                    const scalar_t *data,
                                    const scalar_t *v,
                                    scalar_t *dest,
                                    int N){

  __shared__ volatile scalar_t sums[THREADS_PER_BLOCK * 2];
  const int row_start = rows[blockIdx.x];
  const int row_end   = rows[blockIdx.x+1];

  scalar_t sum = 0.0f;
  for(int i = row_start + threadIdx.x; i < row_end; i += THREADS_PER_BLOCK){
     //sum += data[i] * v[cols[i]];
     sum += data[i] * tex1Dfetch(texRef, cols[i]);
  }

  sums[threadIdx.x] = sum;
  __syncthreads();

  /* hardcoded reduction for 32 threads */
  sums[threadIdx.x] += sums[threadIdx.x + 16];
  sums[threadIdx.x] += sums[threadIdx.x +  8];
  sums[threadIdx.x] += sums[threadIdx.x +  4];
  sums[threadIdx.x] += sums[threadIdx.x +  2];
  sums[threadIdx.x] += sums[threadIdx.x +  1];

  if(threadIdx.x == 0)
     dest[row] = sums[0];
}

/* on CPU */
size_t offset = size_t(-1);
cudaBindTexture(&offset, texRef, v);

Using texture memory increases performance to about 3 milliseconds.

Other potential improvements

I implemented several other improvements which achieved small performance improvements. Results may vary.

Summing more rows per block (more threads per row), ensuring that a high occupancy is met (profile with COMPUTE_PROFILE=1).
Adjusting the number of blocks run.
Aligning each row to a cache line achieves a small speedup, since the coalesced reads will be aligned.

Implementing these further increased performance to about 2 milliseconds.

A final word

If you are in fact writing something similar, existing libraries like cuBLAS and cusparse are likely much faster than what one would create in a reasonable amount of time. The NVIDIA GPU-Accelerated Libraries page is a good place to start. The excellent CUSP C++ library also has a CSR kernel almost equivalent to this one, and is a very convenient library for most sparse matrix applications.

Speed up your rails startup time

2012-02-10T08:00:00Z

I've found when developing that the faster my rails app can start, the faster I can develop. This is mostly because of the speed I can start test cases. But speed of starting a server or a console is also a big factor. I'd like to present a summary of what others have shown to aid in this performance.

Eliminate unnecessary gems

First tip comes courtesy of tenderlove: (The number of gems installed on your system can impact rails boot time)[http://tenderlovemaking.com/2011/11/30/psa-the-number-of-gems-installed-on-your-system-can-impact-rails-boot-time/] In short, the gemspecs on your system are all loaded as your app starts () even when using bundler. Some inconsiderate gemspecs can make this even worse by performing I/O as they're loaded. To remedy this use gemsets if you're using rvm and eliminate unnecessary gems from your system.

Helpful commands,

uninstall outdated gems gem cleanup

uninstall ALL THE gems gem list | cut -d" “ -f1 | xargs gem uninstall -aIx

script/rails

Second, Jesse Storimer recommends running ./script/rails rather than rails itself. For example rails c becomes ./script/rails c. This results in a small but worthwhile performance improvement: ruby and gems are not only loaded once. I've created a shell command (tested in bash and zsh) to run script/rails if it exists and rails otherwise (ex. rails new).

function rails {
   if [ -e "script/rails" ]; then
      ruby script/rails $@
   else
      `which rails` $@
   fi
}

Living dangerously

Another posibilty is using one of the patched ruby interpreters. funny-falcon maintains verious performance patches. Patched interpreters are easily installed with rvm.

Upgrade yer rails

Finally, upgrade your rails apps to version 3.2 if you haven't already. Startup time is significantly improved in this version.

Work, work. Zug zug.

2011-07-16T07:00:00Z

I wrote a little script to block websites which would destract me from work. Inspired by get-shit-done and the Orc voices from Warcraft 2. The script can be called as either workwork or readytowork to enable the block, and workcomplete to unblock the websites.

Warning: Contains a list of websites which will waste your time.

#!/usr/bin/env ruby

SiteList = %w{
  reddit.com
  news.ycombinator.com
  twitter.com
  plus.google.com
  slashdot.org
  cracked.com
  vimeo.com
  youtube.com
  digg.com
}

StartToken = '## workwork'
EndToken = "## zugzug\n"

HostsFile = '/etc/hosts'
ConfigFile = '~/.workwork'

def disabled_sites
  if File.exist?(ConfigFile)
    open(ConfigFile).read.split("\n")
  else
    SiteList
  end
end

def workwork
  lines = []
  lines << StartToken
  disabled_sites.each do |site|
    lines << "127.0.0.1\t#{site}"
    lines << "127.0.0.1\twww.#{site}"
  end
  lines << EndToken

  open(HostsFile, 'a+') do |file|
    file.write lines.join("\n")
  end
end

def workcomplete
  open(HostsFile, 'r+') do |file|
    string = file.read
    string.gsub!(/#{StartToken}.*#{EndToken}/m, '')
    file.seek(0)
    file.truncate(0)
    file.write(string)
  end
end

# switch to root since we need to change /etc/hosts
if ENV["USER"] != "root"
  exec("sudo #{ENV['_']} #{ARGV.join(' ')}")
end

case File.basename($0)
when 'workwork' then
when 'readytowork' then
  workwork
when 'workcomplete' then
  workcomplete
else
  puts "Unknown command"
end

Now I just need a script to remove executable permissions from minecraft.

Dealing with the MySQL slow log on Amazon RDS

2011-07-13T07:00:00Z

Amazon RDS is a web service offered by amazon which provides users with MySQL databases in the cloud. It's an attractive service due to its convenient backups and easy replication.

Amazon doesn't provide shell access to RDS instances. For this reason, the slow query log, which is an invaluable tool for anyone with a somewhat large or complex data set, is not available in the traditional way. Amazon, however, does provide a method for enabling and retrieving the log.

Setup

To enable the slow query log on RDS, you should create or modify a DB parameter group which can be done from the AWS management console (online interface) or through various other tools. There are two attributes that should be changed, slow_query_log, which should be set to 1 to enable, and long_query_time, which should be set to the shortest length to be considered a slow query.

The slow query log is stored in the table mysql.slow_log. Rotating the slow query log is done by calling the stored procedure CALL rds_rotate_slow_log.

Accessing

Update: this script is now available as a gem: rds_slow_log

I've written a little ruby script to print out the slow log in the usual format. From this it can be read, or analyzed with tools like request-log-analyzer.

require 'rubygems'
require 'active_record'

class SlowLog < ActiveRecord::Base
  set_table_name 'slow_log'

  def query_time
    time_parse query_time_before_type_cast
  end
  def lock_time
    time_parse lock_time_before_type_cast
  end
  def to_s
      <<EOS
# Time: #{start_time.strftime('%y%m%d %H:%M:%S')}
# User@Host: #{user_host}
# Query_time: #{query_time}  Lock_time: #{lock_time}  Rows_sent: #{rows_sent}  Rows_examined: #{rows_examined}
#{sql_text};
EOS
  end

  def self.print_all
    find(:all, :order => :start_time).each do |query|
      puts query
    end
  end

  private
  def time_parse string
    hours, minutes, seconds = string.to_s.split(/:/)
    hours.to_i * 60 * 60 + minutes.to_i * 60 + seconds.to_i
  end
end

ActiveRecord::Base.establish_connection :adapter  => 'mysql',
                                        :database => 'mysql',
                                        :username => 'username',
                                        :password => 'password'
SlowLog.print_all

This was written to be compatible with both activerecord 2.3 and 3.0. If compatability with 2.3 were to be dropped, the time_parse hacks could be removed.

Introducing the dkim gem

2011-07-04T07:00:00Z

DKIM (defined in RFC 4871) is a way for email senders to claim responsibility for an email message. It works by adding an extra header to every email sent which cryptographically signs the message's headers and body. By verifying the signature, the receiver of DKIM-signed email can be certain the emails content has been unaltered and comes from who is claimed. By taking responsibility for these messages, the sender's reputation can allow the message to be less scrutinized by spam/phishing filters on the receivers end. For this reason, DKIM is desirable because is should increase deliverability.

I had the need to send a large amount of email through Amazon SES from a rails application. This was made very easy by the terrific fog library for accessing cloud services (I would recommend version 0.8.2 or above, which includes a bug fix I submitted). However I found no good way to DKIM sign email from ruby. There was a rubydkim gem, but it didn't work correctly in my testing, and required an external C library.

This prompted me to create the dkim gem: a simple ruby library for DKIM signing email messages. dkim depends only on ruby built with openssl support (or jruby-openssl if you're using jruby).

Most basic usage (which should be all that is necessary for most) is to configure the gem globally

1
2
3

Dkim::domain      = 'example.com'
Dkim::selector    = 'mail'
Dkim::private_key = open('private.pem').read

and then to sign any messages as needed

mail = <<eos
To: someone@example.com
From: john@example.com
Subject: hi

Howdy
eos

Dkim.sign(mail)
# =>
# To: someone@example.com
# From: john@example.com
# Subject: hi
# DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=example.com; q=dns/txt; s=mail; t=1305917829;
#   bh=qZxwTnSM1ywsrq0Ag9UhQSOtVIG+sW5zDkB+hPbuX08=; h=from:subject:to;
#   b=0mKnNOkxFGiww63Zu4t46J7eZc3Uak3I9km3IH2Le3XcnSNtWJgxiwBX26IZ5yzcT
#   VwJzcCnPKCScIJMQ7yfbfXmNsKVIOV6eSUqu1YvJ1fgzlSAXuDEMNFTjoto5rrdA+
#   BgX849hEY/bWHDl1JJgNpiwtpl4t0Q7M4BVJUd7Lo=
# 
# Howdy

A more detailed explanation can be found in the project's README on github.

I hope the library proves useful to others, and would love to hear from anyone using it.