Running asynchronous job in Resque: How it works :)

Resque is a lightweight, fast, and a very powerful message queuing system used to run Ruby jobs in the background, asynchronously.  It’s generally used when the system asks for better scalability and quick response.

A Resque job is any Ruby class or module with a perform class method.

class MyDesign

  @queue = :myqueue

#Jobs will be placed and picked up from this queue.

# call to queue processing in Resque until later

  def self.perform(seconds)

    sleep(seconds)

  end
end

How the queuing of Redis works:

The real ability of resque comes from its queue,  Redis. Redis is a NoSQL key-value store. Unlike other NoSQL key-value stores, which takes only strings as keys/values, Redis can process lists, sets, hashes, and sorted sets as values, and operates on them atomically.  Resque works on the top of Redis list datatype, where the queue name behaves as a key and list as values.

Jobs of resque are en-queued on the Redis queue, and the designated workers de-queues the same for processing. All these operations are atomic, hence the en-queuers and workers do not bother about the locking and synchronous access. Every element of a list (set, hashes etc.) must be string, and so the data structures are not nested in Redis.

Redis is a very fast, in-memory dataset, and can persist to disk (configurable by time or number of operations), or save operations to a log file for recovery after a re-start, and supports master-slave replication.

Redis owns the command set to read and process the keys and it does not need SQL to inspect its data.

How Queuing with Resque works

Resque stores the job queue in Redis list named “resque:queue:name”, where each element of the list is a hash. Redis also maintains its own management structures,  with an additional “failed” job list.

It keeps all the the operation light.  We don’t need to  pass a lot of data to the job. Instead passing the references to other records, files, etc. is sufficient for the workers to pop and perform the jobs.

When a job is popped from the queue, Resque instantiates an object of its parent class and calls it’s perform method, passing the additional parameters.

Calling external systems with Resque

There are ports of Resque to other languages such as python, C, Java, .NET, node, PHP and Clojure. If your external system is written in one of these languages, then you can start workers listening to their queues. Since you are not talking to a ruby class with arguments, you can set up a placeholder class with the proper queue name. This will allow Resque plugins to fire on enqueue.