What we do

How we do it

What is a Daemon

dae·mon (/ˈdēmən/) Noun

1. A background process that handles requests for services and is normally parallelized via forking or threading.

fork·ing (/fɔrkɪn/) Verb

1. Dividing a process into a parent and a child.

thread·ing (/THredɪn/) Verb

1. Executing part of a program independently of other parts.

This talk is just going to focus on process forking

Our Forking Library

Actively maintained for 5+ years.

Open source just days ago!

The Copy Thumbnailer

Without Forking

Processes files one at a time

Some file types take multiple seconds

Introduce Forking

/* setup forking daemon */
$server = new fork_daemon();
$server->max_children_set(4);
$server->max_work_per_child_set(3);

$server->register_child_run("generate");

/* registered call back function */
function generate($files)
{
  foreach ($files as $file)
  {
  	$file->generate_thumbnails();
  }
}

/* add work units */
$files = db_work_get();
$server->addwork($files);

/* process work blocking mode */
$server->process_work(true);

Documents blocking Images

Buckets

define("IMAGES", 1);
define("DOCUMENTS", 2);

$server->add_bucket(IMAGES);
$server->add_bucket(DOCUMENTS);
$server->max_children_set(2, IMAGES);
$server->max_children_set(5, DOCUMENTS);
$server->register_child_run("generate_images", IMAGES);
$server->register_child_run("generate_documents", DOCUMENTS);

$images = db_work_get('images');
$documents = db_work_get('documents');

$server->addwork($images, '', IMAGES);
$server->addwork($documents, '', DOCUMENTS);

Open Database Connections

Resource Management

/* setup forking daemon */
$server = new fork_daemon();
$server->register_parent_prefork(array('prefork'));
...

function prefork()
{
  db_close_all_connections();
}

Helpers

/* setup forking daemon */
$server = new fork_daemon();
$server->helper_process_spawn('helper');
...

function helper()
{
   pcntl_exec('/usr/bin/image_processor', array('-l'));
}

Storing the work queue

Flat Files

Local = Fast!

Simple

Does Not Scale

Cannot Share

Database

Sharable

Replicatable

Shardable

Sharing DB Queues

         id: 29323023
      ctime: 2013-03-22 13:34:33
     module: generate_thumbnails
       slot: image_raw
server_lock: 32342
 start_time: 2013-03-22 13:34:35
update_time: NULL
error_count: 0
       data: {"user_id":999,"image_id":3234123}

Statistics

Cloud processes a petabyte every 1.5 days

Copy is generating 125 thumbnails per second

In the last 4 months Copy has added 1.3 million users.

Daemonizing PHP

Using PHP to run your backend processes

GitHub Repo

By Andy Blyler (@ablyler) and Jon Kuperman (@insitedesignlab)