Queues

As mentioned in web scheduling article, queues deliver asynchronous trigger promise. So you should be able to do something from inside of trigger in a deferred execution manner. So instead of executing some code in trigger directly you insert data into a queue table. Data is then fetched row by row from your queue table using a queue processor. So the main ingredients are:

  • Queue table
  • Queue processor code module
After table and processor are in place you can start composing your queue. But first things first. Let us start with a queue table.

Queue Table

Queue tables are special kind of tables. they have minimal amount of columns, do not have triggers or conventional tbv's. To create one go to db-manager and press create table, tick the Queue Table check box, give your table a resonable name and create it.

When table is created you can add columns you need.

When you are satisfied with a data model of your queue table you can start using it from your trigger by inserting rows in to it. Alas it will not get processed yet since we have not defined how do we process it. Next we will create a code module to dequeue rows from our table.

Queue Processor

As mentioned before the easiest way to implement a QueueProcessor is creating a code module. In "jobs" article "queue" tab there is a button to "add code module". This makes it simple to get it started.

When created you still need to implement acual logic to handle your row data. Several things need to be kept in mind when implementing IQueueProcessor.UseAsync().

  • JobsContext.Current should be used to communicate with database and filestore.
  • You should process the row as fast as possible. Transaction is open deleting and returning this row with readpast table hint.
  • Uncaught exception will result in transaction rollback. Row will not be used and QueueProcessor will try using it again.

After you have implemented your queue processor code module you can start composing it all together. Do not forget to publish your code module!

Composing it together

So we already have table to be able to insert into it using a trigger and remove from it using a queue processor. We also have implemented a queue processor in a code module. If the code module is published it should now be available in a lookup. Go to "jobs" article "queue" tab and add row by supplying name of a queue processor and name of a table.

Properties described:

  • Batch size - how many subsequent queue reads will happen for single job run.
  • Degree of parallelism - how many simultaneous queue reads will happen for single job run.
  • Is load balanced - applies euristics to manage your batch size and degree of parallelism.
Obviously "Batch size" and "Degree of parallelism" work together. So if your batch size is 16 and degree of parallelism is 2 then 32 rows will be used in a single job run. Load balancing will take in to account the growth rate and total amount of rows to adjust batch size and degree of parallelism.

When the queue is defined you can go to "jobs" tab and use QueueProcessor type and pick queue processor ID from a lookup.

After that it should be up and running. Happy Queueing!