Queues


Queues

As mentioned in web scheduling article, queues deliver asynchronous trigger promise. So you should be able to do something from inside of the trigger in a deferred execution manner. So instead of executing some code in trigger directly, you insert data into a queue table. Data is then fetched row by row from your queue table using a queue processor. So the main ingredients are:

  • Queue table
  • Queue processor code module

After the table and processor are in place you can start composing your queue. But first things first. Let us start with a queue table.

Queue Table

Queue tables are special kinds of tables. they have a minimal amount of columns, do not have triggers or conventional tbv's. To create one go to DB-manager and press create a table, tick the Queue Table checkbox, give your table a reasonable name and create it.

When table is created you can add columns you need.

When you are satisfied with a data model of your queue table you can start using it from your trigger by inserting rows into it. Alas, it will not get processed yet since we have not defined how do we process it. Next, we will create a code module to dequeue rows from our table.

Queue Processor

As mentioned before the easiest way to implement a QueueProcessor is by creating a code module. In the "jobs" article "queue" tab there is a button to "add code module". This makes it simple to get it started.

When created you still need to implement actual logic to handle your row data. Several things need to be kept in mind when implementing IQueueProcessor.UseAsync().

  • JobsContext.Current should be used to communicate with the database and filestore.
  • You should process the row as fast as possible. The transaction is open deleting and returning this row with reading past table hint.
  • An uncaught exception will result in a transaction rollback. The row will not be used and QueueProcessor will try using it again.

After you have implemented your queue processor code module you can start composing it all together. Do not forget to publish your code module!

Composing it together

So we already have the table to be able to insert into it using a trigger and remove it using a queue processor. We also have implemented a queue processor in a code module. If the code module is published it should now be available in a lookup. Go to the "jobs" article "queue" tab and add a row by supplying the name of a queue processor and name of a table.

Properties described:

  • Batch size - how many subsequent queue reads will happen for a single job run.
  • Degree of parallelism - how many simultaneous queue reads will happen for a single job run.
  • Is load-balanced - applies heuristics to manage your batch size and degree of parallelism.

Obviously "Batch size" and "Degree of parallelism" work together. So if your batch size is 16 and the degree of parallelism is 2 then 32 rows will be used in a single job run. Load balancing will take in to account the growth rate and total amount of rows to adjust the batch size and degree of parallelism.

When the queue is defined you can go to the "jobs" tab and use QueueProcessor type and pick queue processor ID from a lookup.

After that it should be up and running. Happy Queueing!


Related articles