LIFO Queue in Python Concurrency with asyncio

Overview of LIFO Queue

LIFO queues, also known as stacks, are data structures that process the most recently added items first. This characteristic makes them particularly useful in scenarios where the application needs to handle the latest inserted item before others. In the context of Python's asyncio library, LIFO queues can be implemented using the LifoQueue class. This type of queue is beneficial when the order of processing requires the most recent item to be handled first.

Example of LIFO Queue

The following example demonstrates how to implement a LIFO queue using Python's asyncio library. The example includes a set of work items that are added to a LIFO queue and processed in reverse order of their insertion.

Listing 12.10 A LIFO queue

import asyncio
from asyncio import Queue, LifoQueue
from dataclasses import dataclass, field
 
@dataclass(order=True)
class WorkItem:
    priority: int
    order: int
    data: str = field(compare=False)
 
async def worker(queue: Queue):
    while not queue.empty():
        work_item: WorkItem = await queue.get()           #1
        print(f'Processing work item {work_item}')
        queue.task_done()
 
async def main():
    lifo_queue = LifoQueue()
 
    work_items = [WorkItem(3, 1, 'Lowest priority first'),
                  WorkItem(3, 2, 'Lowest priority second'),
                  WorkItem(3, 3, 'Lowest priority third'),
                  WorkItem(2, 4, 'Medium priority'),
                  WorkItem(1, 5, 'High priority')]
 
    worker_task = asyncio.create_task(worker(lifo_queue))
 
    for work in work_items:
        lifo_queue.put_nowait(work)                       #2
 
    await asyncio.gather(lifo_queue.join(), worker_task)
 
asyncio.run(main())
#1 Get an item from the queue, or “pop” it, from the stack.
#2 Put an item into the queue, or “push” it, onto the stack.

Explanation

In the above code:

  • We define a WorkItem class using the dataclass decorator, which includes fields for priority, order, and data.
  • The worker coroutine function processes items from the queue until it is empty. It retrieves (or "pops") items from the LIFO queue and marks them as done after processing.
  • The main coroutine function creates a LifoQueue and a list of WorkItem instances with varying priorities and orders.
  • Each work item is added to the LIFO queue using the put_nowait method, which places the item onto the stack.
  • The asyncio.gather function is used to wait for the queue to be fully processed and for the worker task to complete.

Usage Considerations

The decision to use a LIFO queue should be based on the specific requirements of your application. If the processing order necessitates handling the most recently inserted item first, a LIFO queue is the appropriate choice. This example provides a simple demonstration of how workers process elements in a LIFO order using Python's asyncio library.