README.rst
KQ (Kafka Queue) is a light-weight Python library which provides a simple API to queue and process jobs asynchronously in the background. It is backed byApache Kafka and designed primarily for ease of use.
Requirements
- Apache Kafka 0.9+
- Python 2.7+ 3.4+ or 3.5+
Getting Started
First, ensure that your Kafka instance is up and running:
# This command is just an example~$ ./kafka-server-start.sh -daemon server.propertiesLet's say you want to run the following function asynchronously:
import timedefmy_func(foo, bar, baz=None):"""This is a blocking function."""
time.sleep(10)return foo, bar, bazStart a KQ worker:
~$ kq worker --verbose
[INFO] Starting Worker(topic=foobar) ...Enqueue the function call as a job:
# Import the blocking functionfrom my_module import my_func# Initialize a queuefrom kq import Queue
q = Queue()# Enqueue the function call
q.enqueue(my_func, 1, 2, baz=3)Sit back and watch the worker process the job in the background:
~$ kq worker --verbose
[INFO] Starting Worker(topic=default) ...
[INFO] Processing Record(topic=default, partition=5, offset=3) ...
[INFO] Running Job 1b92xle0: my_module.my_func(1, 2, baz=3) ...
[INFO] Job 1b92xle0 returned: (1, 2, 3)Check out the full documentation for more details!
Installation
To install a stable version from PyPI (recommended):
To install the latest version directly from GitHub:
~$ pip install -e git+git@github.com:joowani/kq.git@master#egg=kqYou may need to use sudo depending on your environment setup.
Credits
This project was inspired by RQ and built on top of kafka-python.