We at e2enetworks manage multiple services and manage infrastructures across multiple providers including our inhouse one. Although our app is written in python/django, we needed a task queue that could handle the heterogeneity of our system. Hence, we went with the system where the business intelligenece is mostly handled in the client. Beanstalkd seemed like nice work queue and our solution is written on top of that.
pip install -e git+https://github.com/E2ENetworksPrivateLimited/kamkaji.git#egg=kamkaji
- Beanstalk related:
BEANSTALK_HOST = "localhost"
BEANSTALK_PORT = 11300
- Job Command:
The backend command that will run on worker. This is essentially a django management command called run_job. You can configure this for your application.
JOB_BACKEND_COMMAND_FORMAT = "python manage.py run_job {job_name} --settings=<local settings>"
- Jobs Directory
Directory to look up jobs
JOBS_DIR = os.path.join(BASE_DIR_KEY, "jobs")
python worker/worker.py <host> <port> <log_file> <job_log_dir> <queue>
Example:
python worker/worker.py localhost 11300 /var/log/worker/worker_web.log /var/log/worker/ web
Add jobs in jobs directory in the root of the folder of your django project
- Better handling for retry
- Better handling for delayed jobs
- Suggestions??