Motivation For Sharing

Last Week, I wrote a post on combining two open source projects. I’ve unofficially dubbed this project “medialogue”. One of the major flaws in Medialogue is the time it takes to process video is very high. This problem is easily solved with a queue mechanism. Django RQ is really freaking easy at baseline, but I was confused by the inclusion of all of the advanced features as to what was the bare minimum. This is a guide how do the bare minimum to get django_rq up and running.

This is your app on drugs

Requirements

Docker is really the only hard requirement. I will be assuming you are using my seed project, but you can easily adapt this to your own docker setup.

Getting Started

I am leveraging this on an existing app I am working on, you can easily follow along by cloning medialogue pre-rq.

git clone -b pre-rq git@github.com:derek-adair/medialogue.git

Synchronous Video Conversion

To demonstrate how impactful a queue can be for your user experience lets hook into some signals and convert video. If you’re following along with your own app, just skip ahead to the config.

Note: Signals are a controversial topic in django. Some say they shouldn’t even exist. Personally, I try to reserve them for when I am wishing to extend the functionality of a model that I do not have “ownership” over. Meaning, if you wanted to trigger some code after a user has been added and you’re wishing to not looking to take ownership of the User model. The main argument against signals is that they convolute code and are hard to keep track of. I’ve personally experienced this and can testify under oath this is true… and I dont really care. The proper way to do this is to hook into the Video Models save function somehow. Some day I’ll fix this and dive into the subject more, but i’m lazy and this is how I wrote it on my first pass forgetting about how horrible signals can be.

/medialogue/signals.py

from django.db.models.signals import post_save
from django.dispatch import receiver
from .models import Video
from video_encoding import tasks

@receiver(post_save, sender=Video)
def convert_video(sender, instance, **kwargs):
    tasks.convert_all_videos(instance._meta.app_label, instance._meta.model_name, instance.pk)

/medialogue/apps.py

from django.apps import AppConfig


class AntiTimelineConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'medialogue'

    def ready(self):
        import medialogue.signals

Alrighty then, now it’s time for your to experience the pain of video conversion without a queue. Go ahead and upload some videos by navigation to /admin/medialogue/video/add/ in your app. Upload a video (I’ve only tested mp4). You will now be forced to sit and wait while the video conversions are taking place. However, when they are done you can now see a bunch of different resolutions for your video!

Configuration

  1. Add redis to your docker-compose.yml 
    redis:
  image: redis:latest
  expose:
    - "6379"
  # --apendonly: enable fully-durable alternative to snapshotting
  command: ["sh", "-c", "redis-server --appendonly yes"]
  volumes:
    - ./queue_data:/data
  2. Add django-rq to requirements.txt and install it. (docker-compose build in my seed project)
  3. Add django_rq to the INSTALLED_APPS in settings.py
  4. Configure RQ 
    REDIS_PORT = '6379'
REDIS_HOST = 'redis'
   
RQ_QUEUES = {
    'default': {
        'HOST': REDIS_HOST,
        'PORT': REDIS_PORT,
        'DB': 0,
        #'PASSWORD': REDIS_PASSWORD,
        'DEFAULT_TIMEOUT': 360,
    }
}
  5. Start a rqworker 
    docker-compose run --rm web ./manage.py rqworker default

    NOTE: This will be removed when you stop the container

Asynchronous Queue

Now that we have all the plumbing lined up its pretty straight forward to pass redis our tasks and have them be executed without locking up the UI.

/medialogue/signals.py

from django.db.models.signals import post_save
from django.dispatch import receiver
from django_rq import enqueue
from .models import Video
from video_encoding import tasks

@receiver(post_save, sender=Video)
def convert_video(sender, instance, **kwargs):
    enqueue(tasks.convert_all_videos,
            instance._meta.app_label,
            instance._meta.model_name,
            instance.pk)

If you look closely at the code you can see enqueue takes a function and the arguments you’d like to pass to said function. At this point go ahead and re-upload the same video and observe how much better the UX is. Initially you will only see the first format @ 0%, but refreshing will show you the progress.

It’s also worth adding the django_rq queue views to your admin app. If you are having any troubles debugging this is where you can look to see any error messages, and ensure that things are working as expected.

/settings/urls.py - or wherever your root urls are (optional)

urlpatterns = [
    path('admin/', admin.site.urls),
    path('admin/queues/', include('django_rq.urls')),
    path('photologue/', include('photologue.urls')),
    path('', include('medialogue.urls', namespace='medialogue')),
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Conclusion

It’s a very worthwhile venture integrating with django_rq when faced with expensive tasks like video conversion. When it comes down to it we added a little complexity on the deployment side, some app config and wrap a task in a function. The biggest cost is actually the added complexity to debugging.

The most important part is I made an incremental improvement I set out to do!

  • running time consuming tasks like thumbnail generation/video encoding in a queue like rq

Next Week

I will be documenting how I do bulk upload of photos and videos to two separate models. It’s been a bit obtuse to figure out how to interface with my desired file upload UI - Filepond. Almost entirely because the demos got me hooked and I didn’t really actually read the server side documentation… woops. Regardless it’s not exactly intuitive how one might say, do bulk upload with a tool like filepond.