Blog

My thoughts and experiments.

© 2014. All rights reserved.

Another strange thing - an endless paginator

A little bit about my new program-frankenstein. Now it is a endless Paginator for Django. It sounds crazy, isn't?

Standart Django Paginator uses the count() function for the verification of page number. It is converted to the SELECT COUNT(*) ... query, of course. But as I was explained (I really don't know, maybe it's just an exaggeration - you can post your opinion in the commentaries), this is not a such lightweight query, as we want for the paginated rest api, because of the MVCC in PostgreSQL.

How we can avoid the extra COUNT(*) query? Don't panic, we can trick the Django.

First of all we need to disable count parameter from the api response. We can introduce a custom pagination serializer:

# serializers.py
class CustomPaginationSerializer(BasePaginationSerializer):
    next = NextPageField(source='*')
    previous = PreviousPageField(source='*')

# api.py
class SomeListView(generics.ListAPIView):
    model = SomeModel
    serializer_class = SomeSerializerClass
    pagination_serializer_class = CustomPaginationSerializer

The next our move - disable the page number verification. This can be done by the custom paginator class:

class CustomPaginator(Paginator):
    """ HACK: To avoid unneseccary `SELECT COUNT(*) ...`
        paginator has an infinity page number and a count of elements.
    """
    def _get_num_pages(self):
        """
        Returns the total number of pages.
        """
        return float('inf')

    num_pages = property(_get_num_pages)

    def _get_count(self):
        """
        Returns the total number of objects, across all pages.
        """
        return float('inf')

    count = property(_get_count)

    def _get_page(self, *args, **kwargs):
        return CustomPage(*args, **kwargs)


class SomeListView(generics.ListAPIView):
    model = SomeModel
    serializer_class = SomeSerializerClass
    pagination_serializer_class = CustomPaginationSerializer
    paginator_class = CustomPaginator

Oh, goodness - we introduced the infinity number of the pages and the infinity number of elements... But we want also the correct next/prev links, so one more detail:

class CustomPage(Page):
    def has_next(self):
        """ HACK: Select object_list + 1 element
            to verify next page existense.
        """
        low = self.object_list.query.__dict__['low_mark']
        high = self.object_list.query.__dict__['high_mark']
        self.object_list.query.clear_limits()
        self.object_list.query.set_limits(low=low, high=high+1)

        try:
            # len is used only for small portions of data (one page)
            if len(self.object_list) <= self.paginator.per_page:
                return False

            return True
        finally:
            # restore initial object_list count
            self.object_list = self.object_list[:(high-low)]

This solution looks very questionable, but exciting for me. If you have something to say about this - welcome! =)

comments powered by Disqus