Another strange thing - an endless paginator
21 Apr 2014A little bit about my new program-frankenstein. Now it is an endless Paginator for Django. It sounds crazy, isn’t?
Standart Django Paginator uses the count() function for the verification of page number. It is converted to the SELECT COUNT(*) ... query, of course. But as I was explained (I really don’t know, maybe it’s just an exaggeration - you can post your opinion in the commentaries), this is not a such lightweight query, as we want for the paginated rest api, because of the MVCC in PostgreSQL.
How we can avoid the extra COUNT(*) query? Don’t panic, we can trick the Django.
First of all we need to disable count parameter from the api response. We can introduce a custom pagination serializer:
# serializers.py
class CustomPaginationSerializer(BasePaginationSerializer):
next = NextPageField(source='*')
previous = PreviousPageField(source='*')
# api.py
class SomeListView(generics.ListAPIView):
model = SomeModel
serializer_class = SomeSerializerClass
pagination_serializer_class = CustomPaginationSerializerThe next our move - disable the page number verification. This can be done by the custom paginator class:
class CustomPaginator(Paginator):
""" HACK: To avoid unneseccary `SELECT COUNT(*) ...`
paginator has an infinity page number and a count of elements.
"""
def _get_num_pages(self):
"""
Returns the total number of pages.
"""
return float('inf')
num_pages = property(_get_num_pages)
def _get_count(self):
"""
Returns the total number of objects, across all pages.
"""
return float('inf')
count = property(_get_count)
def _get_page(self, *args, **kwargs):
return CustomPage(*args, **kwargs)
class SomeListView(generics.ListAPIView):
model = SomeModel
serializer_class = SomeSerializerClass
pagination_serializer_class = CustomPaginationSerializer
paginator_class = CustomPaginatorOh, goodness - we introduced the infinity number of the pages and the infinity number of elements… But we want also the correct next/prev links, so one more detail:
class CustomPage(Page):
def has_next(self):
""" HACK: Select object_list + 1 element
to verify next page existense.
"""
low = self.object_list.query.__dict__['low_mark']
high = self.object_list.query.__dict__['high_mark']
self.object_list.query.clear_limits()
self.object_list.query.set_limits(low=low, high=high+1)
try:
# len is used only for small portions of data (one page)
if len(self.object_list) <= self.paginator.per_page:
return False
return True
finally:
# restore initial object_list count
self.object_list = self.object_list[:(high-low)]This solution looks very questionable, but exciting for me. If you have something to say about this - welcome! =)