Handling Large Datasets with Pagination and Cursors in Laravel MongoDB

Modern applications routinely deal with datasets containing millions of records. Whether you’re building an e-commerce platform with extensive product catalogs, a social media feed, or an analytics dashboard, you’ll eventually face the question of how to display large amounts of data without overwhelming your server or your users. Pagination is the standard solution, but not all pagination methods perform equally as your data grows. This article explores two approaches to pagination when working with Laravel and MongoDB: offset-based pagination using skip() and limit(), and cursor-based pagination that uses document pointers. You’ll learn how each method works internally, why offset pagination degrades at scale, and when cursor-based pagination offers a better alternative. By the end, you’ll have practical implementation examples and clear guidance on choosing the right approach for your application.

Offset Pagination: Mechanics and Performance Problems

Offset pagination is the traditional approach most developers learn first. The concept is straightforward: you tell the database to skip a certain number of records and return the next batch. If you want page 5 with 20 items per page, you skip the first 80 records and fetch the next 20.

How Offset Pagination Works

In MongoDB, offset pagination uses skip() to set the starting point and limit() to control how many documents to return. Laravel’s Eloquent provides the paginate() method that handles this automatically, or you can use skip() and take() manually for more control. Here’s a basic implementation:

use AppModelsProduct;

class ProductController extends Controller {
    public function index(Request $request) {
        $page = max((int) $request->input('page', 1), 1);
        $perPage = 20;
        $skip = ($page - 1) * $perPage;
        $products = Product::orderBy('created_at', 'desc')->skip($skip)->take($perPage)->get();
        return response()->json($products);
    }
}

The skip value calculation is simple: ($page – 1) * $perPage. For page 1, you skip 0 records. For page 2, you skip 20. For page 100, you skip 1,980. Laravel’s built-in paginate() method wraps this logic and adds metadata like total pages and navigation links:

$products = Product::orderBy('created_at', 'desc')->paginate(20);

This returns a paginator object with the results plus information about total records, current page, and URLs for previous and next pages.

Why Offset Pagination Fails at Scale

The problem with skip() becomes apparent when you examine what MongoDB does to execute the query. When you call skip(1000000), MongoDB doesn’t jump directly to record 1,000,001. It must scan through all one million documents before returning your results. The database reads and discards every skipped document, which means page 10,000 takes dramatically longer than page 1. Query time grows linearly with the offset value. If page 1 takes 5 milliseconds, page 1,000 might take 500 milliseconds, and page 10,000 could take several seconds.

This degradation happens regardless of how well you’ve indexed your collection because the skip operation itself requires traversing documents. You can observe this behavior using MongoDB’s explain feature:

db.products.find().sort({ created_at: -1 }).skip(1000000).limit(20).explain("executionStats")

The docsExamined field in the output will show over one million documents examined, even though you’re only returning 20.

The Count Problem

Offset pagination usually displays “Page X of Y” in the interface, which requires knowing the total number of documents in the collection. Getting this count on large collections is expensive. MongoDB must scan the entire collection to return an accurate count, and this operation doesn’t benefit from indexes the way filtered queries do. Several strategies can help mitigate the cost:

• Cache the count: store the total count in a separate location and refresh it periodically rather than calculating it on every request.
• Use estimated counts: MongoDB’s estimatedDocumentCount() returns an approximate count much faster than an exact count.
• Avoid displaying totals: show “Next” and “Previous” buttons without revealing the total number of pages.

// Fast estimated count
$estimatedCount = DB::connection('mongodb')->collection('products')->raw(function ($collection) {
    return $collection->estimatedDocumentCount();
});

When Offset Pagination Still Makes Sense

Despite its limitations, offset pagination works well in certain situations:

• Small to medium datasets: Collections under 100,000 records rarely show noticeable performance issues.
• Admin panels: Internal tools where convenience outweighs performance concerns, and datasets are often filtered to manageable sizes.
• Arbitrary page access: When users need to jump directly to page 50 or page 200, offset pagination is the only practical option.

If your application fits these criteria, offset pagination’s simplicity makes it the right choice. The performance problems only arise when users navigate to deep pages in large collections.

Cursor Pagination: The Scalable Alternative

Cursor-based pagination takes a different approach. Instead of counting how many records to skip, you use a pointer to the last record you saw and ask for everything that comes after it. This technique is also known as keyset pagination or seek pagination.

What is Cursor-Based Pagination

The cursor is a value that uniquely identifies a position in your sorted result set. When you request the next page, you pass this cursor, and the database queries for documents where the sort field is greater than (or less than, depending on direction) the cursor value. Consider a timeline of posts sorted by creation date. Instead of saying “skip the first 100 posts,” you say “give me posts created after this timestamp.” The database can use an index to jump directly to that position without scanning earlier documents.

How Cursor Pagination Works

The flow typically follows this pattern:

1. First request: fetch the first N records and note the cursor value of the last record.
2. Subsequent requests: fetch N records where the sort field is greater than the cursor value.
3. Repeat: each response includes a new cursor for the next page.

The cursor is typically the _id field, a timestamp like created_at, or a compound value when sorting by non-unique fields.

// First page - no cursor needed
$products = Product::orderBy('_id', 'asc')->limit(20)->get();
$lastId = $products->last()->_id;

// Second page - use the cursor
$products = Product::where('_id', '>', $lastId)->orderBy('_id', 'asc')->limit(20)->get(); 

Because _id is always indexed in MongoDB, this query executes with consistent performance regardless of how deep into the dataset you’ve navigated.

Why Cursors are Efficient

The key difference is how the database executes the query. With a cursor condition like where(‘_id’, ‘>’, $lastId), MongoDB uses the index to jump directly to the starting point. There’s no scanning or discarding of documents. The query examines only the documents it returns. This gives cursor pagination O(1) time complexity for any “page.” Whether you’re fetching the equivalent of page 1 or page 10,000, the query takes the same amount of time.

Choosing the Right Pagination Method

When deciding between offset and cursor pagination, consider the following factors:

• Dataset Size: For small to medium datasets, offset pagination may suffice. For large datasets, cursor pagination is usually more efficient.
• User Experience: If users need to jump to arbitrary pages, offset pagination may be necessary. Cursor pagination is better for sequential browsing.
• Performance Needs: If performance is critical and you expect high traffic, cursor pagination is the better choice due to its efficiency.
• Implementation Complexity: Offset pagination is simpler to implement, while cursor pagination may require more careful handling of cursors.

By weighing these factors, you can choose the pagination method that best fits your application’s needs.

Frequently Asked Questions

Call To Action

Note: Choosing the right pagination method is crucial for maintaining performance and user experience in applications dealing with large datasets. Understanding the strengths and weaknesses of each method will help you make informed decisions for your project.