Re-creating collection-repeat: an improved ng-repeat for large sets

Following the brainwave setforth by my previous post on the directive ng-repeat, below are two examples of a re-creation of the collection-repeat directive authored by the Ionic team.

Our custom module is aptly and very boring-ly named collect-repeat and it optomizes ng-repeat by only rendering the number of items that are in view and rendering the rest as needed while scrolling...particulary useful for massive data sets.

It's a great concept created by the Ionic team and a terrific exercise for developers. Below are the two nearly identical implementations of collect-repeat (our custom module) save for the difference that the former contains 100 items in the collection, whereas the latter contains 1,000,000 items.

With 100 items...

With 1 million items...

As you might have guessed (maybe), they perform nearly identically.


Some Considerations

collect-repeat does not holistically take into consideration all edge cases.

  • Since it was developed as a concept piece, it only takes into account one type of presentation, one with elements stacked from top to bottom and with a width of 100% relative to the window. It will not function as expected for those wishing to stack items from left to right. This directive does not cover all edge cases but it does stand at 275 lines as compared to just over 900 lines for collection-repeat :)

  • The directive relies on the $ionicPosition module, which measures the position of a div relative to its parent, and that a div named .scroll-content be a parent of the directive. .scroll-content is a div rendered onto the DOM by Ionic and represents the scrollable portion of the view, excluding any headers or footers. With that said, both the module and div could be more generally abstracted to accommodate also browser based angular applications, not only hybrid apps built with Ionic.

  • It positions all items absolutely. All items are given the CSS property position: absolute and translated to its proper position using transform. Therefore, any items positioned to be static, relative, or fixed will be changed to absolute. This is similar to how Ionic achieves collection-repeat and it's done so that every item has a defined position in space to ensure accurate scrolling. If items were positioned statically as they are by default, the items would cascade up as they are removed when the user scrolls down.

  • Gutter margins and manual changes to width and height are not incorporated into the directive. Ionic's collection-repeat provides additional attributes such as item-render-buffer, item-width, and item-height to aide with the customization of those values, but this directive doesn't :(

    But wait...you can add it yourself! As an exercise!


How it works?

The module is divided into two components, $repeatFactory (a service where helper functions are defined) and collectRepeat (a directive where most of the logic lives).

The most important takeaway from the factory is the repeatManager method. It is the contructor that is instantiated at the beginning of the link function of the directive and is responsible for maintaining the state of the scrolling view.

On creating a new instance of repeatManager, it's also necessary to generate a new object (or map) to provide as reference for all the items that have already been transcluded, rendered, and whose scope has been updated. The enclosing parent element must also be registered so that we can later change it's height to the height of all the items in the collection.

Next, a watch is set on the collection with an anonymous function set to be invoked whenever the collection changes. This is where the bulk of the logic resides. It's split into three steps.

  1. Transclude and render the first element of the collection so that the height and other properties of each individual item may be registered, the height of the parent div adjusted, and the size of the viewport calculated.

  2. With the size of the viewport determined, loop over and render the n number of items that will fit inside the viewport (plus a few extra for smoothness).

  3. Set a 'scroll' event listener on .scroll-content to return the scrollHeight of the first element of the collection and render only the necessary elements.

There are a few additional tidbits (also known as comments) available in the code below. It should help alleviate some confusion if you have trouble following the above synopsis. The module is also available on github.

Enjoy!

GitHub

The Code

index.html

Pretty standard markup as standardized by ng-repeat.

<div class="list">  
  <div class="item" collect-repeat="item in collection">{{ $index }}</div>
</div>
app.js

Here, a new collection with 1 million items is attached to the controller's scope.

angular.module('starter', ['ionic'])

.controller('MainCtrl', function($scope) {

  var newArr = [];
  for(var i = 0; i < 1000000; i++) {
    newArr.push(i);
  }

  $scope.collection = newArr;

})
ionic.collect.repeat.js

It's long but I've included comments where I found necessary. This includes both the factory and directive.

angular.module('collectRepeat', [])

.factory('$repeatFactory', ['$animate', '$ionicPosition', function($animate,$ionicPosition){

  var calculateParentHeight = function(collectionLength, nodeHeight) {
    return (collectionLength + 1) * nodeHeight;
  }

  var calculateInViewCount = function(windowHeight, nodeHeight, extra) {
    return Math.ceil((windowHeight / nodeHeight) + extra);
  }

  var calculateScrollHeight = function(inViewCount, nodeHeight) {
    return inViewCount * nodeHeight;
  }

  var calculateEndIndex = function(collectionLength, inViewCount) {
    return (collectionLength < inViewCount ? collectionLength : inViewCount) - 1;
  }

  var RepeatManager = function(collection, map){
    this.map = map;
    this.collection = collection;
    this.startIndex = 0;
    this.endIndex = null;
    this.lowerThreshold = null;
    this.nodeHeight = null;
    this.inViewCount = null;
    this.scrollHeight = null;
    this.parentElement = null;
    this.parentHeight = null;
  }

  RepeatManager.prototype = {
    setDefaults: function(clone, windowHeight) {
      // Set the view height
      this.windowHeight = windowHeight;
      // Set the height for individual nodes
      this.nodeHeight = Math.round($ionicPosition.offset(clone).height);
      // Set the height of the enclosing parent to sums of all the nodes
      this.parentHeight = calculateParentHeight(this.collection.length, this.nodeHeight);
      // Set the # of clones to be rendered at any given time
      this.inViewCount = calculateInViewCount(this.windowHeight, this.nodeHeight, 4);
      // Set the comparative scroll height of all the rendered divs
      this.scrollHeight = calculateScrollHeight(this.inViewCount, this.nodeHeight);
      // Set the endIndex: either the collection or inViewCount, whichever is greater
      this.endIndex = calculateEndIndex(this.collection.length, this.inViewCount);
      // Set the lowerThreshold so we know when to render a new node below the fold
      this.lowerThreshold = this.nodeHeight;
    },
    registerNode: function(node) {
      this.map[node.index] = node;
    },
    isNodeRegistered: function(node) {
      return !typeof node === 'undefined';
    },
    isBelowLowerThreshold: function(scrollHeight) {
      return scrollHeight <= -(this.nodeHeight*2);
    },
    isAboveLowerThreshold: function(scrollHeight){
      return scrollHeight >= -(this.nodeHeight*2);
    },
    isAtEndOfArray: function() {
      return this.endIndex < this.collection.length;
    }
  }

  return {
    RepeatManager: RepeatManager,
    createMap: function() {
      return Object.create(null);
    },
    updateScope: function(obj) {
      var scope = obj.scope;

      scope[obj.valueIdentifier] = obj.value;
      scope.$index = obj.index;
      scope.$first = (obj.index === 0);
      scope.$last = (obj.index === (obj.arrayLength - 1));
      scope.$middle = !(scope.$first || scope.$last);
      scope.$odd = !(scope.$even = (obj.index&1) === 0);
    },
    renderNode: function(node, parent, previous) {
      $animate.enter(node, parent, previous);
    },
    removeNode: function(node) {
      $animate.leave(node);
    },
    styleClone: function(clone, x) {
      clone.style.position = 'absolute';
      clone.style.width = '100%';
      clone.style.transform = clone.style.webkitTransform = 'translate3d(0,' + (x + 'px') + ',0)';
    },
    styleParent: function(parent, height) {
      parent.style.position = 'relative';
      parent.style.height = height + 'px';
    },
    transcludeClone: function(clone, scope, Manager, index, valueIdentifier, value, key, collection, previousNode) {

      // Position and translate the node to it's proper vertical position
      this.styleClone(clone[0], Manager.nodeHeight*index);

      // Render the cloned directive onto the DOM
      this.renderNode(clone, null, angular.element(previousNode));

      // Set the previousNode to this clone (used for next node)
      previousNode = clone;

      // Update this clone's scope
      this.updateScope({
        scope: scope,
        index: index,
        valueIdentifier: valueIdentifier,
        value: value,
        key: key,
        collectionLength: collection.length
      });

      // Register the node with the Manager
      Manager.registerNode({
        index: index,
        value: collection[key],
        clone: clone,
        scope: scope,
        previousNode: (index) ? previousNode : null
      });
    }
  }

}])


.directive('collectRepeat', ['$repeatFactory', '$ionicPosition', function($repeatFactory, $ionicPosition) {

  return {
    restrict: 'A',
    priority: 1000,
    transclude: 'element',
    compile: function($element, $attrs) {

      var expression = $attrs.collectRepeat;

      var match = expression.match(/^\s*([\s\S]+?)\s+in\s+([\s\S]+?)?\s*$/);

      var valueIdentifier = match[1];
      var collection = match[2];

      return function($scope, $element, $attr, ctrl, $transclude) {

        // Create a new object (map) to help keep track of which nodes have been transcluded
        var newMap = $repeatFactory.createMap();

        // Create a new instance of RepeatManager (the object that maintains the state of our repeat)
        var Manager = new $repeatFactory.RepeatManager($scope.collection, newMap);

        // Register the parent element
        Manager.parentElement = $element[0].parentElement;

      // ********* WATCH COLLECTION PROCEDURE *********
      //  Every time the collection changes, this block will be executed.
      //    1. Render the first element to get dimensions for the Manager
      //    2. Loop and render only those elements that are in view
      //    3. Set a 'scroll' event listener to render and remove depending on view

        $scope.$watchCollection(collection, function(collection) {

          var index, length, previousNode = $element[0], collectionLength, key, value, collectionLength = collection.length;

          // Query the 'scroll-content' div (specific to ionic) and set windowHeight in Manager
          var content = document.getElementsByClassName('scroll-content');
          var windowHeight = content[0].clientHeight;

      // ********* 1. *********
      // Render the first element to get dimensions for the Manager

          // Transclude the first element of the collection
          $transclude(function(clone, scope) {

            index = key = 0;
            value = collection[key];

            // Render the clone
            $repeatFactory.transcludeClone(clone, scope, Manager, index, valueIdentifier, value, key, collection, previousNode);

            // Set the default values for the Manager
            Manager.setDefaults(clone, windowHeight);

          });

          // Set it's position to relative and change it's height to accommodate all the nodes
          $repeatFactory.styleParent(Manager.parentElement, Manager.parentHeight);

      // ********* 2. *********
      // Loop and render only those elements that are in view

          // Loop through all the nodes that will be inView
          for (index = (Manager.startIndex + 1); index < Manager.inViewCount; index++) {
            key = index;
            value = collection[key];

            // Transclude the node
            $transclude(function(clone, scope) {

              // Render the clone
              $repeatFactory.transcludeClone(clone, scope, Manager, index, valueIdentifier, value, key, collection, previousNode);

            });

          }

      // ********* 3. *********
      // Set a 'scroll' event listener to render and remove depending on view

          $scope.currentScroll = 0;

          // Create listener on scroll event
          // On Scroll, return the height of the top most node in view
          angular.element(content[0]).on('scroll', function(e) {
            $scope.currentScroll = $ionicPosition.offset(Manager.map[Manager.startIndex].clone).top;
            $scope.$apply();
          });

          // Set watch on $scope.currentScroll and return scrollHeight
          $scope.$watch('currentScroll', function(scrollHeight) {

            // Is scrollHeight below the predetermined threshold and are there still elements to be shown?
            if(Manager.isBelowLowerThreshold(scrollHeight) && Manager.isAtEndOfArray()) {

              key = index = ++Manager.endIndex;
              value = collection[key];

              // Has the node already been registered with the Manager?
              if(Manager.isNodeRegistered(key)) {

                // Render the registered node
                $repeatFactory.renderNode(Manager.map[Manager.endIndex].clone, Manager.parentElement, null);

              } else {

                // Register and render a new clone
                $transclude(function(clone, scope) {

                  $repeatFactory.transcludeClone(clone, scope, Manager, index, valueIdentifier, value, key, collection, previousNode);

                });

              }

              // Is the view set to it's top most position?
              if(Manager.startIndex === 0) return;

              // Remove the top node
              $repeatFactory.removeNode(Manager.map[Manager.startIndex].previousNode);


            } else if(Manager.isAboveLowerThreshold(scrollHeight)) {

              // Is the view set to it's top most position?
              if(Manager.startIndex === 0) return;

              // Render the top node
              $repeatFactory.renderNode(Manager.map[Manager.startIndex--].previousNode, null, $element[0]);
              // Remove the bottom node
              $repeatFactory.removeNode(Manager.map[Manager.endIndex--].clone);

            }
          });

        });

      }
    }
  }

}])