Paging

pmMDA must support data objects graphs (DOGs) which consists of many heavyweight data objects. The memory used to handle this objects may be bigger as the memory which is available on the client but not on the server. Furthermore, the DOGs are transferred over a network and pmMDA wants to keep the latency as small as possible. Thus pmMDA must define mechanisms to handle such huge DOGs. And this mechanism is called "paging".

Paging root objects

Paging is used for retrieving a huge amount of root data objects. Unlike paged indexed properties, root objects must not be configured as paged or not at design time. It is possible to load all root objects at once or to load them paged.
To retrieve the root objects paged use the IDataObjectHandler.RetrievePaged method. The returned collection is of type PagedReadOnlyCollection. It is impossible to add or remove items from that collection because root objects are created using its contructor and the IDataObjectHandler.Store(IDataObject) method and they are deleted using the IDataObjectHandler.Remove(IDataObject) method.
There are two possibilities to store the root objects. You can eighter store each modified root object using the IDataObjectHandler.Store(IDataObject) method or you can store the full PagedReadOnlyCollection using the IDataObjectHandler.Store(PagedReadOnlyCollection) method.

Paged indexed properties

Indexed properties can have a huge amount of items (up to one million items). Therefore they can be paged.

Specify paged indexed properties

Whether an indexed property is paged and the default paging data must be specified at design time. Some of the paging data is adjustable at runtime.

Design time

Whether an indexed property is paged is specified at design time by setting the tagged value dog-paged to true.
The collection type is specified at runtime using the tagged value dog-.net-collection-type.

Paged collection implementations:
Collection type Class name Indexed Ordered Allow duplicates
Set PagedSet No No No
Bag PagedBag Yes No Yes
List PagedList Yes Yes Yes

Indexed: The items are accessible by index.
Ordered: The order of the items is persistent.
Allow duplicates: The same item may be located multiple times in the same collection.

The default paging amount and the count of items that are hold in the local cache (maximum loaded items count) are also specified at design time using the tagged values dog-paging-amount and dog-paging-max-loaded-item-count. This data is only needed if the indexed property is paged.

Runtime

Wheter a collection is paged and the collection type cannot be changed at runtime.

Every data object gets two properties per paged indexed property. The name of the properties are %Name%PagingAmount and %Name%MaxLoadedItemCount where %Name% is the name of the property in plural. The initial value of these properties is specified at design time by using the the tagged values dog-paging-amount and dog-paging-max-loaded-item-count. The property values can be changed at runtime, but their values will not be used until items have to be reloaded.

Using paged indexed properties

Paging is done transparently to the developer who uses the data objects. The methods and properties which represent the indexed property (Add, Remove, Insert, Count, ...) are almost the same for unpaged and paged indexed propeties. The following table defines which methods are available.
Method or property Description Set PagedSet List and Bag PagedList and PagedBag
%Name%PagingAmount Gets the count of items that are reloaded when an unloaded item is requested. No Yes No Yes
%Name%MaxLoadedItemCount Gets the count of items that are hold in the local memory. No Yes No Yes
%Name%Count Gets the total count of items in the indexed property. Yes Yes Yes Yes
Contains%Name% Gets whether an item is located in the collection. Yes Yes Yes Yes
IDataObject[] %Name% Returns an array that contains all items of the indexed property. Only available if the property is readable. Yes Yes, but slowly Yes Yes, but slowly
IEnumerable Enumerate%Name% Allows to iterate over the items of the indexed property. Yes Yes Yes Yes
IEnumerable Enumerate%Name%(int firstIndex, int lastIndex) Allows to iterate over a range of items of the indexed property. No No No Yes
IEnumerable Enumerate%Name%(object first, object last) Allows to iterate over a range of items of the indexed property. No Yes No No
int IndexOf%Name% Returns the index of the first occurrence of a specified item. Only available if the property is readable. No No Yes Yes
Get%Name% Returns the item at a specified index. Only available if the property is readable. No No Yes Yes
Set%Name% Sets the item at a specified index. Only available if the property is writable. No No Yes Yes
Insert%Name% Inserts an item at the specified index. Only available if the property is writable. No No Yes Yes
RemoveAt%Name% Removes the item at the specified index. Only available if the property is writable. No No Yes Yes
Add%Name% Adds a collection of items to the indexed property. Only available if the property is writable. Yes Yes Yes Yes
Add%Name% Add one item to the end of this indexed property. Only available if the property is writable. Yes Yes Yes Yes
Removes%Name% Removes an item from the indexed property. Only available if the property is writable. Yes Yes Yes Yes
Clear%Name% Removes all items in the indexed property. Only available if the property is writable. Yes Yes Yes Yes

Accessing elements of indexed properties

Because paging is done transparently the usage of indexed properties is the same as if the properties would be unpaged. But accessing the items of paged indexed property may take a long time, so there are a few optimisation:

Sets

It's impossible to request an item of a paged set by its index. The only possibilities to get the items is iterating over all items using the foreach keyword or to request the items by specifying the first and/or last item of the iteration using one of the overloaded PagedSet.Enumerate methods.
If a first or last item is specified, the pmMDA framework doesn't need to reload all the items before the first or after the last item. This makes it possible for a GUI component to display a ragne of items which are located in the middle of a set.

Example:

PagedSet pagedSet /*=...*/;
foreach (IDataObject item in pagedSet) { /*...*/ }
foreach (IDataObject item in pagedSet.Enumerate(firstItem, lastItem)) { /*...*/ }

To iterate over the items of a paged indexed property of a data object which is represented through a PagedSet you can use the generated Enumerate%Name%() or Enumerate%Name%(object first, object last) methods. The %Name% is the name of the property in plural.

Example:

foreach (IDataObject item in myDataObject.EnumerateMyItems()) { /*...*/ }
foreach (IDataObject item in myDataObject.EnumerateMyItems(firstItem, lastItem)) { /*...*/ }

Lists and Bags

The difference between bags and lists is that the order of items in a list is persistent where items in a bag may have a different order after reloading. Making the indexes persistent is a job of the persistence handler. From a pure paging view bags and lists are the same. The PagedBag and PagedList derives from the class BaseListBase and do not define or implement any own method or property.

Paged lists (inclusive bags) access their items by index. An item at a specified index may be local or persistent. Items at local indexes were eighter added or inserted, or they replace another item at the specified index. If the item at the specified index is not local it is persistent. The accessing of items by index is done the same way as in unpaged properties. If the item at a specified index is requested, the paged list (or bag) may reload the next portion of items. Setting the item at a specified index is a local operation untill the owner of the indexed property is stored. Therefore setting an item does not access the network.

Examples:

PagedList and PagedBag

IDataObject item = pagedBag[13];
pagedBag[7] = newItem;
Indexed property
// For an indexed property with the name "Items":
IDataObject item = myDataObject.GetItem(13);
myDataObject.SetItem(7, newItem);

It is possible to iterate over all items of a paged list (or bag) or to iterate over a range of items. To iterate over a range of items the index of the first item and/or the index of the last item must be specified.

Examples:

PagedList and PagedBag

foreach (IDataObject item in pagedList) { /*...*/ }
foreach (IDataObject item in pagedList.Enumerate(10, 55)) { /*...*/ }
Indexed property
// For an indexed property with the name "Items":
foreach (IDataObject item in dataObject.EnumerateItems()) { /*...*/ }
foreach (IDataObject item in dataObject.EnumerateItems(10, 55)) { /*...*/ }
// Using the "Items" property will reload all items in small portions at once and then return
// an array which contains all items.

Modify items of paged collections

If a reference to an item is stored while enumerating a paged collection and the iteration continues then it is possible that the paged collection discards this item from the local cache. If this item is modified later then its changes will not be stored while the storange of the paged collection.
There are two possibilities to avoid that error. The first method is recommended.

If a data object is modified while iterating over the items of an indexed property or of a paged root object then nothing is special.

Example

foreach (IDataObject item in pagedCollection) {
   if (/*...*/) {
      item.Name = "New value";
   }
   // ...
}
// ...
Locator.DataObjectManager.DataObjectHandler.Store(pagedCollection);

Error handling

Data objects may be modified by another transaction. I.e. it is possible that another client modifies and stores the owner of the paged indexed property while the first client is waiting for a user input. If the first client needs to reload items after the modification, pmMDA framework recognise that optimistic concurrency vialoation.

Because the reloading is done transparently to the developer (i.e. if he is iterating over the items of a paged collection) it would be bad to throw an error that was not expected by the developer. Also it is possible that the first client never modifies the indexed property, and therefore the error should be ignored. If the first client plans to modify the data object it would be good to throw an exception as soon as possible so that the client can reload the data object before he is changing it. Thus the way pmMDA reacts must be configurable.

An optimistic concurrency vialoation may have different impacts to sets than to bags or lists. In sets, the current iteration aborts if the current item of the enumerator was removed from the paged indexed property before the end of the set is reached. In lists or bags the impact is even worse because the indexes at which items were replaced, inserted or removed may not match anymore with the persistent indexes.

Default error behaviour

Collection type Action Clear cache Discard modifications Description
Paged set Never throw an exception false false Reload errors are fully ignored. This means that the developer who iterates over the paged set will not recognise that the owner has been modified. Items in the local cache, which have been removed, will be iterated until they are thrown away because the "es;maximum loaded item count"es; is exceeded. Storing an owner which has been modified by another transaction will result in a PersistenceException.
Paged bags and lists Throw exception once true false A PagingException is thrown, the first time items are reloaded and the owner has been stored by another transaction. This means that the storage by the other transactions will only be reported once. Local cached items are removed to avoid that removed items are iterated and new items are omitted.

Specifying error behaviour

It is possible to adjust the default action or to specify the action every time an error occured. To adjust the default behaviour the DefaultOwnerModifiedAction property of a paged collection in the protected constructor of each data object type can be modified. To specify the action every time an error occured, you can register to the IPagedCollection.OwnerModified event in the same constructor. Change the property values of the parameters provided by the event method to adjust the action.
Currently it is only possible to adjust the action within the data objects. It is planned that users of data objects may also adjust the optimistic concurrency violation action.
Example:

[Serializable]
public class PagedListOwner : IDataObject, ISerializable
{
  // ...
  protected PagedListOwner(SerializationInfo info, StreamingContext context)
  {
    m_id = info.GetInt32("m_id");
    m_timestamp = info.GetDateTime("m_timestamp");
    m_modified = info.GetBoolean("m_modified");
    m_update = new DataObjectUpdate(this);
    m_itemsPagingAmount = 10;
    m_itemsMaxLoadedItemCount = 50;
    m_items = new PagedList(this, typeof(PagedItem), "m_items", ItemsPagingAmount, ItemsMaxLoadedItemCount);
    /* %% USER DEFINED INITIALIZATION START %% */
    // adjust default error action
    IPagedCollection items = (IPagedCollection) m_items;
    items.DefaultOwnerModifiedAction.ClearCache = true;
    items.DefaultOwnerModifiedAction.DiscardModifications = true;
    items.DefaultOwnerModifiedAction.ExAction = ExceptionAction.AlwaysThrowException;

    // register to the OwnerModified event to be able to specify the error action every time
    // items are reloaded and an optimistic concurrence violation is detected.
    items.OwnerModified += new OwnerModifiedEventHandler(m_items_OwnerModified);
    /* %% USER DEFINED INITIALIZATION END   %% */
  }

  // ...

  private void items_OwnerModified(object sender, OwnerModifiedEventArgs e)
  {
    e.Action.DiscardModifications = false;
  }

}