Automation arrays

Automation has a good support for arrays. Arrays can be used to pass large amount of data to and from automation servers. This data can be anything from very flexible structures holding diverse data to a very fast way to pass binary data. Automation arrays are self descriptive, beside the data itself they house information on the array size and dimensions as well on the element size.

Safearrays

Automation arrays are called safearrays. Safe because they themselves have the knowledge of their dimensions, so they provide a protection against "out of bounds" errors. They are also safe because they have a knowledge of the size of the elements, getting or setting an item will address the correct range of memory.

In the Delphi VCL class library safearrays will be wrapped up as Variant arrays. These variantarrays are compatible with an OleVariant. The VCL unit variants.pas houses a couple of routines and types to work with variantarrays.

A variant array is declared as record.

TVarArray = packed record
   DimCount: Word;
   Flags: Word;
   ElementSize: Integer;
   LockCount: Integer;
   Data: Pointer;
   Bounds: TVarArrayBoundArray;
end;

You can see how the different elements of the record describe the array: a count of the number of dimensions, a flag to work with custom variants, the size of an individual array element, the number of locks on the array (for efficient use), a pointer to the array's data and an array describing the lower and upper bounds of all the dimensions of the array. You can see here that the description of the array and the actual data itself are separated. We will meet most of these in the story to come.

The elements of a variant array can be almost any type, as I am covering automation I will limit the types used to automation compatible types. OleVariant is among these types. A variant array is compatible with an OleVariant so I can construct another variant array and put that in the item. Repeating the process could lead to ragged treelike structured. You might wander where the elementsize fits in in this scenario. As the items are arrays themselves there seems no longer to be a fixed item size. Let's take a sidestep and look at the variant type itself in some more detail.

The Variant type

Delphi variants can be very complicated types, in Delphi 6 it is even possible to construct custom variant types. The entire VCL unit variants.pas is dedicated to them. The OleVariant in the narrower, automation sense is described in System.pas. It is also a record

PVarData = ^TVarData;
{$EXTERNALSYM PVarData}
TVarData = packed record
   VType: TVarType;
   case Integer of
           0: (Reserved1: Word;
                case Integer of
                       0: (Reserved2, Reserved3: Word;
                           case Integer of
                           varSmallInt: (VSmallInt: SmallInt);
                           varInteger: (VInteger: Integer);
                           varSingle: (VSingle: Single);
                           varDouble: (VDouble: Double);
                           varCurrency: (VCurrency: Currency);
                           varDate: (VDate: TDateTime);
                           varOleStr: (VOleStr: PWideChar);
                           varDispatch: (VDispatch: Pointer);
                           varError: (VError: LongWord);
                           varBoolean: (VBoolean: WordBool);
                           varUnknown: (VUnknown: Pointer);
                           varShortInt: (VShortInt: ShortInt);
                           varByte: (VByte: Byte);
                           varWord: (VWord: Word);
                           varLongWord: (VLongWord: LongWord);
                           varInt64: (VInt64: Int64);
                           varString: (VString: Pointer);
                           varAny: (VAny: Pointer);
                           varArray: (VArray: PVarArray);
                           varByRef: (VPointer: Pointer);
                           );
                      1: (VLongs: array[0..2] of LongInt);
                      );
          2: (VWords: array [0..6] of Word);
          3: (VBytes: array [0..13] of Byte);
end;
{$EXTERNALSYM TVarData}

This declaration and the values of the constants tells us some things:

A variant can be typecasted to this tVarData type and can be used for testing the actual content of the variant.

if tVarData(MyVariant).VType <> varString then
   ShowMessage( 'This variant does not hold a string !' );

When querying variants for their type you will meet Vtypes that are unknown to the TvarData type. A Delphi VariantArray has a vType of $200C. This is the varArray bit $2000 and the vType $C. This does stand for varVariant, which is omitted in tVarData. This is where the Delphi Variant gets separated from the Windows variant.

Another thing you can do with tVarData is casting the variant's value to a certain type :

if tVarData(MyVariant).VType = varDate then
   ShowMessage('DateTimeToStr( tVarData(MyVariant).VDate ) );

Now we have seen some of the internals of a variant I will return to the variant array.

Creating a ragged variant array

In the filemanager example I will add a new property GroupedByName to the manager. This property will hold a list of files grouped by filename, excluding the extension. These grouped names will be returned in a variant array, each item of the array holds a subarray. The first item of the subarray will hold the filename, the next items the extensions found for this filename.

In the typelibrary editor I will create a new read only property  GroupedByName, of type variant. In the property getter the array is created and filled. 

I will need a helper function to remove the extension from the filename :

function BlankFileName(InString : string): string;
   var Ext : string;
   begin
   Ext:= ExtractFileExt(InString);
   result:= Copy(InString, 1, Length(Instring) - Length(Ext));
   end;

To build the array will take the following steps

This can be done with the following code:

function TFileZapper.Get_GroupedByName: OleVariant;
var i, j : integer;
     TempArray : Variant;
   begin
   result:= VarArrayCreate( [0, Form1.FileListBox1.Items.Count], varVariant );

VarArraycreate creates as new safearray with one dimension, ranging from  0 to the number of files in the FileListbox. The second parameter indicates that all the items in the array will be variants.

   for i:= 0 to Form1.FileListBox1.Items.Count -1 do
      begin
      j:= -1;
      Found:= False;
      while ( j < VarArrayHighBound(result,1) ) and not Found do

I will scan first dimension of the result-array, its upperbound is returned by the VarArrayHighbound function .

         begin
         inc(j);
         TempArray:= result[ j ];
         Found:= ( VarArrayDimCount( TempArray ) = 1 ) and
            ( CompareText( TempArray[0], BlankFileName( Form1.FileListBox1.Items[i]) ) = 0 );
         end;

The array item to be tested and potentially altered is copied to the private variant TempArray. Using  VarArrayDimCount I can check the number of dimensions, which will be 0 in an uninitialized element. When the element is initialized the filename will be in element 0.

         if not Found then
            begin
            TempArray:= VarArrayCreate( [0, 1], VarOleStr );
            TempArray[ 0 ]:= BlankFileName( Form1.FileListBox1.Items[i] );
            j:= 0;
            while VarArrayDimCount( result[ j ]) > 0 do
               inc(j);
            end;

When not found, the new filename will be added to the result.  VarArrayCreate will create the new subarray. This time I pass varOLEstr to the function, all items in the array will be a (Wide)string.  To find the index of the first uninitialized array item, I will scan the result array with VarArrayDimCount. All initialized items are arrays with (at least) one dimension and will be skipped. The new item will be added after the last item containing data.

Now I will add the extension of the file to the subarray.

         VarArrayRedim( TempArray, VarArrayHighBound(TempArray, 1)  +  1 );
         TempArray[ VarArrayHighBound(TempArray, 1) ]:= ExtractFileExt( Form1.FileListBox1.Items[i] );

I will increase the size of the subarray to the currenct VarArrayHighBound plus one. VarArrayRedim will resize the subarray to this length. The file-extension is copied into the new item and to conclude the TempArray will be copied into the result array again.

        result[ j ]:= TempArray;
end;

Rests cleaning up. There is a big chance that some of the items in result are unused. The number of items was initialized as the number of filenames. If a filename occurs more than once, with just a different extension, then there will be some empty items left.

j:= VarArrayHighBound(result,1);
if j > 0 then
   begin
   while VarArrayDimCount( result[j] ) = 0 do
       dec( j );
   VarArrayRedim( result, j );
   end;

By checking the element for zero dimensions I can remove the items by resizing the result to a smaller size, using VarArrayRedim again.

Using the ragged array

Word would be a good user of the variant array. In a macro I will use the property to create a report on all files grouped by name. Before I can work with a variant array I will first have to copy the array to a local variant as VBA will interpret MyManager.GroupedByName(1) as the groupedByName being an indexed property.

Sub ListRaggedArray()
   Dim AllFiles As Variant
   Dim FileGroup As Variant
   Selection.EndKey Unit:= wdStory
   AllFiles = MyManager.GroupedByName
   For i = 0 To UBound( AllFiles )
         FileGroup = AllFiles( i )
         For j = 0 To UBound( FileGroup )
               Selection.InsertAfter ( FileGroup( j ) & vbTab )
         Next
         Selection.InsertParagraphAfter
   Next
   Selection.EndKey Unit:= wdLine
   Selection.TypeParagraph
End Sub

The Groupedbyname property is copied to the local variant AllFiles. The VBA function Ubound is the VBA counterpart of VarArrayHighBound, it will return the number of elements in the array. Again the subarray property is first copied to the FileGroup local variant. And the items of FileGroup can be inserted into the document.

Evaluating the variantarray

Working with a variant array we have seen a couple of things

This approach to safearrays is very flexible but of course there is a lot of overhead while resizing and copying. The coding sample would have been a lot faster if the data was pre-processed first so that all arrays could be created an filled in one go. I did it this way to show as much of the of the array manipulating functions and their possibilities as possible.

Creating an integer safearray

Now I will dive deeper into the safearray api to create a fast and efficient safe array. The VCL functions all work with VariantArrays and house in the Delphi variants unit. In the ActiveX API there are more safearray handling routines. Let's take a look.

In ActiveX.pas you will find this declaration of a safearray

PSafeArray = ^TSafeArray;
{$EXTERNALSYM tagSAFEARRAY}
tagSAFEARRAY = record
   cDims: Word;
   fFeatures: Word;
   cbElements: Longint;
   cLocks: Longint;
   pvData: Pointer;
   rgsabound: array[0..0] of TSafeArrayBound;
end;
TSafeArray = tagSAFEARRAY;
{$EXTERNALSYM SAFEARRAY}
SAFEARRAY = TSafeArray;

Only the names differ from the variant safearray we met before. A number of API functions work with these arrays. I will use these to create a typed safearray.

In the typelibrary editor I will create a property FileSizes, as type I will choose SAFEARRAY(long).  Which describes an array of (long) integers.

 The property will hold an array with the sizes of all files currently in the FileListbox. VBA (in Word) will recognize the FileSizes property being an array of long (integers). So SAFEARRAY is an automatable property type.

The filemanager implementation in Delphi will have to create the safearray. The safearray is accessed  via a PSafeArray, a pointer. This will be the result of property getter. In the getter I will create the safearray and allocate it's data. I will fill a record with the array bounds and I can create the array. After accessing the data I will fill its items with file-sizes.

function TFileZapper.Get_FileSizes: PSafeArray;
   var ArrayBounds : TSafeArrayBound;
   i : integer;
   ArrayData : pointer;
   F : tSearchRec;
   fFileSizes : PSafeArray;
   type
      IntegerArray = Array of integer;

   begin
   ArrayBounds.lLbound:=  0;
   ArrayBounds.cElements:=  Form1.FileListBox1.Items.Count;
   fFileSizes:= SafeArrayCreate( varInteger, 1, ArrayBounds );
   if SafeArrayAccessData( fFileSizes, ArrayData ) = S_OK then
      begin
      for i:= 0 to Form1.FileListBox1.Items.Count - 1 do
            if FindFirst( Form1.FileListBox1.Items[i], faAnyFile, F) = 0 then
               begin
               IntegerArray(ArrayData) [i]:= F.Size;
               FindClose(F);
               end;
      SafeArrayUnAccessData(fFileSizes);
      end;
    result:= fFileSizes;
   end;

The record holding the arrays dimensions, tSafeArrayBound is declared in ActiveX.Pas. The lower bound is set to 0 and the upper to the number of files in the listbox. SafeArrayCreate will create an array of integers. The data itself is accessed through SafeArrayAccessData.

Working with the variant array I did a lot on resizing and reshaping of the array. The ability to resize an array makes life hard on other pieces of code which are using the array. To speed up access safearrays can be locked. A locked array cannot be resized or reshaped.  SafeArrayAccessData places a lock on the array and provides a pointer to the data. I can work with this data as an array of integer if I typecast it to the proper type.  The IntegerArray type will do just that, now I can fill the array with file sizes. SafeArrayUnAccessData frees the lock. A safearray keeps a count of the number of locks placed, when the number of locks has sunken to 0 the array can be reshaped or destroyed.

VBA did recognize the property and I can use it just like the variant array. After copying the array to a local variant again it can be scanned and its items inserted into the document.

Sub ListSizes()
   Dim VarSizes As Variant
   Selection.EndKey Unit:=wdStory
   VarSizes = MyManager.FileSizes
   For i = 0 To UBound(VarSizes)
         Selection.InsertAfter (VarSizes(i))
         Selection.InsertParagraphAfter
   Next
   Selection.EndKey Unit:=wdLine
   Selection.TypeParagraph
End Sub

SafeArray versus VariantArray

With a VCL VariantArray you can do almost anything you can do with a safearray. Locking and reshaping function are available to both. The big difference between a variant- and a safe- array is that a variantarray is to COM of type OleVariant and cannot be used to implement an automation property of type SAFEARRAY. Properties of type SAFEARRAY can be recognized by an automation client as being an array of a specific type. Variant arrays are published as VARIANT in the typelibraray. As a variant can be almost anything the client will have to inspect the property to find out if it is an array and what type of items it holds. 

SafeArrays have as a second advantage that they do not need the variants unit. Some people find it be desirable to keep that unit out of their application.

Passing binary data

The locked integer safearray works a lot faster than the variantarray. Safearrays can be used to pass large amounts of data really fast. If you want to pass any binary data from an automation clients to automation servers you should work with a locked safearray of bytes. Any binary object, like an image or a document can be serialized to an array of bytes. The size of a safearray can be over 64K, so it is possible to pass a huge amount in one go. Passed bytes do not need any translation.  With multibyte values you have to decide on the ordering of the upper and lower byte.

Arrays in VBscript

VBscript does know how to work with arrays. Arrays can be declared and resized or even be created in one go with the Array() function. When working with array results VBA did have some trouble with the distinction between an array and an indexed property. You can also create automation objects, like the FileManager, in a script. VBscript works fine with the object but mistakes all array properties for indexed properties and will report error $800A000D "Type mismatch". That is, the type is unknown to VBScript. As a workaround indexed properties can be a full functional replacement for arrays and are easy to implement. In many situations a collection will be sufficient.

Where are we ?

The automation API provides with a number of structures and functions to work with array structures. These have been designed and implemented in such a manner that they are safe to use. Besides that they are selfdescribing, trying to access an element outside the range will not lead to a serious crash.

Arrays can be very flexible, when it comes to more than one dimension they are even more flexible than collections. Arrays can also be a very fast and efficient manner to pass (large amounts of) data. On the other hand VBA and VBscript do not always know how to handle an array the right way.

The Delphi VCL provides a couple of wrapper functions to work with safearrays of variant. For a better understanding of these some insight in the internals of a variant can be very enlightening. Working directly with the automation safearray API requires a little more knowledge. 

What's next