How to do run-time (or explicit) linking of C++ plug-in components and objects

Introduction

How can we create a program that will beable to work with objects that do not even exist at the time when ourexecutable is conceived? Would it not be nice if we could design a program,which functionality we would be able to extend without altering the originalexecutable’s source (and even without recompiling): an application where we canplug-in additional functionality on the fly.

This is possible if we use the proposeddevelopment scheme of working with plug-in objects. What we need is a frameworkthat works for predefined base objects, but that is able to load DLLscontaining (new) child objects that extend, alter or specialize the behaviour.It is even possible to manipulate different generations of these objects: Theyco-exist peacefully together.

One of the difficulties of working withDLLs and allocating and de-allocating (and thus exchanging) memory is the waythe heap manager works. Unless one uses GlobalAlloc, it is imperative that allthe memory allocated in a particular DLL, is also deleted or de-allocated bythe same DLL (read 'the same heap manager' instead of 'the same DLL'). If onedoes not follow this rule, there is the distinct possibility that one will tryto delete memory that is unknown by the heap manager of the ‘called’ DLL. Thesebugs are fatal and very difficult to find!

Let say you created an object 'A' insidethe DLL called 'A-DLL'. You pass this object to the executable ('Bogus.exe') oranother DLL called 'B-DLL' and you try to de-allocate the memory associatedwith Object 'A'. The heap manager of 'Bogus.exe' or 'B-DLL' does not know aboutthe existence of the memory associated with object 'A' and could thus deleteinvalid memory. If you would use the GlobalAlloc function and sorts, you do notuse the heap managers, but instead memory is directly obtained from theoperating system. This approach certainly works, but IMHO is very ugly andhardly C++.

What is proposed here is a scheme for C++ Objects.

It is better to enforce the rule"de-allocate where you allocated it". That's where the DLLProxy Classcomes into the picture. Only one instance of this DLLProxy exists per DLL. Andthis instance is responsible for allocation and de-allocation of our (plug-in)objects in the DLLs. In fact, these objects, handle the memory management forobjects created inside a DLL.

When the application is designed, one mustof course know what kind of functionality is going to be extended in thefuture: communication protocols, MFC views or documents, windows, differentlanguages for code generation, support future hardware equipment or newalgorithms… Almost everything is possible if the function class interface isproperly defined.

The behaviour will be altered by thevirtual function mechanism later on. In the example, you can see that thefunction DoSomething() is called, but what actually happens inside thisfunction is determined by the implementation in the Plug_Object_Child class.The Plug_Object_Child class is defined inside the Plug-in DLL and was allocatedby a particular DLLProxyChild Object. This DLLProxyChild Object is aspecialization of the DLLProxy class because it has to know whichPlug_Object(_Child) objects to allocate and de-allocate!

Explicit linking is required, because theplug-in enabled executable is not even aware of the existence of the particularplug-in DLLs at compile and link time.

What kind of objects?

When we design our application, we must of course know what kind offunctionality we are going to extend in the future. Do we want toextend communication protocols, MFC views or documents, windows,different languages for code generation, support future hardwareequipment or new algorithms… you name it. Almost everything ispossible.

We then create a virtual base class for our to-be-extendedfunctionality. Our application will always work with this classdefinition. The behavior will be altered by the virtual functionmechanism later on.In our example, we will call DoSomething(), but what actually happensinside this function is determined by the implementation in thePlug_Object_Child class. The Plug_Object_Child class is defined insidea DLL and was allocated by a particular DLLProxyChild Object. ThisDLLProxyChild Object is a specialization of the DLLProxy class becauseit has to know which Plug_Object(_Child) objects to allocate anddeallocate!

This is a simple explanation of how it works. Sounds simple foranyone familiar with DLLs and the virtual function mechanism, does itnot?Of course we need EXPLICIT linking, because we do not know whether theDLLs with extended functionality exist (or how many there exist in thefuture) when building our plug-in enabled application.

The Procedure

What we do is quite simple.

1. First, one searches a particular directoryfor the existence of DLLs, then their usability is determined: the DLLs mustcontain the necessary exported functions to obtain a DLLProxy Object. (In otherwords, it is a check whether the DLLs are really plug-in DLLs)
2. Next, the DLLs are mapped in theexecutable’s address space (Load the library) and
3. the function pointers to the requiredfunctions are stored. (Obtained by calling GetProcAddress)
4. These FARPROC function pointers can becaste to the correct function prototype, so that these functions can be calledto obtain the DLLProxy object.
5. This DLLProxy Object is used to create anddelete Plug_Object instances.
6. In the end, when it is certain that no moreDLLProxy objects exist, the DLLs can be mapped out (FreeLibrary).
7. The DLLRTLoader (DLL Run-time Loader) classautomated this functionality.
8. However, do not forget to define GetDLLProxyfunction in every Plug-in DLL. It is a function with the following prototype:
   Collapse

   typedef DLLProxy* (WINAPI *GETDLLPROXY)(void);
   

   It is exported through the DLLs .def file when building the DLL project.

The Classes and their Interfaces

Collapse

// Our DLL Run-Time  Loader looks like this:

// This class is used for searching out the DLLs we can use in a particular directory.

// We can map in or map out the DLLs dependent on whether we need them or not

// Functions obtained from the DLLs can be called through this class.

class DLLRTLoader  
{
public:
DLLRTLoader();
virtual ~DLLRTLoader();

bool LookForDlls(const char* searchdirectory, bool recursive = true);
void FlushUnusedDLLs(void);

// necessary func names

void AddFuncName(const char* theFN);
FileFinder* GetFileFinder(void);
DLLFileEntry* SearchDLLFileEntry(const char* completepath);
DynamicSortedArray<DLLFileEntry*>* GetMappedDLLList(void); // TOO list not elements

DynamicSortedArray<DLLFileEntry*>* GetDLLList(void); // TOO list not elements


unsigned  GetNumberOfUsableDlls(void);
unsigned  GetNumberOfMappedDLLs(void);

void Debug(ostream& theStream);
protected:
virtual void CheckForDLLFuncs(void);
FileFinder myFileFinder;
DynamicSortedArray<DLLFileEntry*> myDLLs;
DynamicArray<char*> myFuncNames;
};

 

Collapse

// This class is used to implement some kind of reference counting mechanism on the usage

// of the DLLs. If an object is created from a particular DLL, its DLL usage reference count will be 

// incremented and that is done by manipulating these objects

class DLLFileEntry : public FileEntry
{
public:
DLLFileEntry(const char* path);
DLLFileEntry(const FileEntry&);
virtual ~DLLFileEntry();

inline void IncDllUsage(void) // increment Usage

{
_ASSERT(myMapped_In == true);
myRefCount++;
};

inline void DecDllUsage(void) // decrement Usage

{
_ASSERT(myMapped_In == true);
_ASSERT(myRefCount > 0);
myRefCount--;
};

inline HINSTANCE GetModuleHandle(void) // returns unsafe lib handle

{
return myLibHandle;
}

unsigned  GetRefCount(void);
void SetRefCount(unsigned  ref);

void SetProxy(DLLProxy* theProxy);
DLLProxy* GetProxy(void);

bool GetDllOK(void);

bool IsMappedIn(void); // returns whether mapped in or not

virtual  MapIn(void); // only allowed if not mapped in and usage count == 0

virtual bool CheckAndStoreFuncs(void); // checks and stores the function names

virtual  MapOut(void); // only allowed if mapped in and usage count == 0

virtual FARPROC GetFuncAddress(const char* funcname); 
// Get the FARPROC function address

void SetFuncNameList(DynamicArray<char*>* theFuncNames);

void Debug(ostream& theStream);
protected:
DLLFileEntry(const DLLFileEntry&); // do not allow copy

DLLFileEntry& operator=(const DLLFileEntry&); // do not allow copy


void SetDllOK(bool value);

unsigned  myRefCount;
bool myMapped_In;
HINSTANCE myLibHandle;
Tree<FARPROC> myFuncAddresses;
DynamicArray<char*>* myFuncNames;
private:
bool myDLLisOK;
DLLProxy* myProxy;
};

 

Collapse

// this class will be used to implement the Object Creation/Destruction

// Allocation and De-allocation inside the DLL

// since memory allocated inside a DLL must also be deleted by the same DLL

class CLASS_DECL_DLL DLLProxy
{
public:
// construction

DLLProxy();
virtual ~DLLProxy();

// creation and deletion

virtual Plug_Object* CreateObject(void);
virtual void DeleteObject(ProxyInterface* theObject);

// operations

// for reference counting

void SetDLLFE(DLLFileEntry* theDLLFE);
void DecDllUsage(void); // decrement Usage

void IncDllUsage(void); // increment Usage

char* GetDLLRelativePath(void); // get relative path to DLL for the proxy


// return valid HANDLE to loaded module or NULL

HMODULE GetSafeModuleHandle(void);
// return root path to module or NULL

static char* GetRootModulePath(void);
static void SetRootModulePath(const char* modulerootpath);

protected:
// do not allow copy

DLLProxy(const DLLProxy&);
DLLProxy& operator=(const DLLProxy&);

DLLFileEntry* myDLLFE;

static char* myModuleRootPath;
};

 

Collapse

// This class is a mix-in class and just brings in the DLLProxy member into an 

// (existing) plug-in class

class CLASS_DECL_DLL ProxyInterface  
{
public:
ProxyInterface();
virtual ~ProxyInterface();

// theDLL Proxy Access Functions

DLLProxy* GetProxy(void);
void SetProxy(DLLProxy* theProxy);

protected:
// the DLLProxy pointer is not owned by the ProxyInterface

// therefore it is not allocated or de-allocated here

DLLProxy* myProxy;
};

 

Collapse

// Plug-in Base Class

// Has all the common functionality

// This base class is virtual

// pointers to it cannot be deleted

// since the constructor and destructors are protected

class CLASS_DECL_DLL Plug_Object : public ProxyInterface  
{
public:
// virtual functions specifying the interface of the Plug-in Objects

virtual void DoSomething(void) = 0;
protected:
Plug_Object(DLLProxy* theProxy)
{
myProxy = theProxy;
}
virtual ~Plug_Object();
};

 

Collapse

// Implementation of Create and Delete inside a DLLProxy

// JUST for DEMONSTRATION purposes

// will NEVER be called!

Plug_Object* DLLProxy::CreateObject(void)
{
_ASSERT(myDLLFE != NULL);

// this line of code will be DIFFERENT for EVERY PLUG_OBJECT and DLLPROXY

// here a Plug_Object specialization object will be allocated, created

// and initialized

Plug_Object* toReturn = NULL;
if (toReturn != NULL)
{
// Do Some Reference Counting

myDLLFE->IncDllUsage();
toReturn->SetProxy(this);
}
else
{
_ASSERT(NULL);
}
return toReturn;
}

 

 

Collapse

void DLLProxy::DeleteObject(ProxyInterface* theObject)
{
if (theObject != NULL && theObject->GetProxy() == this)
{
delete theObject;
theObject = NULL;
if (myDLLFE != NULL)
{
// Do Some Reference Counting

myDLLFE->DecDllUsage();
}
}
else
{
_ASSERT(NULL);
}
}

 

Create a new DLL Project or in other words what does a Plug-in DLL contain?

One Global pointer to a DLLProxy object

Collapse

DLLProxyChild1 theProxy;
DLLProxy* theProxy = &theProxy;

1 exported function (put the name in the .DEF file) =

Collapse

extern "C" DLLProxy* GetDLLProxy(void)
{
return theProxy;
}

Definitions of the specific specialization Plug_Object class and the specific specialization DLLProxy class:

Collapse

class DLLProxyChild1 : public DLLProxy  
{
public:
DLLProxyChild1 ();
virtual ~ DLLProxyChild1 ();

Plug_Object* CreateObject(void);
void DeleteObject(ProxyInterface* theObject);
};

 

Collapse

// Plug-in Class

// Has all the common functionality

class CLASS_DECL_DLL Plug_Object_Child1 : public Plug_Object  
{
public:
Plug_Object_Child1(DLLProxy* theProxy);
virtual ~Plug_Object_Child1();

// ACTUALLY do Something!!

void DoSomething(void);
protected:
};

 

Collapse

// Implementation of Create and Delete inside a DLLProxyChild1

Plug_Object* DLLProxyChild1::CreateObject(void)
{
_ASSERT(myDLLFE != NULL);

// this line of code will be DIFFERENT for EVERY PLUG_OBJECT and DLLPROXY

Plug_Object_Child1* toReturn = new Plug_Object_Child1;
if (toReturn != NULL)
{
// Do Some Reference Counting

myDLLFE->IncDllUsage();
toReturn->SetProxy(this);
}
else
{
_ASSERT(NULL);
}
return toReturn;
}

 

Collapse

void DLLProxyChild1::DeleteObject(ProxyInterface* theObject)
{
if (theObject != NULL && theObject->GetProxy() == this)
{
delete theObject;
theObject = NULL;
if (myDLLFE != NULL)
{
// Do Some Reference Counting

myDLLFE->DecDllUsage();
}
}
else
{
_ASSERT(NULL);
}
}

 

Collapse

// shared memory!

#pragma data_seg( ".GLOBALS")
int nProcessCount = 0;
int nThreadCount = 0;
#pragma data_seg()

// remember! For every Process

DLLProxyChild1 theProxy;
DLLProxy* theProxy = &theProxy;

extern "C" BOOL APIENTRY
DllMain(HANDLE hModule, 
            DWORD  ul_reason_for_call, 
            LPVOID lpReserved
)
{
// Remove this if you use lpReserved

UNREFERENCED_PARAMETER(lpReserved);

switch( ul_reason_for_call ) 
{
case DLL_PROCESS_ATTACH:
{
nProcessCount++;
break;
}
case DLL_THREAD_ATTACH:
{
nThreadCount++;
break;
}
case DLL_THREAD_DETACH:
{
nThreadCount--;
break;
}
case DLL_PROCESS_DETACH:
{
nProcessCount--;
break;
}
default:
{
break;
}
    }
    return TRUE;
}

Serialization of Plug-in Objects

Concerning serialization of Plug-in Objects, we can distinguish between projects or programs using MFC and projects without MFC.

A. With MFC

To Save

1. Define a Serialize(CArchive& theArchive) function for every Plug_Object.
2. As usual, call the Serialize function of the CDocument.
3. The Plug_Objects store themselves to file through their Serialize function

To Load

1. Before the Serialize function of the CDocument is performed,all the necessary run-able code must be present in memory. All theDLLs, present in the DLL root path are recursively mapped in.
2. Load the file
3. Flush the unused DLLs.

B. Without MFC

Remember we have the DLL root path and we also have the relative path of the DLLs to the DLL root path.So write a function in every Plug_Object that does this:

Collapse

// This function will reconstruct the object from a BYTE stream and return NULL if not successful, 

// Otherwise it will return the pointer for the next object to serialize. 

// So if a particular Plug_Object needs 5 bytes to DeSerialize itself, the Original BYTE pointer 

// address + 5 is returned  (which you can pass to the next DeSerialize…)

BYTE*  DeSerialize(const BYTE* pFileData);

// This function constructs the BYTE stream to save and returns it, and of course also

// sets the size of the BYTE stream so you know how many bytes to write to your file/pipe/…

BYTE* Serialize( unsigned int& serialization_byte_size );

The serialisation scheme then looks like this

To save:

Collapse

While (still_objects_to_save)
{
Save relative plug-in DLL path size in 4 bytes (so you know which plug-in code to load when loading the file)
Save the relative plug-in DLL path itself (to identify the correct object)
Serialize() the object and save it
}

To load:

Collapse

While (still_data_to_read)
{
Read byte stream
4 bytes (for the size) -> relative path string
Load Plug-in DLL if not already loaded (with relative path)
Construct object from Plug-in DLL
Call DeSerialize() on Plug-in object (in fact this is a 2 phase construction!)
    // Repeat the above steps until the stream is finished (the entire file is read)

}

Some more info...

More information about the difference between a similar scheme,which I discovered some time ago in MSDN, and COM can be found in theMSDN Article "From CPP to COM" by Markus Horstmann, where COM ispresented as a superior (?) solution.

It has been pointed out to me that there is a more generalsolution to be found on Dynamic C++ Classes as "a lightweight mechanismto update code in a running program" at http://actcomm.dartmouth.edu/dynamic/

See the sample project for a demonstration of its usage.I hope all things all clear. If they are not: try stepping through the debugger, that sometimes helps. If something is not clear in the above explanation, let me know and I will try to clarify things!

Updates

Now works with VC++ 6

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)