Device drivers on SMP systems

The Colin WallsBlog: Embedded software and stuff that interests me...

November14th, 2011, by Colin Walls |Permalink

More Sharing ServicesShare

Tags: device drivers, embedded software, multi-core, Nucleus, Nucleus OS, Nucleus SMP, SMP

No Comments

As I am on vacation, I thought that I would invite colleagues toprovide a guest blog and Faheem Sheikh came up withthe goods …

Multicore embedded designs are becoming increasingly common – atopic that I have addressed before. This presents some new challenges tosoftware developers. Faheem was talking to an existing user of ourNucleus RTOS, who is considering a multicore design and, hence, atransition to the recently announced Nucleus SMP product. The matter of device drivers wasraised, which brings up some interesting issues …

Symmetric multiprocessing (SMP) is a configuration that exposesmultiple hardware resources to the software with the objective ofachieving high performance and/or increased responsiveness from thesystem. Roughly speaking an OS can do this by exploiting threadlevel parallelism (TLP) which means running multiple threads(software contexts) simultaneously on separate cores. It is up tothe developer to utilize TLP according to the requirements of theirsystem. More often than not, it necessitates a new programmingparadigm (both for development and debugging), if the developerswant to benefit from performance gains of multiple cores. However,before high performance can be achieved, most developers mustconcern themselves with the functionality of their legacy softwareand device drivers once they move to a symmetric multicoresolution. Of course, performance and responsiveness are onlyrelevant if the code is functionally correct. The question is whatdoes it take to make your existing low-level software to work in amulticore environment? The answer, somewhat surprisingly, is notmuch, especially if the SMP support in the OS is robust.

Here are some general guidelines:

• All shared data structures accessed inside the InterruptService Routine(s) (ISR) should be protected using spinlocks.
• If a device driver API contains any re-entrant code, make surethat interrupts are disabled before spinlock is acquired. This willavoid deadlocks, by not allowing another instantiation of the codeon the same core.
• Use separate locks for independent critical sectionpaths/structures for reduced contention. However, using a largenumber of locks increases the possibility of a deadlock, so thereis a trade-off.
• Explicitly tell the application developer if the correct SMPsafe behavior of an API depends on the use-case.

As an example, consider a serial UART driver working on an SoC.This system is transitioning to a multicore processor. Whatprecautions should be taken to guarantee that the driver will stillwork in a new SMP context? Under ideal circumstances therequirement would be no change in the public interfaces and verylittle change in the code of the driver. We will assume aninterrupt driven serial driver, as the polling case is relativelysimpler. Most serial drivers supports APIs to initialize the UART,get and put character on the serial port, as well as printing astring on the UART console. In addition, there might be some otherinternal functions that help set up the UART and service the UARTinterrupts etc. We will assume that the following three basic APIsare exposed:

• MySerial_Getchar (VOID) – Receives a characterfrom default UART
• MySerial_Putchar (int c) – Places a characterat default UART
• MySerial_Puts (const char* s) – Prints acharacter string from default UART

Some simple use-cases will help to illustrate the potentialhazards and their solutions while porting our UART driver to amulticore processor:

1. Two threads each executing on different cores are trying toprint strings on the UART console by callingMySerial_Puts(). If this API has been internallyprotected via a OS semaphore/mutex for correct multithreadedexecution, and the OS has robust SMP support (meaning it hasupgraded all its components including semaphores or mutex for SMPoperation), then no change in this API is required. The concurrentexecution will be handled by internal OS spinlocks.
2. One thread is printing on the UART usingMySerial_Putchar(). The other one is waiting for acommand on the serial line callsMySerial_getchar() as a result of RX serialinterrupt. This can be a potential problem, since both the coresmight be running the ISR simultaneously. A first level approach isto protect the whole ISR with a spinlock. Although this will ensurecorrectness of the driver it can drag down performance. Thedeveloper can think about introducing separate spinlocks fortransmit and receive paths, if there is mutual exclusion in therespective data paths.
3. Sometimes it is just not worth the effort to make the devicedriver absolutely safe for multicore execution. Often, it is up tothe application code to make sure it is judiciously using a sharedresource. For instance, if two threads are simultaneously usingMySerial_Putchar(), they should know that theircharacters can get lost or get printed in no particular order.Instead of introducing new spinlocks in theMySerial_Putchar() API, it is better to documentthis functionality so that an application programmer will take carewhen using this API in an SMP context.

Hopefully this post will help somewhat by allying “fears ofconcurrency” in embedded developers eying a multicore upgrade totheir systems.