Orocos DataPort 解析: orocos lock free data object

Orocos有两种 dataPort (Input, Output)，在 conncet(Output, Input) 之后，往 Output 中写的数据能够传递到 Input 中并读取, 一个OutputPort可以连接多个InputPort(类似于ROS中的Pub&Sub)，这极大地提高了模块化设计的数据交换。

其中能够实现数据交换的关键是 ChannelDataElement 数据结构，在 connect 两个端口的同时，会 new 一个 ChannelDataElement 数据结构，然后在两个DataPort中分别保存其指针，这个 ChannelDataElement 承担着数据中转站的角色，有 write 和 read 方法分别供 OutputPort 和 InputPort 使用。而在其中起到关键作用的成员变量是 DataObject 对象，它可以是 DATA，BUFFER，CIRCULAR_BUFFER 三种类型之一，提供暂存数据的作用。

ChannelDataElement 数据结构的代码如下：

template<typename T>class ChannelDataElement : public base::ChannelElement<T>{    bool written, mread;    typename base::DataObjectInterface<T>::shared_ptr data;public:    typedef typename base::ChannelElement<T>::param_t param_t;    typedef typename base::ChannelElement<T>::reference_t reference_t;    ChannelDataElement(typename base::DataObjectInterface<T>::shared_ptr sample)        : written(false), mread(false), data(sample) {}    /** Update the data sample stored in this element.     * It always returns true. */    virtual bool write(param_t sample)    {        data->Set(sample);        written = true;        mread = false;        return this->signal();    }    /** Reads the last sample given to write()     *     * @return false if no sample has ever been written, true otherwise     */    virtual FlowStatus read(reference_t sample, bool copy_old_data)    {        if (written)        {            if ( !mread ) {  data->Get(sample);                mread = true;                return NewData;  // 枚举变量            }if(copy_old_data)  data->Get(sample);            return OldData;  // 枚举变量        }        return NoData;  // 枚举变量    }    /** Resets the stored sample. After clear() has been called, read()     * returns false     */    virtual void clear()    {        written = false;        mread = false;        base::ChannelElement<T>::clear();    }    virtual bool data_sample(param_t sample)    {        data->data_sample(sample);        return base::ChannelElement<T>::data_sample(sample);    }    virtual T data_sample()    {        return data->Get();    }};

其中一个DataObjectLockFree 结构如下：lock free 的 circular bufer， 一个 producer + 多个 consumer
这里还有一个Lock free 的 Mulity Writer Single Reader 队列数据结构 see link

相比于使用mutex, lock free 的好处就是，当获得这个共享资源的线程被挂起的时候，其他线程照样能够正常读写该资源，可以说是 real-time 的读写。

关于多线程 lock_free 的设计： In general, any time you have a small amount of data protected by a mutex, and you can pack that data entirely into a 32- or 64-bit integer type, you can always convert your mutex-based operations into lock-free RMW operations, no matter what those operations actually do! 但是很多情况用户自定义数据结构(类)有很多数据成员，比64-bit要大得多，而且不一定是 natively aligned，所以读写操作不可能是 atomic 的，存在读写自定义数据的过程中被其他线程中断(如再次进行写操作)的情况。不过，我们可以设计 std::atomic<MyClass*> read_ptr 和 std::atomic<MyClass*> write_ptr，通过 compare_exchange_weak 局部变量来实现安全的指针交换。

Orocos DataObjectLockFree:

namespace RTT{ namespace base {    /**     * @brief This DataObject is a Lock-Free implementation,     * such that reads and writes can happen concurrently without priority     * inversions.     *     * When there are more writes than reads, the last write will     * be returned. The internal buffer can get full if too many     * concurrent reads are taking to long. In that case, each new     * read will read the element the previous read returned.     *     * @verbatim     * The following Truth table applies when a Low Priority thread is     * preempted by a High Priority thread :     *     *   L\H | Set | Get |     *   Set | Ok  | Ok  |     *   Get | Ok  | Ok  |     *     * legend : L : Low Priority thread     *          H : High Priority thread     *          Blk: Blocks High Priority thread (bad!)     *          internal::NA : Not allowed !     * @endverbatim     * Further, multiple reads may occur before, during and after     * a write operation simultaneously. The buffer needs readers+2*writers     * elements to be guaranteed non blocking.     * @ingroup PortBuffers     */    template<class T>    class DataObjectLockFree        : public DataObjectInterface<T>    {    public:        /**         * The type of the data.         */        typedef T DataType;        /**         * @brief The maximum number of threads.         *         * When used in data flow, this is always 2.         */        const unsigned int MAX_THREADS; // = 2    private:        /**         * Conversion of number of threads to size of buffer.         */        const unsigned int BUF_LEN; // = MAX_THREADS+2        /**         * Internal buffer structure.         * Both the read and write pointers pointing to this struct         * must be declared volatile, since they are modified in other threads.         * I did not declare data as volatile,         * since we only read/write it in secured buffers.         */        struct DataBuf {            DataBuf()                : data(), counter(), next()            {                oro_atomic_set(&counter, 0);            }            DataType data; mutable oro_atomic_t counter; DataBuf* next;        };        typedef DataBuf* volatile VolPtrType;        typedef DataBuf  ValueType;        typedef DataBuf* PtrType;        VolPtrType read_ptr;        VolPtrType write_ptr;        /**         * A 3 element Data buffer         */        DataBuf* data;    public:        /**         * Construct a DataObjectLockFree by name.         *         * @param _name The name of this DataObject.         * @param initial_value The initial value of this DataObject.         */        DataObjectLockFree( const T& initial_value = T(), unsigned int max_threads = 2 )            : MAX_THREADS(max_threads), BUF_LEN( max_threads + 2),              read_ptr(0),              write_ptr(0)        {            data = new DataBuf[BUF_LEN];            read_ptr = &data[0];            write_ptr = &data[1];            data_sample(initial_value);        }        ~DataObjectLockFree() {            delete[] data;        }        /**         * Get a copy of the data.         * This method will allocate memory twice if data is not a value type.         * Use Get(DataType&) for the non-allocating version.         *         * @return A copy of the data.         */        virtual DataType Get() const {DataType cache; Get(cache); return cache; }        /**         * Get a copy of the Data (non allocating).         * If pull has reserved enough memory to store the copy,         * no memory will be allocated.         *         * @param pull A copy of the data.         */        virtual void Get( DataType& pull ) const        {            PtrType reading;            // loop to combine Read/Modify of counter            // This avoids a race condition where read_ptr            // could become write_ptr ( then we would read corrupted data).            do {                reading = read_ptr;            // copy buffer location                oro_atomic_inc(&reading->counter); // lock buffer, no more writes 不能在读的过程中进行写操作                // XXX smp_mb                if ( reading != read_ptr )     // if read_ptr changed,                    oro_atomic_dec(&reading->counter); // better to start over.                else                    break;            } while ( true );            // from here on we are sure that 'reading'            // is a valid buffer to read from.            // 这里是关键，由于用户自定义数据不一定是natively aligned，所以读操作不一定是atomic的，可能在读取自定义数据的过程中被其他线程中断(如再次进行写操作)。             pull = reading->data;               // takes some time            // XXX smp_mb            oro_atomic_dec(&reading->counter);       // release buffer        }        /**         * Set the data to a certain value (non blocking).         *         * @param push The data which must be set.         */        virtual void Set( const DataType& push )        {            /**             * This method can not be called concurrently (only one             * producer). With a minimum of 3 buffers, if the             * write_ptr+1 field is not occupied, it will remain so             * because the read_ptr is at write_ptr-1 (and can             * not increment the counter on write_ptr+1). Hence, no             * locking is needed.             */            // writeout in any case，只有一个producer,并且write_ptr!=read_ptr，所以永远是安全的！            write_ptr->data = push;            PtrType wrote_ptr = write_ptr;            // if next field is occupied (by read_ptr or counter),            // go to next and check again...            // 环形buffer，如果是前一个还没有读完，这时候新的写进去又加进来新的读取就会有这种事情发生;            // 如果前一个读取没有完成(就是那个takes some time的地方)，后续最多能再进行两次写操作（buffer大小为4）.            while ( oro_atomic_read( &write_ptr            ->next->counter ) != 0 || write_ptr->next == read_ptr )                {                    write_ptr = write_ptr->next;                    if (write_ptr == wrote_ptr)                        return; // nothing found, to many readers !                }            // we will be able to move, so replace read_ptr            read_ptr  = wrote_ptr;            write_ptr = write_ptr->next; // we checked this in the while loop        }        virtual void data_sample( const DataType& sample ) {            // prepare the buffer.            for (unsigned int i = 0; i < BUF_LEN-1; ++i) {                data[i].data = sample;                data[i].next = &data[i+1];            }            data[BUF_LEN-1].data = sample;            data[BUF_LEN-1].next = &data[0];        }    };}}

reference link:
1. https://msdn.microsoft.com/en-us/library/windows/desktop/ee418650(v=vs.85).aspx
2. http://www.boost.org/doc/libs/1_60_0/doc/html/lockfree.html
3. http://preshing.com/20150402/you-can-do-any-kind-of-atomic-read-modify-write-operation/
4. http://preshing.com/20120612/an-introduction-to-lock-free-programming/
5. https://en.wikipedia.org/wiki/ABA_problem