android 使用SMD(共享内存)作為跨processes溝通媒介

http://eeepage.info/android-smd-wcnss/

前言: 多核間的通訊

在智慧型手機SOC平台中，為了保證個別功能在平台上的流暢運行，都會依照功能使用專門的處理器來處理。如在Qucalcomm MSM 7K 平台上，就包含了4 個處理器內核，Qucalcomm MSM 7K 平台採用ARM 9（mARM，modemARM）處理基帶業務；採用mDSP（Modem DSP）來處理協議protocol的事情；採用ARM 11（aARM ，application ARM）來負責Linux 操作系統的運行；採用aDSP（Application DSP）來處理多媒體業務方面的編/解碼加速工作。

在2010 年末，smartphone or Tablet 終端的CPU 得到了快速發展，高端的smart devices終端已經採用了Cortex-A8 的雙核處理器。而在平板電腦上，Nvidia 開發的基於Cortex-A9 的雙核處理器Tegra 2 則成了市場的寵兒。通信離不開memory記憶體的操作，在Qualcomm平台上，記憶體一般分為3種：基帶內存（Modem Memory）、應用記憶體（Application Memory）和共享記憶體，其中系統MPU保護基帶記憶體不被aARM接入，ARM MMU保護應用記憶體不被mARM接入。

共享內存share-meory

在Linux 中，實現進程通信的機制有很多種，如信號signal、管道Pipeline、信號量Semaphore 、消息隊列Message queue、共享內存share-meory和套接字socket等，但共享內存的方式效率最高。

共享內存是多核通信的物理基礎，其實現主要包括3 個部分：共享內存驅動（SMD，Shared Memory Driver）、共享內存狀態機（SMSM，Shared Memory State Machine）和共享內存管理器（SMEM，Shared Memory Manager）。
其中
SMD 用於多核之間的數據通信；
SMSM用於多核之間的狀態通信；
SMEM是一個底層的協議，是物理RAM共享內存的管理接口，是SMD和SMSM的基礎。

SMEM 具有兩種分配模式：動態SMEM 和靜態SMEM，
動態SMEM 根據需要實時分配，靜態SMEM則會預先分配。

SMEM的主要接口為：smem_alloc()、smem_find()、smem_init()等。
SMEM、SMD、SMSM的實現都需要硬件平台廠商提供支持。

同步與互斥- Synchronization and mutual exclusion

Share meomry用到了自旋鎖(spinlock)和互斥鎖(Mutual exclusion)的概念。
自旋鎖(spinlock)是Linux 內核的同步機制之一，與互斥鎖(Mutual exclusion)類似，但自旋鎖使用者一般保持鎖的時間非常短。自旋鎖的效率遠高於互斥鎖。
自旋鎖的定義位於 xxx\include\linux\spinlock_*.h文件中。

typedef struct {volatile unsigned int lock; // unsigned integer} Raw_spinlock_t;

lock雖然被定義為無符號整數，但是實際上被當做有符號整數使用。
slock值為1 代表鎖未被佔用，值為0 或負數代表鎖被佔用。初始化時slock被置為1。
與信號量Semaphore和讀寫信號量導致調用者睡眠不同，自旋鎖spinlock不會引起調用者睡眠。如果自旋鎖已被別的執行單元保持，調用者就一直循環以確定是否該自旋鎖的保持者已經釋放了鎖。

由於自旋鎖spinlock適用的訪問共享資源的時間非常短， 導致自旋鎖spinlock通常應用於中斷上下文Interrupt Context訪問和對共享資源訪問文件非常短的場景中，如果被保護的共享資源在進程process上下文訪問，則應使用信號量。

與信號量Semaphore和讀寫信號量rw_semaphore在保持期間可以被搶占的情況不同，自旋鎖spinlock在保持期間是搶占失效的，自旋鎖只有在內核可搶占或SMP（Symmetrical Multi-Processing）的情況下才真正需要，在單CPU且不可搶占的內核下，自旋鎖spinlock的所有操作都是空操作。

由於smart devices平台上通常存在多個CPU 或DSP，自旋鎖的運用就顯得非常重要。
在mach-msm Kernel裡的SMD 和SMSM的實現上，自旋鎖spinlock主要運用於中斷處理、信道列表List_of_WLAN_channels和信道狀態Channel State Information的變更過程中，自旋鎖的定義如下：

/* the spinlock is used to synchronize between the * irq handler and code that mutates the channel * list or fiddles with channel state */static DEFINE_SPINLOCK(smd_lock);DEFINE_SPINLOCK(smem_lock);

互斥鎖主要用於實現Linux 內核中的互斥訪問功能，在mach-msm Kernel的SMD 的實現上，互斥鎖主要用於SMD 信道的打開或關閉過程。定義如下：

/* the mutex is used during open() and close() * operations to avoid races while creating or * destroying smd_channel structures */static DEFINE_MUTEX(smd_creation_mutex);

SMD 控制訊號傳遞

在Linux 中，基於SMD 的訊號傳遞是以channel的形式作為一個設備存在的，作為一種雙向信道Channel，其接口的實現遵循Linux 設備驅動規範。
在Qucalcomm 平台上，SMD 的緩衝buffer大小為8192bit，最大信道數為64，SMD的頭head大小為20bit。

SMD 的相關代碼實現主要位於 arch\arm\mach-msm目錄下。
主要文件包括： smd.c、smd_nmea.c、smd_qmi.c、smd_rpcrouter.c、smd_rpcrouter_clients.c、smd_rpcrouter_device.c、smd_rpcrouter_servers.c、smd_tty.c等。

SMD 信道Channel需要同時維護接收信道、發送信道的狀態和數據信息，SMD 的信道定義如下：

smd_private.h

struct smd_channel {volatile void __iomem *send; /* some variant of smd_half_channel 發送握手channel訊息 */volatile void __iomem *recv; /* some variant of smd_half_channel 接收握手channel訊息*/unsigned char *send_data; //發送chaneel 數據資料unsigned char *recv_data; //接收chaneel 數據資料unsigned fifo_size;unsigned fifo_mask;struct list_head ch_list;unsigned current_packet;unsigned n;void *priv;void (*notify)(void *priv, unsigned flags);int (*read)(smd_channel_t *ch, void *data, int len, int user_buf); //讀取int (*write)(smd_channel_t *ch, const void *data, int len,  //寫入int user_buf);int (*read_avail)(smd_channel_t *ch); //可否讀取int (*write_avail)(smd_channel_t *ch); //可否寫入int (*read_from_cb)(smd_channel_t *ch, void *data, int len,int user_buf);void (*update_state)(smd_channel_t *ch);unsigned last_state;void (*notify_other_cpu)(smd_channel_t *ch);char name[20];struct platform_device pdev;unsigned type;int pending_pkt_sz;char is_pkt_ch;/* * private internal functions to access *send and *recv. * never to be exported outside of smd */struct smd_half_channel_access *half_ch;};

SMD 初始化過程

msm_smd_init()

for (i = 0; i < NUM_SMD_SUBSYSTEMS; ++i) {remote_info[i].remote_pid = i;remote_info[i].free_space = UINT_MAX;INIT_WORK(&remote_info[i].probe_work, smd_channel_probe_worker);INIT_LIST_HEAD(&remote_info[i].ch_list);}

smd_channel_probe_worker()
– Scan for newly created SMD channels and init local structures so the channels are visable to local clients
==> scan_alloc_table(shared, &(r_info->ch_allocated[SMEM_NUM_SMD_STREAM_CHANNELS]), SEC_ALLOC_TBL, tbl_size / sizeof(*shared), r_info);

scan_alloc_table()

/** * scan_alloc_table - Scans a specified SMD channel allocation table in SMEM for newly created channels that need to be made locally visable * @shared: pointer to the table array in SMEM * @smd_ch_allocated: pointer to an array indicating already allocated channels * @table_id: identifier for this channel allocation table * @num_entries: number of entries in this allocation table * @r_info: pointer to the info structure of the remote proc we care about * The smd_probe_lock must be locked by the calling function.  Shared and smd_ch_allocated are assumed to be valid pointers. */static void scan_alloc_table(struct smd_alloc_elm *shared, char *smd_ch_allocated, int table_id, unsigned num_entries, struct remote_proc_info *r_info){unsigned n;uint32_t type;for (n = 0; n < num_entries; n++) {if (smd_ch_allocated[n])continue;/* * channel should be allocated only if APPS processor is involved */type = SMD_CHANNEL_TYPE(shared[n].type);if (!pid_is_on_edge(type, SMD_APPS) ||!pid_is_on_edge(type, r_info->remote_pid))continue;if (!shared[n].ref_count)continue;if (!shared[n].name[0])continue;if (!smd_initialized && !smd_edge_inited(type)) {SMD_INFO("Probe skipping proc %d, tbl %d, ch %d, edge not inited r_info->remote_pid, table_id, n);continue;}if (!smd_alloc_channel(&shared[n], table_id, r_info))smd_ch_allocated[n] = 1;elseSMD_INFO("Probe skipping proc %d, tbl %d, ch %d, not allocated r_info->remote_pid, table_id, n);}}

smd_alloc_channel(): 分配channel

/** * smd_alloc_channel() - Create and init local structures for a newly allocated SMD channel * @alloc_elm: the allocation element stored in SMEM for this channel * @table_id: the id of the table this channel resides in. 1 = first table, 2 = *seconds table, etc * @r_info: pointer to the info structure of the remote proc for this channel * @returns: -1 for failure; 0 for success */static int smd_alloc_channel(struct smd_alloc_elm *alloc_elm, int table_id,struct remote_proc_info *r_info){struct smd_channel *ch;ch = kzalloc(sizeof(struct smd_channel), GFP_KERNEL);if (ch == 0) {pr_err("smd_alloc_channel() out of memory\n");return -1;}ch->n = alloc_elm->cid;ch->type = SMD_CHANNEL_TYPE(alloc_elm->type);if (smd_alloc_v2(ch, table_id, r_info) && smd_alloc_v1(ch)) {kfree(ch);return -1;}ch->fifo_mask = ch->fifo_size - 1;/* probe_worker guarentees ch->type will be a valid type */if (ch->type == SMD_APPS_MODEM)ch->notify_other_cpu = notify_modem_smd;else if (ch->type == SMD_APPS_QDSP)ch->notify_other_cpu = notify_dsp_smd;else if (ch->type == SMD_APPS_DSPS)ch->notify_other_cpu = notify_dsps_smd;else if (ch->type == SMD_APPS_WCNSS)ch->notify_other_cpu = notify_wcnss_smd;else if (ch->type == SMD_APPS_RPM)ch->notify_other_cpu = notify_rpm_smd;if (smd_is_packet(alloc_elm)) {   //如果是 packet包 channelch->read = smd_packet_read;ch->write = smd_packet_write;ch->read_avail = smd_packet_read_avail;ch->write_avail = smd_packet_write_avail;ch->update_state = update_packet_state;ch->read_from_cb = smd_packet_read_from_cb;ch->is_pkt_ch = 1;} else {                     //如果是 stream流 channelch->read = smd_stream_read;ch->write = smd_stream_write;ch->read_avail = smd_stream_read_avail;ch->write_avail = smd_stream_write_avail;ch->update_state = update_stream_state;ch->read_from_cb = smd_stream_read;}memcpy(ch->name, alloc_elm->name, SMD_MAX_CH_NAME_LEN);ch->name[SMD_MAX_CH_NAME_LEN-1] = 0;ch->pdev.name = ch->name;ch->pdev.id = ch->type;SMD_INFO("smd_alloc_channel() '%s' cid=%d\n", ch->name, ch->n);mutex_lock(&smd_creation_mutex);/互斥锁  list_add(&ch->ch_list, &smd_ch_closed_list); //将信道添加到“smd_ch_closed_list”列表中mutex_unlock(&smd_creation_mutex);platform_device_register(&ch->pdev); //注册设备if (!strncmp(ch->name, "LOOPBACK", 8) && ch->type == SMD_APPS_MODEM) {/* create a platform driver to be used by smd_tty driver * so that it can access the loopback port */loopback_tty_pdev.id = ch->type;platform_device_register(&loopback_tty_pdev);}return 0;}

開啟channel- Open the SMD channel

常見在wifi driver初始化過程中會呼叫, 或是該driver需要SMD的話就會打開SMD啦…
為了打開一個信道channel，首先要判斷SMD 信道是否已經初始化。
如果SMD 信道已經初始化，就根據信道channel名獲得信道channel，將信道channel加入到“smd_ch_list”信道列表中並設置該信道的狀態為SMD_SS_OPENING，然後調用notify_other_smd()函數通知其他的信道該信道已經激活enable。

在默認default情況下，其信道類型為SMD_APPS_MODEM，打開一個SMD信道的實現如下：

int smd_named_open_on_edge(const char *name, uint32_t edge,   smd_channel_t **_ch,   void *priv, void (*notify)(void *, unsigned)){struct smd_channel *ch;unsigned long flags;if (smd_initialized == 0 && !smd_edge_inited(edge)) {  //判断SMD 信道是否已初始化 SMD_INFO("smd_open() before smd_init()\n");return -ENODEV;}SMD_DBG("smd_open('%s', %p, %p)\n", name, priv, notify);ch = smd_get_channel(name, edge);  //獲得channel頻道if (!ch) {/* check closing list for port */spin_lock_irqsave(&smd_lock, flags); //自旋锁  list_for_each_entry(ch, &smd_ch_closing_list, ch_list) {if (!strncmp(name, ch->name, 20) &&(edge == ch->type)) {/* channel exists, but is being closed */spin_unlock_irqrestore(&smd_lock, flags);return -EAGAIN;}}/* check closing workqueue list for port */list_for_each_entry(ch, &smd_ch_to_close_list, ch_list) {if (!strncmp(name, ch->name, 20) &&(edge == ch->type)) {/* channel exists, but is being closed */spin_unlock_irqrestore(&smd_lock, flags);return -EAGAIN;}}spin_unlock_irqrestore(&smd_lock, flags);/* one final check to handle closing->closed race condition */ch = smd_get_channel(name, edge);if (!ch)return -ENODEV;}if (notify == 0)notify = do_nothing_notify;ch->notify = notify;ch->current_packet = 0;ch->last_state = SMD_SS_CLOSED;ch->priv = priv;if (edge == SMD_LOOPBACK_TYPE) {ch->last_state = SMD_SS_OPENED;ch->half_ch->set_state(ch->send, SMD_SS_OPENED);ch->half_ch->set_fDSR(ch->send, 1);ch->half_ch->set_fCTS(ch->send, 1);ch->half_ch->set_fCD(ch->send, 1);}*_ch = ch;SMD_DBG("smd_open: opening '%s'\n", ch->name);spin_lock_irqsave(&smd_lock, flags); //自旋锁 if (unlikely(ch->type == SMD_LOOPBACK_TYPE))list_add(&ch->ch_list, &smd_ch_list_loopback); //将信道添加到“smd_ch_list_loopback”列表中elselist_add(&ch->ch_list, &remote_info[edge_to_pids[ch->type].remote_pid].ch_list);SMD_DBG("%s: opening ch %d\n", __func__, ch->n);if (edge != SMD_LOOPBACK_TYPE)smd_state_change(ch, ch->last_state, SMD_SS_OPENING); //信道状态变更 spin_unlock_irqrestore(&smd_lock, flags);return 0;}EXPORT_SYMBOL(smd_named_open_on_edge);

關閉信道- smd_close()

關閉信道的操作相對簡單，首先將信道從“smd_ch_list”信道列表中刪除，然後將信道狀態設置為SMD_SS_CLOSED，並將信道添加到“smd_ch_closed_list”信道列表中即可。關閉SMD 信道的實現如下：

int smd_close(smd_channel_t *ch){unsigned long flags;if (ch == 0)return -1;SMD_INFO("smd_close(%s)\n", ch->name);spin_lock_irqsave(&smd_lock, flags); //自旋锁list_del(&ch->ch_list); //从打开信道列表中去除该信道if (ch->n == SMD_LOOPBACK_CID) {ch->half_ch->set_fDSR(ch->send, 0);ch->half_ch->set_fCTS(ch->send, 0);ch->half_ch->set_fCD(ch->send, 0);ch->half_ch->set_state(ch->send, SMD_SS_CLOSED);} elsech_set_state(ch, SMD_SS_CLOSED);   //设置信道状态if (ch->half_ch->get_state(ch->recv) == SMD_SS_OPENED) {list_add(&ch->ch_list, &smd_ch_closing_list);spin_unlock_irqrestore(&smd_lock, flags);} else {spin_unlock_irqrestore(&smd_lock, flags);ch->notify = do_nothing_notify;mutex_lock(&smd_creation_mutex);  //互斥锁list_add(&ch->ch_list, &smd_ch_closed_list); //将信道添加至关闭信道列表 mutex_unlock(&smd_creation_mutex);}return 0;}EXPORT_SYMBOL(smd_close);

channel內容讀取-1. smd_packet_read

packet channel的內容讀取涉及緩衝的複制、與其他SMD 的消息通信和包狀態的更新。
從packet channel讀取數據的實現如下：

static int smd_packet_read(smd_channel_t *ch, void *data, int len, int user_buf){unsigned long flags;int r;if (len < 0)return -EINVAL;if (len > ch->current_packet)len = ch->current_packet;r = ch_read(ch, data, len, user_buf); //读取数据if (r > 0)if (!read_intr_blocked(ch))ch->notify_other_cpu(ch);spin_lock_irqsave(&smd_lock, flags); //自旋锁ch->current_packet -= r;update_packet_state(ch); //更新包状态 spin_unlock_irqrestore(&smd_lock, flags);return r;

channel內容讀取-2. smd_stream_read

流信道的內容讀取非常簡單，只需要調用ch_read()函數讀取數據並通知其他SMD 該信道處於打開狀態即可。
smd_stream_read 流信道讀取的實現如下：

static int smd_stream_read(smd_channel_t *ch, void *data, int len, int user_buf){int r;if (len < 0)return -EINVAL;r = ch_read(ch, data, len, user_buf); //读取数据 if (r > 0)if (!read_intr_blocked(ch))ch->notify_other_cpu(ch); //通知其他CPU，该信道处于激活状态。return r;}

流信道和包信道在讀取信道數據時，
都需要調用ch_read()函數來實現真正的數據讀取

ch_read()函數的實現如下：

/* basic read interface to ch_read_{buffer,done} used * by smd_*_read() and update_packet_state() * will read-and-discard if the _data pointer is null */static int ch_read(struct smd_channel *ch, void *_data, int len, int user_buf){void *ptr;unsigned n;unsigned char *data = _data;int orig_len = len;int r = 0;while (len > 0) {n = ch_read_buffer(ch, &ptr); //將channel的if (n == 0)break;if (n > len)n = len;if (_data) {if (user_buf) { //如果是來自user space的, 但我們目前看到的是 kernel 內部呼叫所以為else..r = copy_to_user(data, ptr, n);if (r > 0) {pr_err("%s: ""copy_to_user could not copy ""%i bytes.\n",__func__,r);}} elsememcpy(data, ptr, n); //数据复制}data += n;len -= n;ch_read_done(ch, n); //读取完成}return orig_len - len;}

信道寫入-smd_packet_write

在信道的數據寫入方面，同樣分為流信道數據的寫入和包信道數據的寫入兩種類型。向包信道中寫入數據的過程和讀取數據不同，在寫入數據前，要首先利用smd_stream_write_avail()函數判斷是否有數據可供寫入，在確定有可供寫入的數據的情況下才調用sm​​d_stream_write()函數執行數據的寫入操作。包信道寫入數據的實現如下

static int smd_packet_write(smd_channel_t *ch, const void *_data, int len,int user_buf){int ret;unsigned hdr[5];SMD_DBG("smd_packet_write() %d -> ch%d\n", len, ch->n);if (len < 0)return -EINVAL;else if (len == 0)return 0;if (smd_stream_write_avail(ch) < (len + SMD_HEADER_SIZE))return -ENOMEM;hdr[0] = len;hdr[1] = hdr[2] = hdr[3] = hdr[4] = 0;ret = smd_stream_write(ch, hdr, sizeof(hdr), 0); //流写入if (ret < 0 || ret != sizeof(hdr)) {SMD_DBG("%s failed to write pkt header: ""%d returned\n", __func__, ret);return -1;}ret = smd_stream_write(ch, _data, len, user_buf);if (ret < 0 || ret != len) {SMD_DBG("%s failed to write pkt data: ""%d returned\n", __func__, ret);return ret;}return len;}

smd_stream_write流信道數據的寫入
和包信道數據的寫入不同，
其首先獲得下一段可用緩衝的指針，然後執行內存複製，流信道寫入數據的實現如下：

static int smd_stream_write(smd_channel_t *ch, const void *_data, int len,int user_buf){void *ptr;const unsigned char *buf = _data;unsigned xfer;int orig_len = len;int r = 0;SMD_DBG("smd_stream_write() %d -> ch%d\n", len, ch->n);if (len < 0)return -EINVAL;else if (len == 0)return 0;while ((xfer = ch_write_buffer(ch, &ptr)) != 0) { //写入数据if (!ch_is_open(ch)) {len = orig_len;break;}if (xfer > len)xfer = len;if (user_buf) {r = copy_from_user(ptr, buf, xfer);if (r > 0) {pr_err("%s: ""copy_from_user could not copy %i ""bytes.\n",__func__,r);}} elsememcpy(ptr, buf, xfer);ch_write_done(ch, xfer); //完成写入数据len -= xfer;buf += xfer;if (len == 0)break;}if (orig_len - len)ch->notify_other_cpu(ch);return orig_len - len;}

更仔細看 ch_write_buffer()

會提供指標給呼叫者, 也回傳寫入數目數字

static unsigned ch_write_buffer(struct smd_channel *ch, void **ptr){unsigned head = ch->half_ch->get_head(ch->send); //得到head 頭的數字unsigned tail = ch->half_ch->get_tail(ch->send); //得到tail 尾的數字*ptr = (void *) (ch->send_data + head);   //將channel的傳送資料位置加上頭的數字, 則此指標就是可以寫入資料的位置if (head < tail) {return tail - head - SMD_FIFO_FULL_RESERVE; //如果頭數字反而比較小, 表示只有尾到頭這段是可以寫入(還要再扣掉保留區)} else {if (tail < SMD_FIFO_FULL_RESERVE)return ch->fifo_size + tail - head - SMD_FIFO_FULL_RESERVE; //如果頭比尾的數字大, 可寫入的位置就是全部-頭+尾(還要再扣掉保留區)elsereturn ch->fifo_size - head;}}

FIFO 示意圖

更仔細看 ch_read_buffer()

會在FIFO中, 提供一個指標和可讀數據長度

static unsigned ch_read_buffer(struct smd_channel *ch, void **ptr){unsigned head = ch->half_ch->get_head(ch->recv); //得到head 頭的數字unsigned tail = ch->half_ch->get_tail(ch->recv); //得到tail 尾的數字*ptr = (void *) (ch->recv_data + tail); //將channel的接受資料位置加上尾的數字, 則此指標就是可以接收資料的位置if (tail <= head)return head - tail; //由於要接收, 所以直觀上若尾比頭小, 表示尾到頭這段是可以接收的elsereturn ch->fifo_size - tail; //反之, 由於要接收, 所以若頭比尾小, 表示全部扣掉尾那段是可以接收的}

SMD 是多核通信的基礎，
通過SMD，可以為系統提供RPC、DIAG、AT命令、NMEA（GPS數據）、數據服務、撥號等服務。