【源码剖析】threadpool —— 基于 pthread 实现的简单线程池
来源:互联网 发布:杭州龙翔桥到淘宝城 编辑:程序博客网 时间:2024/06/05 12:51
线程池介绍
线程池可以说是项目中经常会用到的组件,在这里假设读者都有一定的多线程基础,如果没有的话不妨在这里进行了解:POSIX 多线程基础。
线程池是什么?我的简单理解是有一组预先派生的线程,然后有一个管理员来管理和调度这些线程,你只需不断把需要完成的任务交给他,他就会调度线程的资源来帮你完成。
那么管理员是怎么做的呢?一种简单的方式就是,管理员管理一个任务的队列,如果收到新的任务,就把任务加到队列尾。每个线程盯着队列,如果队列非空,就去队列头拿一个任务来处理(每个任务只能被一个线程拿到),处理完了就继续去队列取任务。如果没有任务了,线程就休眠,直到任务队列不为空。如果这个管理员更聪明一点,他可能会在没有任务或任务少的时候减少线程的数量,任务处理不过来的时候增加线程的数量,这样就实现了资源的动态管理。
那么任务是什么呢?以后台服务器为例,每一个用户的请求就是一个任务,线程不断的在请求队列里取出请求,完成后继续处理下一个请求。
简单图示为:
线程池有一个好处就是减少线程创建和销毁的时间,在任务处理时间比较短的时候这个好处非常显著,可以提升任务处理的效率。
线程池实现
这里介绍的是线程池的一个简单实现,在创建的时候预先派生指定数量的线程,然后去任务队列取添加进来的任务进行处理就好。
作者说之后会添加更多特性,我们作为学习之后就以这个版本为准就好了。
项目主页:https://github.com/yangtingcheng/projectwithcomment/tree/master/threadpool
数据结构
主要有两个自定义的数据结构
threadpool_task_t
用于保存一个等待执行的任务。一个任务需要指明:要运行的对应函数及函数的参数。所以这里的 struct 里有函数指针和 void 指针。
typedef struct { void (*function)(void *); void *argument;} threadpool_task_t;
- 1
- 2
- 3
- 4
- 1
- 2
- 3
- 4
thread_pool_t
一个线程池的结构。因为是 C 语言,所以这里任务队列是用数组,并维护队列头和队列尾来实现。
struct threadpool_t { pthread_mutex_t lock; /* 互斥锁 */ pthread_cond_t notify; /* 条件变量 */ pthread_t *threads; /* 线程数组的起始指针 */ threadpool_task_t *queue; /* 任务队列数组的起始指针 */ int thread_count; /* 线程数量 */ int queue_size; /* 任务队列长度 */ int head; /* 当前任务队列头 */ int tail; /* 当前任务队列尾 */ int count; /* 当前待运行的任务数 */ int shutdown; /* 线程池当前状态是否关闭 */ int started; /* 正在运行的线程数 */};
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
函数
对外接口
threadpool_t *threadpool_create(int thread_count, int queue_size, int flags);
创建线程池,用 thread_count 指定派生线程数,queue_size 指定任务队列长度,flags 为保留参数,未使用。int threadpool_add(threadpool_t *pool, void (*routine)(void *),void *arg, int flags);
添加需要执行的任务。第二个参数为对应函数指针,第三个为对应函数参数。flags 未使用。int threadpool_destroy(threadpool_t *pool, int flags);
销毁存在的线程池。flags 可以指定是立刻结束还是平和结束。立刻结束指不管任务队列是否为空,立刻结束。平和结束指等待任务队列的任务全部执行完后再结束,在这个过程中不可以添加新的任务。
内部辅助函数
static void *threadpool_thread(void *threadpool);
线程池每个线程所执行的函数。int threadpool_free(threadpool_t *pool);
释放线程池所申请的内存资源。
线程池使用
编译
参考项目根目录下的 Makefile, 直接用 make
编译。
测试用例
项目提供了三个测试用例(见threadpool/test/
),我们可以以此来学习线程池的用法并测试是否正常工作。这里提供其中一个:
#define THREAD 32#define QUEUE 256#include <stdio.h>#include <pthread.h>#include <unistd.h>#include <assert.h>#include "threadpool.h"int tasks = 0, done = 0;pthread_mutex_t lock;void dummy_task(void *arg) { usleep(10000); pthread_mutex_lock(&lock); /* 记录成功完成的任务数 */ done++; pthread_mutex_unlock(&lock);}int main(int argc, char **argv){ threadpool_t *pool; /* 初始化互斥锁 */ pthread_mutex_init(&lock, NULL); /* 断言线程池创建成功 */ assert((pool = threadpool_create(THREAD, QUEUE, 0)) != NULL); fprintf(stderr, "Pool started with %d threads and " "queue size of %d\n", THREAD, QUEUE); /* 只要任务队列还没满,就一直添加 */ while(threadpool_add(pool, &dummy_task, NULL, 0) == 0) { pthread_mutex_lock(&lock); tasks++; pthread_mutex_unlock(&lock); } fprintf(stderr, "Added %d tasks\n", tasks); /* 不断检查任务数是否完成一半以上,没有则继续休眠 */ while((tasks / 2) > done) { usleep(10000); } /* 这时候销毁线程池,0 代表 immediate_shutdown */ assert(threadpool_destroy(pool, 0) == 0); fprintf(stderr, "Did %d tasks\n", done); return 0;}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
源码注释
源码注释一并放在 github, 点我。
threadpool.h
/* * Copyright (c) 2013, Mathias Brossard <mathias@brossard.org>. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */#ifndef _THREADPOOL_H_#define _THREADPOOL_H_#ifdef __cplusplus/* 对于 C++ 编译器,指定用 C 的语法编译 */extern "C" {#endif/** * @file threadpool.h * @brief Threadpool Header File */ /** * Increase this constants at your own risk * Large values might slow down your system */#define MAX_THREADS 64#define MAX_QUEUE 65536/* 简化变量定义 */typedef struct threadpool_t threadpool_t;/* 定义错误码 */typedef enum { threadpool_invalid = -1, threadpool_lock_failure = -2, threadpool_queue_full = -3, threadpool_shutdown = -4, threadpool_thread_failure = -5} threadpool_error_t;typedef enum { threadpool_graceful = 1} threadpool_destroy_flags_t;/* 以下是线程池三个对外 API *//** * @function threadpool_create * @brief Creates a threadpool_t object. * @param thread_count Number of worker threads. * @param queue_size Size of the queue. * @param flags Unused parameter. * @return a newly created thread pool or NULL *//** * 创建线程池,有 thread_count 个线程,容纳 queue_size 个的任务队列,flags 参数没有使用 */threadpool_t *threadpool_create(int thread_count, int queue_size, int flags);/** * @function threadpool_add * @brief add a new task in the queue of a thread pool * @param pool Thread pool to which add the task. * @param function Pointer to the function that will perform the task. * @param argument Argument to be passed to the function. * @param flags Unused parameter. * @return 0 if all goes well, negative values in case of error (@see * threadpool_error_t for codes). *//** * 添加任务到线程池, pool 为线程池指针,routine 为函数指针, arg 为函数参数, flags 未使用 */int threadpool_add(threadpool_t *pool, void (*routine)(void *), void *arg, int flags);/** * @function threadpool_destroy * @brief Stops and destroys a thread pool. * @param pool Thread pool to destroy. * @param flags Flags for shutdown * * Known values for flags are 0 (default) and threadpool_graceful in * which case the thread pool doesn't accept any new tasks but * processes all pending tasks before shutdown. *//** * 销毁线程池,flags 可以用来指定关闭的方式 */int threadpool_destroy(threadpool_t *pool, int flags);#ifdef __cplusplus}#endif#endif /* _THREADPOOL_H_ */
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
threadpool.c
/* * Copyright (c) 2013, Mathias Brossard <mathias@brossard.org>. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. *//** * @file threadpool.c * @brief Threadpool implementation file */#include <stdlib.h>#include <pthread.h>#include <unistd.h>#include "threadpool.h"/** * 线程池关闭的方式 */typedef enum { immediate_shutdown = 1, graceful_shutdown = 2} threadpool_shutdown_t;/** * @struct threadpool_task * @brief the work struct * * @var function Pointer to the function that will perform the task. * @var argument Argument to be passed to the function. *//** * 线程池一个任务的定义 */typedef struct { void (*function)(void *); void *argument;} threadpool_task_t;/** * @struct threadpool * @brief The threadpool struct * * @var notify Condition variable to notify worker threads. * @var threads Array containing worker threads ID. * @var thread_count Number of threads * @var queue Array containing the task queue. * @var queue_size Size of the task queue. * @var head Index of the first element. * @var tail Index of the next element. * @var count Number of pending tasks * @var shutdown Flag indicating if the pool is shutting down * @var started Number of started threads *//** * 线程池的结构定义 * @var lock 用于内部工作的互斥锁 * @var notify 线程间通知的条件变量 * @var threads 线程数组,这里用指针来表示,数组名 = 首元素指针 * @var thread_count 线程数量 * @var queue 存储任务的数组,即任务队列 * @var queue_size 任务队列大小 * @var head 任务队列中首个任务位置(注:任务队列中所有任务都是未开始运行的) * @var tail 任务队列中最后一个任务的下一个位置(注:队列以数组存储,head 和 tail 指示队列位置) * @var count 任务队列里的任务数量,即等待运行的任务数 * @var shutdown 表示线程池是否关闭 * @var started 开始的线程数 */struct threadpool_t { pthread_mutex_t lock; pthread_cond_t notify; pthread_t *threads; threadpool_task_t *queue; int thread_count; int queue_size; int head; int tail; int count; int shutdown; int started;};/** * @function void *threadpool_thread(void *threadpool) * @brief the worker thread * @param threadpool the pool which own the thread *//** * 线程池里每个线程在跑的函数 * 声明 static 应该只为了使函数只在本文件内有效 */static void *threadpool_thread(void *threadpool);int threadpool_free(threadpool_t *pool);threadpool_t *threadpool_create(int thread_count, int queue_size, int flags){ if(thread_count <= 0 || thread_count > MAX_THREADS || queue_size <= 0 || queue_size > MAX_QUEUE) { return NULL; } threadpool_t *pool; int i; /* 申请内存创建内存池对象 */ if((pool = (threadpool_t *)malloc(sizeof(threadpool_t))) == NULL) { goto err; } /* Initialize */ pool->thread_count = 0; pool->queue_size = queue_size; pool->head = pool->tail = pool->count = 0; pool->shutdown = pool->started = 0; /* Allocate thread and task queue */ /* 申请线程数组和任务队列所需的内存 */ pool->threads = (pthread_t *)malloc(sizeof(pthread_t) * thread_count); pool->queue = (threadpool_task_t *)malloc (sizeof(threadpool_task_t) * queue_size); /* Initialize mutex and conditional variable first */ /* 初始化互斥锁和条件变量 */ if((pthread_mutex_init(&(pool->lock), NULL) != 0) || (pthread_cond_init(&(pool->notify), NULL) != 0) || (pool->threads == NULL) || (pool->queue == NULL)) { goto err; } /* Start worker threads */ /* 创建指定数量的线程开始运行 */ for(i = 0; i < thread_count; i++) { if(pthread_create(&(pool->threads[i]), NULL, threadpool_thread, (void*)pool) != 0) { threadpool_destroy(pool, 0); return NULL; } pool->thread_count++; pool->started++; } return pool; err: if(pool) { threadpool_free(pool); } return NULL;}int threadpool_add(threadpool_t *pool, void (*function)(void *), void *argument, int flags){ int err = 0; int next; if(pool == NULL || function == NULL) { return threadpool_invalid; } /* 必须先取得互斥锁所有权 */ if(pthread_mutex_lock(&(pool->lock)) != 0) { return threadpool_lock_failure; } /* 计算下一个可以存储 task 的位置 */ next = pool->tail + 1; next = (next == pool->queue_size) ? 0 : next; do { /* Are we full ? */ /* 检查是否任务队列满 */ if(pool->count == pool->queue_size) { err = threadpool_queue_full; break; } /* Are we shutting down ? */ /* 检查当前线程池状态是否关闭 */ if(pool->shutdown) { err = threadpool_shutdown; break; } /* Add task to queue */ /* 在 tail 的位置放置函数指针和参数,添加到任务队列 */ pool->queue[pool->tail].function = function; pool->queue[pool->tail].argument = argument; /* 更新 tail 和 count */ pool->tail = next; pool->count += 1; /* pthread_cond_broadcast */ /* * 发出 signal,表示有 task 被添加进来了 * 如果由因为任务队列空阻塞的线程,此时会有一个被唤醒 * 如果没有则什么都不做 */ if(pthread_cond_signal(&(pool->notify)) != 0) { err = threadpool_lock_failure; break; } /* * 这里用的是 do { ... } while(0) 结构 * 保证过程最多被执行一次,但在中间方便因为异常而跳出执行块 */ } while(0); /* 释放互斥锁资源 */ if(pthread_mutex_unlock(&pool->lock) != 0) { err = threadpool_lock_failure; } return err;}int threadpool_destroy(threadpool_t *pool, int flags){ int i, err = 0; if(pool == NULL) { return threadpool_invalid; } /* 取得互斥锁资源 */ if(pthread_mutex_lock(&(pool->lock)) != 0) { return threadpool_lock_failure; } do { /* Already shutting down */ /* 判断是否已在其他地方关闭 */ if(pool->shutdown) { err = threadpool_shutdown; break; } /* 获取指定的关闭方式 */ pool->shutdown = (flags & threadpool_graceful) ? graceful_shutdown : immediate_shutdown; /* Wake up all worker threads */ /* 唤醒所有因条件变量阻塞的线程,并释放互斥锁 */ if((pthread_cond_broadcast(&(pool->notify)) != 0) || (pthread_mutex_unlock(&(pool->lock)) != 0)) { err = threadpool_lock_failure; break; } /* Join all worker thread */ /* 等待所有线程结束 */ for(i = 0; i < pool->thread_count; i++) { if(pthread_join(pool->threads[i], NULL) != 0) { err = threadpool_thread_failure; } } /* 同样是 do{...} while(0) 结构*/ } while(0); /* Only if everything went well do we deallocate the pool */ if(!err) { /* 释放内存资源 */ threadpool_free(pool); } return err;}int threadpool_free(threadpool_t *pool){ if(pool == NULL || pool->started > 0) { return -1; } /* Did we manage to allocate ? */ /* 释放线程 任务队列 互斥锁 条件变量 线程池所占内存资源 */ if(pool->threads) { free(pool->threads); free(pool->queue); /* Because we allocate pool->threads after initializing the mutex and condition variable, we're sure they're initialized. Let's lock the mutex just in case. */ pthread_mutex_lock(&(pool->lock)); pthread_mutex_destroy(&(pool->lock)); pthread_cond_destroy(&(pool->notify)); } free(pool); return 0;}static void *threadpool_thread(void *threadpool){ threadpool_t *pool = (threadpool_t *)threadpool; threadpool_task_t task; for(;;) { /* Lock must be taken to wait on conditional variable */ /* 取得互斥锁资源 */ pthread_mutex_lock(&(pool->lock)); /* Wait on condition variable, check for spurious wakeups. When returning from pthread_cond_wait(), we own the lock. */ /* 用 while 是为了在唤醒时重新检查条件 */ while((pool->count == 0) && (!pool->shutdown)) { /* 任务队列为空,且线程池没有关闭时阻塞在这里 */ pthread_cond_wait(&(pool->notify), &(pool->lock)); } /* 关闭的处理 */ if((pool->shutdown == immediate_shutdown) || ((pool->shutdown == graceful_shutdown) && (pool->count == 0))) { break; } /* Grab our task */ /* 取得任务队列的第一个任务 */ task.function = pool->queue[pool->head].function; task.argument = pool->queue[pool->head].argument; /* 更新 head 和 count */ pool->head += 1; pool->head = (pool->head == pool->queue_size) ? 0 : pool->head; pool->count -= 1; /* Unlock */ /* 释放互斥锁 */ pthread_mutex_unlock(&(pool->lock)); /* Get to work */ /* 开始运行任务 */ (*(task.function))(task.argument); /* 这里一个任务运行结束 */ } /* 线程将结束,更新运行线程数 */ pool->started--; pthread_mutex_unlock(&(pool->lock)); pthread_exit(NULL); return(NULL);}
- 【源码剖析】threadpool —— 基于 pthread 实现的简单线程池
- 【源码剖析】threadpool —— 基于 pthread 实现的简单线程池
- 【源码剖析】threadpool —— 基于 pthread 实现的简单线程池
- Nodejs事件引擎libuv源码剖析之:高效线程池(threadpool)的实现
- 基于pthread的线程池,C++实现
- 线程池(ThreadPool)——简单描述
- 线程池—ThreadPool
- C#异步编程的实现方式(3)——ThreadPool线程池
- threadpool——python线程池
- Pthread实现面向对象的线程处理简单框架
- muduo源码分析:线程池类ThreadPool
- Linux下调用pthread库实现简单线程池
- 【源码剖析】MemoryPool —— 简单高效的内存池 allocator 实现
- 【源码剖析】MemoryPool —— 简单高效的内存池 allocator 实现
- ThreadPool 线程池的使用
- [并发并行]_[pthread]_[线程池的简单设计与实现]
- 简单看看ThreadPool的源码以及从中看出线程间传值的另一种方法
- Java ThreadPool源码简单的解析
- model转json
- 【qscoj】喵哈哈村的嘟嘟熊魔法(3)
- (OK) ping6 ipv6-address
- Kubernetes编排工具-helm中使用grpc
- 假如你想成为全栈工程师…
- 【源码剖析】threadpool —— 基于 pthread 实现的简单线程池
- 62. Unique Paths
- Non-abundant sums
- 系统恢复技术-内核出现问题,如何修复
- 排序算法06:快速排序
- 二叉树的实现-Huffman树-摘自数据结构实现java版本(个人笔记整理)
- javascript 用户代理字符串检测技术-
- BeanFactory和ApplicationContext的介绍
- Windows 7+Sublime Text 3配置C/C++开发环境