关于python logging模块日志记录与oss模块文件上传的问题

来源:互联网 发布:香港域名注册不备案 编辑:程序博客网 时间:2024/06/03 18:49

最近在做http使用POST请求上传图片到阿里云对象存储oss的时候发生一件很诡异的事情!
我的环境是python3.5+django11.6+gunicorn部署的web后台服务settings文件logging配置如下:

LOGGING = {    'version': 1,    'disable_existing_loggers': False,    'filters': {        'require_debug_false': {            '()': 'django.utils.log.RequireDebugFalse'        }    },    'formatters': {        'verbose': {            'format': '%(levelname)s %(asctime)s %(module)s '                      '%(process)d %(thread)d %(message)s'        },        'simple': {            'format': '%(message)s',        }    },    'handlers': {        'console': {            'level': 'DEBUG',            'class': 'logging.StreamHandler',            'formatter': 'verbose',        },        'report_data': {            'level': 'INFO',            'class': 'xxx.libs.logging.handlers.TimedRotatingFileHandlerMultiProcess',            'filename': '/var/log/xxx/info.log',            'formatter': 'simple'        },    },    'loggers': {        'django.security.DisallowedHost': {            'level': 'ERROR',            'handlers': ['console'],            'propagate': True        },        'info': {            'level': 'INFO',            'handlers': ['report_data'],        },        ...    }}

views.py日志记录部分代码如下:

@api_view(['POST'])@permission_classes((TokenHasReadWriteScope, ))def report_data_view(request, format=None):    report_header = request.META    if request.method == 'POST':        report_data = request.data        logger = logging.getLogger("info")        logger.info(report_data)        ...

按道理是POST请求提交数据将会被记录到/var/log/xxx/info.log中,实际上也确实正常运行。(虽然这样做有一点另类),但是加上文件上传功能之后就出问题了:
客户端图片上传部分代码如下:

def upload_images(task):    if task.success and len(task.result):        token_string = _get_oauth_token(local=True)        headers = {'Authorization': token_string, 'Image': 'True'}        for item in task.result:            data = {"id": item['id']}            files = {}            for image_url, image_path in item['files'].items():                files[image_url] = (open(image_path, 'rb'))            res = requests.post(AGENT_UPLOAD_URI, headers=headers, data=data, files=files)            if res.status_code == 201:                status = 2   # 表示上传成功            else:                status = 1   # 表示上传失败            update_uploaded_status(item['table_name'], item['course_id'], item['url_path_str'], status)

服务器端接收图片以及上传oss部分代码如下:

@api_view(['POST'])@permission_classes((TokenHasReadWriteScope, ))def upload_image_view(request, format=None):    if request.method == 'POST':        if 'HTTP_IMAGE' in report_header.keys():            if not request.FILES:                return Response(None, status=status.HTTP_200_OK)            else:                id = request.data['id'] # 形如"201709168888"                date = '-'.join([id[0:4], id[4:6], id[6:8]])                try:                    for url_path, fd in request.FILES.items():                        tmp_dir = os.path.join('/tmp/', get_hash_val(os.path.dirname(url_path)))                        if not os.path.exists(tmp_dir):                            os.mkdir(tmp_dir)                        file_path = os.path.join(tmp_dir, os.path.basename(url_path))                        with open(file_path, 'wb+') as f:                            for chrunk in fd.chunks():                                f.write(chrunk)                        upload_image(date, url_path, file_path)                    return Response(None, status=status.HTTP_201_CREATED)                except Exception as e:                    print("Unexpected error: {0}".format(e))                    ...def upload_image(date, key, file_path):    auth = oss2.Auth(ALIYUN_ACCESS_KEY_ID, ALIYUN_ACCESS_KEY_SECRET)    bucket = oss2.Bucket(auth, ALIYUN_OSS_SZ_ENDPOINT, OSS_IMAGE_BUCKET)    try:        oss2.resumable_upload(bucket, key, file_path,                              store=oss2.ResumableStore(root='/tmp'),                              multipart_threshold=100 * 1024,                              part_size=100 * 1024,                              num_threads=5)    except oss2.exceptions.OssError as e:        print("Upload image[{0}] faild. Details:{1}.".format(url_path, str(e)))

两个几乎完全没有联系的功能模块凑一块就搞出了幺蛾子,gunicorn日志中竟然莫名其妙记录了POST请求发送的数据!虽然问题不是很严重,但是久而久之文件大小会越来越大占用磁盘,而且总感觉像埋了地雷一样不让人舒坦。没办法只能一段一段注释代码再运行测试!最后发现下载写到文件但不上传oss是没有问题的,大胆猜测是oss文件上传导致http请求中的header变化了?因为文件上传是“Content-Type: multipart/form-data”,其他请求是“Content-Type: application/json”。或者环境变量发生改变之类的?解决方案是新增一个进程调用upload_image函数:

def upload_image_view(request, format=None):    ...    #upload_image(date, url_path, file_path)    p = multiprocessing.Process(target=upload_image, args=(date, url_path, file_path))    p.start()    ...

最后也没能去验证这个想法,写第一篇博客提醒自己留心注意这个问题同时记录一下关于python文件上传和以及oss模块的使用方法。
新手上路,请多多指教!

原创粉丝点击