每日一搏 | 细说 Python logging

细说 Python logging

可在 我的博客 阅读更多内容

最近有个需求是把以前字符串输出的log 改为json 格式,看了别人的例子,还是有些比较茫然,索性就把logging 整个翻了一边,做点小总结.

初看log

在程序中, log 的用处写代码的你用你知道,log 有等级,DEBUG, INFO,...之类,还会记录时间,log 发生的位置,在Python 中用的多的就是logging 这个标准库中的包了.当打log 的时候究竟发生了什么? 是如何把不同级别的log 输出到不同文件里,还能在控制台输出.......

最简单的用法

` import logging logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG) logging.debug('This message should go to the log file') logging.info('So should this') logging.warning('And this, too')

`

Logging Levels

|Level |Numeric value| |----------|----------| |CRITICAL| 50| |ERROR |40| |WARNING |30| |INFO| 20| |DEBUG |10| |NOTSET |0| 共有几个等级, 每个等级对应一个Int 型整数 ,每个等级都会有一个方法与之对应,这样输出的内容就有了不同的等级.

logger 流程,

整个过程,还是不是很详细,贴个图吧, 现在看还太早,也说不清真个过程到底发生了什么,先放着,回头来看会比较好懂.

读代码

代码结构

logging 在源码中有三个文件,结构如下:

` ├── config.py ├── handlers.py └── __init__.py

`

objects

LogRecord Objects

每一次log 都会实例化一个Record 对象，这个对象有很多属性,最后对LogRecord 做一下format 就输出了，格式化的log ，里面就基本就是这个对象的属性了。 ``` class LogRecord(object): def init (self, name, level, pathname, lineno, msg, args, exc info, func=None): ct = time.time() self.name = name self.msg = msg if (args and len(args) == 1 and isinstance(args[0], collections.Mapping) and args[0]): args = args[0] self.args = args self.levelname = getLevelName(level) self.levelno = level self.pathname = pathname try: self.filename = os.path.basename(pathname) self.module = os.path.splitext(self.filename)[0] except (TypeError, ValueError, AttributeError): self.filename = pathname self.module = "Unknown module" self.exc info = exc info self.exc text = None # used to cache the traceback text self.lineno = lineno self.funcName = func self.created = ct self.msecs = (ct - long(ct)) * 1000 self.relativeCreated = (self.created - startTime) * 1000 if logThreads and thread: self.thread = thread.get ident() self.threadName = threading.current thread().name else: self.thread = None self.threadName = None if not logMultiprocessing: self.processName = None else: self.processName = 'MainProcess' mp = sys.modules.get('multiprocessing') if mp is not None: try: self.processName = mp.current process().name except StandardError: pass if logProcesses and hasattr(os, 'getpid'): self.process = os.getpid() else: self.process = None

def __str__(self):     return '<LogRecord: %s, %s, %s, %s, "%s">'%(self.name, self.levelno,         self.pathname, self.lineno, self.msg)  def getMessage(self):      pass

``` 看代码就发现， 这个类没做什么事情，就是一个model 而已, 有一个得到msg 的方法

Formatter Objects

Formatter 就是对Record 专门格式化的对象，它有一个format 方法，我们实现这个方法就能 做到不同的输出，我的需求是做json 格式的log 其实关键就在写一个Formatter 就好了 ```

class Formatter(object): converter = time.localtime

def __init__(self, fmt=None, datefmt=None):     if fmt:         self._fmt = fmt     else:         self._fmt = "%(message)s"     self.datefmt = datefmt  def formatTime(self, record, datefmt=None):     pass   def formatException(self, ei):    pass  def usesTime(self):     return self._fmt.find("%(asctime)") >= 0  def format(self, record):    pass

``` 删掉源代码中的实现细节，这个类里面主要的是format 方法，这是默认最基本的Formater ，还有专门对exception ，时间做格式化的方法。具体是哪个，看方法名就很清楚了，具体每个方法怎么实现的，一眼也就懂了。fmt 是制定格式化的，具体怎么指定在最基础的用法中就有例子，datefmt 是对时间格式的指定。

Filter Objects

这个类是Logger 和Handler 的基类，主要有一个Filter 方法，和一个filters 属性

Handler Objects

叫Handler 的类还真的不少,在SocketServer 中也有看到,具体的功能都在Handler 中.在这里,组合所有的Formatter ,和控制log 的输出的方向，继承自Filter. ``` def init (self, level=NOTSET): Filterer. init (self) self. name = None self.level = checkLevel(level) self.formatter = None _addHandlerRef(self) self.createLock()

``` 在 init 方法中看到，Handler 也有一个属性，通过把自身的属性和LogRecord 的level对比来决定是否处理这个LogRecord 的。每个Handler 都有一个Formatter 属性，其实就是上面介绍的Formatter 。Handler 就是来控制LogRecord 和Formatter 的，它还可以控制输出的方式，在后面会有， StreamHandler , FileHandler 等。通过名称也就能明白具体能干什么，这就是编程取名的智慧。

Logger Objects

这个类通常会通过 getLogger() 或者 getLogger(name) 来得到,不会直接new 一个出来.它会有 info(msg, args, kwargs) , warn(msg, args, kwargs)**等方法,

` def __init__(self, name, level=NOTSET): Filterer.__init__(self) self.name = name self.level = _checkLevel(level) self.parent = Noneou self.handlers = [] self.disabled = 0

从**__init__**方法中能看到**handlers **属性,这是一个list ，每个LogRecord 通过Handlers 不同的handlers 就能以不同的格式输出到不同的地方了。每个**Logger** 可以通过**addHandler(hdlr)**方法来添加各种Handler, 知道这些你就基本可以随意定制化了 下面就是我实现的json 格式的Formater，支持控制台颜色变化，当然前提是你的控制终端支持(Ubuntu14.04测试通过) ``

import re import logging import socket import json import traceback import datetime import time

try: from collections import OrderedDict except ImportError: pass

RESERVED ATTRS = ( 'args', 'asctime', 'created', 'exc info', 'exc text', 'filename', 'funcName', 'levelname', 'levelno', 'lineno', 'module', 'msecs', 'message', 'msg', 'name', 'pathname', 'process', 'processName', 'relativeCreated', 'stack info', 'thread', 'threadName')

RESERVED ATTR HASH = dict(zip(RESERVED ATTRS, RESERVED ATTRS))

COLORS ={ 'HEADER' : '/033[95m', 'INFO' : '/033[94m', 'DEBUG' : '/033[92m', 'WARNING' : '/033[93m', 'ERROR' : '/033[91m', 'ENDC' : '/033[0m', }

def merge record extra(record, target, reserved=RESERVED ATTR HASH): for key, value in record. dict .items(): if (key not in reserved and not (hasattr(key, "startswith") and key.startswith('_'))): target[key] = value return target

def get host info(): host name = '' local ip = '' try: host name = socket.gethostname() local ip = socket.gethostbyname(host_name) except Exception, e: pass

return host_name, local_ip

class JsonFormatterBase(logging.Formatter):

def __init__(self,  *args, **kwargs):      logging.Formatter.__init__(self, *args, **kwargs)     self._required_fields = self.parse()     self._skip_fields = dict(zip(self._required_fields,self._required_fields))     self._skip_fields.update(RESERVED_ATTR_HASH) def parse(self):     standard_formatters = re.compile(r'/((.+?)/)', re.IGNORECASE)     return standard_formatters.findall(self._fmt)   def add_fields(self, record ):     log_record = {}      for field in self._required_fields:         log_record[field] = record.__dict__.get(field)      host_name , local_ip = get_host_info()      log_record[u'@hostName'] = host_name     log_record[u'@localIp'] = local_ip     return log_record      #merge_record_extra(record, log_record, reserved=self._skip_fields)   def process_log_record(self, log_record):     """     Override this method to implement custom logic     on the possibly ordered dictionary.     """      try:         new_record = OrderedDict()     except Exception, e:         return log_record      key_list = [         'asctime',         'levelname',         '@hostName',         '@localIp',         'threadName',         'thread',         'name',         'pathname',         'lineno',         'message',     ]     for k in key_list:         new_record[k] = log_record.get(k)     new_record.update(log_record)     return new_record  def jsonify_log_record(self, log_record):     """Returns a json string of the log record."""      return json.dumps(log_record, ensure_ascii=False)   def format_col(self, message_str, level_name):     """      是否需要颜色     """     return  message_str  def formatTime(self, record, datefmt=None):     ct = self.converter(record.created)     if datefmt:         s = time.strftime(datefmt, ct)     else:         t = time.strftime("%Y-%m-%d %H:%M:%S", ct)         s = "%s.%03d" % (t, record.msecs)     return s  def format(self, record):       if isinstance(record.msg, dict):         record.message = record.msg      elif isinstance(record.msg, list) or isinstance(record.msg, tuple):         record.message = record.msg      elif isinstance(record.msg, basestring):         record.message = record.getMessage().split('/n')      elif isinstance(record.msg, Exception):         record.message = traceback.format_exc(record.msg).split('/n')      else :         record.message = repr(record.msg)      if "asctime" in self._required_fields:         record.asctime = self.formatTime(record, self.datefmt)      #     # if record.exc_info and not message_dict.get('exc_info'):     #     message_dict['message'] = traceback.format_exception(*record.exc_info)     log_record = self.add_fields(record)     log_record = self.process_log_record(log_record)     message_str  = self.jsonify_log_record(log_record)     message_str = self.format_col(message_str, level_name=record.levelname)     return message_str

class ConsoleFormater(JsonFormatterBase):

def __init__(self, *args, **kwargs):     JsonFormatterBase.__init__(self, *args, **kwargs)  def format_col(self, message_str, level_name):     if level_name in COLORS.keys():         message_str = COLORS.get(level_name) + message_str + COLORS.get('ENDC')     return message_str  def jsonify_log_record(self, log_record):     return json.dumps(log_record, ensure_ascii=False, indent=4)

class JsonFileFormater(JsonFormatterBase):

def __init__(self, *args, **kewars):     JsonFormatterBase.__init__(self, *args, **kewars)  def jsonify_log_record(self, log_record):     return json.dumps(log_record, ensure_ascii=False)

```

配置

很多时候我们并不是这样自己去实现一些Handler ，Formater ,之类的代码，用logging 提供的config 就能做到了，如何写config下面举个例子解释下， ```

SC LOGGING CONF = { "version": 1, "disable existing loggers": False, "formatters": { "simple": { "format": "%(asctime)s [%(levelname)s] [%(threadName)s:%(thread)d] [%(name)s:%(lineno)d] - %(message)s" } },

"handlers": {     "console": {         "class": "logging.StreamHandler",         "level": "DEBUG",         "formatter": "simple",         "stream": "ext://sys.stdout"     },     "info_file_handler": {         "class": "logging.handlers.RotatingFileHandler",         "level": "INFO",         "formatter": "simple",         "filename": PATH + "info-" + date.today().isoformat() + ".log",         "maxBytes": 10485760,         "backupCount": 20,         "encoding": "utf8"     },     "error_file_handler": {         "class": "logging.handlers.RotatingFileHandler",         "level": "ERROR",         "formatter": "simple",         "filename": PATH + "errors-" + date.today().isoformat() + ".log",         "maxBytes": 10485760,         "backupCount": 20,         "encoding": "utf8"     } },     "": {         "level": "INFO",         "handlers": ["console", "info_file_handler", "error_file_handler"]     } }

}

` 首先定义了一个formater 叫simaple , 然后定义了三个Handler ,分别是输出到控制台，输出到文件和info,error的。

logging.config.dictConfig(CONFIG.SC_LOGGING_CONF) ``

做的事情，不需要写很多代码也能定制个性化的log.。