python and the logger

ramdyne — Sun, 11 Apr 2021 20:44:24 GMT

I've been working with logging in Python for the last few weeks. For some reason I ended up having to log to a MySQL (MariaDB?) server and I hadn't done that before. Turns out that a lot of people want to log MySQL queries to the same logging server as Python, but not a lot of people want to log to a MySQL server. The Python logger is actually quite a nicely documented piece of software if your needs are basic, but want something more and you might end up between a suggestion and a hard place...

Next to logging to MySQL I also wanted to configure the loger using code, not an ini file or a YAML file. Enter DictConfig. The example in the documentation is massive, does much more than anyone needs and is too complicated for when you are starting out with logging. So here's a basic example:

logging_config = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        },
    },
    'handlers': {
        'default_handler': {
            'level': 'DEBUG',
            'formatter': 'standard',
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout',
        },
    },
    'loggers': {
        '': {
            'level': 'NOTSET',
            'handlers' : ['default_handler']
        },
    },
}

import logging.config
logging.config.dictConfig(logging_config)
logger = logging.getLogger(os.path.basename(__file__))
# Now you can use logger to log
logger.error('Panic')

This should be sufficient for a simple logger to stdout (the CLI, console, DOS box whatever you call it). See the documentation if you want to change something or make it more complicated.

When you want to log to a MySQL database, you need to write your own custom handler class, inheriting from logging.Handler. Or you could google for an existing MySQLHandler, like this one I found on Github.

I did bring that handler up to Python 3 specs, and followed whatever Pycharm wanted me to fix on this code.

The one "big" thing I did, was to change the constructor of the handler class to the following:

def __init__(self, host: str, port: int, db_user: str, db_password: str, db_name: str):
    pass

I'll leave the implementation of this function up to you, but the signature is really important when you want to use this custom handler in combination with DictConfig.

logging_config = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'default_handler': {
            'class': 'my_custom_handler.MyCustomHandler',
            'level': 'DEBUG',
            'formatter': 'standard',
            'host': "localhost",
            'port': 3306,
            'db_user': "user",
            'db_password': "password",
            'db_name': "Log",
        }
    },
    'loggers': {
        '': {
            'level': 'NOTSET',
            'handlers' : ['default_handler']
        },
    },
}

import logging.config
logging.config.dictConfig(logging_config)
logger = logging.getLogger(os.path.basename(__file__))
# Now you can use logger to log
logger.error('Panic')

The custom handler in this example has the class name MyCustomhandler and can be found in the my_custom_handler module. As you can see the parameters for a custom handler in DictConfig reflect the parameters of the constructor of the custom handler. Note also that a database handler doesn't really need a formatter when your datastructure looks like the format of log messages.

ramdyne (Posts about syslog)

python and the logger