Real-time metrics for nginx server

Overview

ngxtop - real-time metrics for nginx server (and others)

ngxtop parses your nginx access log and outputs useful, top-like, metrics of your nginx server. So you can tell what is happening with your server in real-time.

ngxtop is designed to run in a short-period time just like the top command for troubleshooting and monitoring your Nginx server at the moment. If you need a long running monitoring process or storing your webserver stats in external monitoring / graphing system, you can try Luameter.

ngxtop tries to determine the correct location and format of nginx access log file by default, so you can just run ngxtop and having a close look at all requests coming to your nginx server. But it does not limit you to nginx and the default top view. ngxtop is flexible enough for you to configure and change most of its behaviours. You can query for different things, specify your log and format, even parse remote Apache common access log with ease. See sample usages below for some ideas about what you can do with it.

Installation

pip install ngxtop

Note: ngxtop is primarily developed and tested with python2 but also supports python3.

Usage

Usage:
    ngxtop [options]
    ngxtop [options] (print|top|avg|sum) <var>
    ngxtop info

Options:
    -l <file>, --access-log <file>  access log file to parse.
    -f <format>, --log-format <format>  log format as specify in log_format directive.
    --no-follow  ngxtop default behavior is to ignore current lines in log
                     and only watch for new lines as they are written to the access log.
                     Use this flag to tell ngxtop to process the current content of the access log instead.
    -t <seconds>, --interval <seconds>  report interval when running in follow mode [default: 2.0]

    -g <var>, --group-by <var>  group by variable [default: request_path]
    -w <var>, --having <expr>  having clause [default: 1]
    -o <var>, --order-by <var>  order of output for default query [default: count]
    -n <number>, --limit <number>  limit the number of records included in report for top command [default: 10]
    -a <exp> ..., --a <exp> ...  add exp (must be aggregation exp: sum, avg, min, max, etc.) into output

    -v, --verbose  more verbose output
    -d, --debug  print every line and parsed record
    -h, --help  print this help message.
    --version  print version information.

    Advanced / experimental options:
    -c <file>, --config <file>  allow ngxtop to parse nginx config file for log format and location.
    -i <filter-expression>, --filter <filter-expression>  filter in, records satisfied given expression are processed.
    -p <filter-expression>, --pre-filter <filter-expression> in-filter expression to check in pre-parsing phase.

Samples

Default output

$ ngxtop
running for 411 seconds, 64332 records processed: 156.60 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|   64332 |         2775.251 | 61262 |  2994 |    71 |     5 |

Detailed:
| request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /abc/xyz/xxxx                            |   20946 |          434.693 | 20935 |     0 |    11 |     0 |
| /xxxxx.json                              |    5633 |         1483.723 |  5633 |     0 |     0 |     0 |
| /xxxxx/xxx/xxxxxxxxxxxxx                 |    3629 |         6835.499 |  3626 |     0 |     3 |     0 |
| /xxxxx/xxx/xxxxxxxx                      |    3627 |        15971.885 |  3623 |     0 |     4 |     0 |
| /xxxxx/xxx/xxxxxxx                       |    3624 |         7830.236 |  3621 |     0 |     3 |     0 |
| /static/js/minified/utils.min.js         |    3031 |         1781.155 |  2104 |   927 |     0 |     0 |
| /static/js/minified/xxxxxxx.min.v1.js    |    2889 |         2210.235 |  2068 |   821 |     0 |     0 |
| /static/tracking/js/xxxxxxxx.js          |    2594 |         1325.681 |  1927 |   667 |     0 |     0 |
| /xxxxx/xxx.html                          |    2521 |          573.597 |  2520 |     0 |     1 |     0 |
| /xxxxx/xxxx.json                         |    1840 |          800.542 |  1839 |     0 |     1 |     0 |

View top source IPs of clients

$ ngxtop top remote_addr
running for 20 seconds, 3215 records processed: 159.62 req/sec

top remote_addr
| remote_addr     |   count |
|-----------------+---------|
| 118.173.177.161 |      20 |
| 110.78.145.3    |      16 |
| 171.7.153.7     |      16 |
| 180.183.67.155  |      16 |
| 183.89.65.9     |      16 |
| 202.28.182.5    |      16 |
| 1.47.170.12     |      15 |
| 119.46.184.2    |      15 |
| 125.26.135.219  |      15 |
| 125.26.213.203  |      15 |

List 4xx or 5xx responses together with HTTP referer

$ ngxtop -i 'status >= 400' print request status http_referer
running for 2 seconds, 28 records processed: 13.95 req/sec

request, status, http_referer:
| request   |   status | http_referer   |
|-----------+----------+----------------|
| -         |      400 | -              |

Parse apache log from remote server with common format

$ ssh [email protected]_server tail -f /var/log/apache2/access.log | ngxtop -f common
running for 20 seconds, 1068 records processed: 53.01 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    1068 |        28026.763 |  1029 |    20 |    19 |     0 |

Detailed:
| request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /xxxxxxxxxx                              |     199 |        55150.402 |   199 |     0 |     0 |     0 |
| /xxxxxxxx/xxxxx                          |     167 |        47591.826 |   167 |     0 |     0 |     0 |
| /xxxxxxxxxxxxx/xxxxxx                    |      25 |         7432.200 |    25 |     0 |     0 |     0 |
| /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxxx      |      22 |          698.727 |    22 |     0 |     0 |     0 |
| /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxx       |      19 |         7431.632 |    19 |     0 |     0 |     0 |
| /xxxxx/xxxxx/                            |      18 |         7840.889 |    18 |     0 |     0 |     0 |
| /xxxxxxxx/xxxxxxxxxxxxxxxxx              |      15 |         7356.000 |    15 |     0 |     0 |     0 |
| /xxxxxxxxxxx/xxxxxxxx                    |      15 |         9978.800 |    15 |     0 |     0 |     0 |
| /xxxxx/                                  |      14 |            0.000 |     0 |    14 |     0 |     0 |
| /xxxxxxxxxx/xxxxxxxx/xxxxx               |      13 |        20530.154 |    13 |     0 |     0 |     0 |
Comments
  • ImportError: No module named 'config_parser'

    ImportError: No module named 'config_parser'

    After installing when I am trying to run ngxtop its giving me error

    Traceback (most recent call last):
      File "/usr/bin/ngxtop", line 7, in <module>
        from ngxtop.ngxtop import main
      File "/usr/lib64/python3.4/site-packages/ngxtop/ngxtop.py", line 77, in <module>
        from config_parser import detect_log_config, detect_config_path, extract_variables, build_pattern
    ImportError: No module named 'config_parser'
    
    opened by wonderer007 8
  • Support for reading from stdin and dumping db to a file

    Support for reading from stdin and dumping db to a file

    If the only access you have to the log is via cat or tail from ssh connection, the script will need reading from stdin. And maybe you want backup in a file the data you're printing.

    opened by mcortizo 8
  • ngxtop no data gathering from access log

    ngxtop no data gathering from access log

    Can you help me what problem is? no data summary from access log

    ngxtop

    running for 42 seconds, 0 records processed: 0.00 req/sec

    Summary: | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | |---------+------------------+-------+-------+-------+-------| | 0 || 0 | 0 | 0 | 0 |

    Detailed: | request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | |----------------+---------+------------------+-------+-------+-------+-------|246

    opened by jk28189 3
  • --access-log doesn't seem to be working

    --access-log doesn't seem to be working

    ngxtop -l /var/log/nginx/access.log
    Traceback (most recent call last):
      File "/home/adys/bin/ngxtop", line 437, in <module>
        main(args)
      File "/home/adys/bin/ngxtop", line 420, in main
        total = process_log(source, pattern, processor, arguments)
      File "/home/adys/bin/ngxtop", line 325, in process_log
        total = processor.process(records)
      File "/home/adys/bin/ngxtop", line 272, in process
        for r in records:
      File "/home/adys/bin/ngxtop", line 206, in add_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 196, in map_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 196, in map_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 206, in add_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 206, in add_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 196, in map_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 245, in <genexpr>
        records = (m.groupdict() for m in matches if m is not None)
      File "/home/adys/bin/ngxtop", line 244, in <genexpr>
        matches = (pattern.match(l) for l in lines)
      File "/home/adys/bin/ngxtop", line 181, in follow
        with open(the_file) as f:
    IOError: [Errno 2] No such file or directory: 'logs/access.log'
    
    opened by jleclanche 3
  • Error

    Error

    Hi

    $ ngxtop info Traceback (most recent call last): File "/usr/local/bin/ngxtop", line 6, in from ngxtop.ngxtop import main File "/usr/local/lib/python3.6/dist-packages/ngxtop/ngxtop.py", line 77, in from config_parser import detect_log_config, detect_config_path, extract_variables, build_pattern ModuleNotFoundError: No module named 'config_parser'

    What's wrong/missing?

    opened by matt-matt1 2
  • Any way to filter the request_path ?

    Any way to filter the request_path ?

    Hello

    Is possible that any way to filter the request_path? For example I want to see analyze the just ".css" files (hit count /4xx 3xx 2xx errors)

    Are there any parameter ?

    opened by maxiceptr 2
  • No matching distribution found for sqlite-devel

    No matching distribution found for sqlite-devel

    I run the that command via ssh yum -y install python-pip; pip install ngxtop sqlite-devel

    And it says:

    Installed: python-pip.noarch 0:7.1.0-1.el7
    Complete! /usr/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning You are using pip version 7.1.0, however version 7.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting ngxtop /usr/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Downloading ngxtop-0.0.2-py2.py3-none-any.whl Collecting sqlite-devel Could not find a version that satisfies the requirement sqlite-devel (from versions: ) No matching distribution found for sqlite-devel

    I think, a sqlite-devel problem.

    P.S: Operation system is centos 7 64bit

    opened by maxiceptr 2
  • ngxtop work with buffered access_log ?

    ngxtop work with buffered access_log ?

    Hi I am playing around with ngxtop for my Centmin Mod Nginx project https://gist.github.com/centminmod/d613128cdb9eed86c9ec and was wondering do ngxtop live stats work with buffered access_log files ?

    I can not see any live stats but when i use --no-follow i do see previously logged data

    thanks

    opened by centminmod 2
  • Support multiple fields in --group-by

    Support multiple fields in --group-by

    This allows --group-by to take a comma-separated list of field names, and processes the field list before passing it to the SQLProcessor. When displaying the results, it displays as two separate fields.

    opened by amyreese 2
  • Multiple group by columns

    Multiple group by columns

    I'd like to show requests by hostname and request path, so I tried:

    ngxtop -g http_host,request_path
    

    which gives me

    running for 8 seconds, 0 records processed: 0.00 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |       0 |                  |     0 |     0 |     0 |     0 |
    
    Detailed:
    
    Traceback (most recent call last):
      File "/usr/bin/ngxtop", line 9, in <module>
        load_entry_point('ngxtop==0.0.1', 'console_scripts', 'ngxtop')()
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 446, in main
        process(args)
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 429, in process
        total = process_log(source, pattern, processor, arguments)
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 334, in process_log
        total = processor.process(records)
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 282, in process
        cursor.execute(insert, r)
    sqlite3.OperationalError: no such column: request_path
    

    So then I tried nginxtop -g http_host -g request_path, which just gives me back the usage information. Grouping by either http_host or request_path works fine, it's just the combination together that I can't seem to get working.

    opened by alanpearce 2
  • Got error messages in centos 6.5

    Got error messages in centos 6.5

    Centos 6.5

    Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)

    ./ngxtop.py
    Traceback (most recent call last):
      File "./ngxtop.py", line 440, in <module>
        main(args)
      File "./ngxtop.py", line 402, in main
        config = get_nginx_conf_path()
      File "./ngxtop.py", line 117, in get_nginx_conf_path
        proc = subprocess.Popen(['nginx', '-V'], stderr=subprocess.PIPE)
      File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
        errread, errwrite)
      File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
        raise child_exception
    OSError: [Errno 2] No such file or directory
    
    opened by jiajie999 2
  • [Looking for Help] Show all IPs that have accesses in the last 5 minutes

    [Looking for Help] Show all IPs that have accesses in the last 5 minutes

    Looking for some help to use ngxtop to show just the connections in the last 5 minutes and not all time.

    We're using this to gauge a live listener count for HLS streams. Using the following command works great, but can we make IPs drop off the list if there have been no requests in X amount of time?

    ngxtop -l /var/log/nginx/stream-HLS-access.log --group-by remote_addr

    opened by CFMCoromandel 0
  • go forward to ngxtop version 0.1.0.

    go forward to ngxtop version 0.1.0.

    1. add traff speed summery info
    2. add avg_request_time in each pannel
    3. modify: -g , --group-by group by variable [default: request_path,http_host,remote_addr]
    4. set version 0.1.0
    opened by pythonwood 1
  • Buffed logs

    Buffed logs

    Please add warning about buffered nginx access logs to readme. Because it's not obvious that buffered logs can make ngxtop displaying incorrect values.

    Thank you very much for such a good tool.

    opened by AlexanderMatveev 0
  • support inheritance log_format for child configs

    support inheritance log_format for child configs

    In case main nginx.conf uses include directive for child sites but log_format specified in nginx.conf tool couldn't capture log strings. It stops with Error: Incorrect format name set in config for access log file or produces nothing.

    This patch looks for log_format in main nginx.conf and applies it if needed.

    Known limitation: works only when site config is specified, not log.

    opened by IgorPyN 1
Releases(v0.0.3)
  • v0.0.3(Dec 21, 2020)

  • 0.0.2(Apr 9, 2014)

    Improvements:

    • Proper parsing of nginx configuration file with PyParsing: better detection of access log file and format, should work out of the box in most cases.
    • Get rid of reporting thread, use timer signal instead.
    • Use curses to display metrics instead of repeating clear screen.
    • Allow parsing of log from stdin and include 'combined' log format of Apache.
    Source code(tar.gz)
    Source code(zip)
Owner
Binh Le
Curious programmer, brave sysadmin. SRE @google, previously @cloudflare and @adatao.
Binh Le
Cobalt Strike random C2 Profile generator

Random C2 Profile Generator Cobalt Strike random C2 Profile generator Author: Joe Vest (@joevest) This project is designed to generate malleable c2 pr

Threat Express 482 Jan 08, 2023
🚴 Call stack profiler for Python. Shows you why your code is slow!

pyinstrument Pyinstrument is a Python profiler. A profiler is a tool to help you 'optimize' your code - make it faster. It sounds obvious, but to get

Joe Rickerby 5k Jan 01, 2023
System monitor - A python-based real-time system monitoring tool

System monitor A python-based real-time system monitoring tool Screenshots Installation Run My project with these commands pip install -r requiremen

Sachit Yadav 4 Feb 11, 2022
Tracy Profiler module for the Godot Engine

GodotTracy Tracy Profiler module for the Godot Engine git clone --recurse-submodules https://github.com/Pineapple/GodotTracy.git Copy godot_tracy fold

Pineapple Works 17 Aug 23, 2022
Call-graph profiling for TwinCAT 3

Twingrind This project brings profiling to TwinCAT PLCs. The general idea of the implementation is as follows. Twingrind is a TwinCAT library that inc

stefanbesler 10 Oct 12, 2022
Linux/OSX/FreeBSD resource monitor

Index Documents Description Features Themes Support and funding Prerequisites (Read this if you are having issues!) Dependencies Screenshots Installat

9k Jan 08, 2023
Prometheus instrumentation library for Python applications

Prometheus Python Client The official Python 2 and 3 client for Prometheus. Three Step Demo One: Install the client: pip install prometheus-client Tw

Prometheus 3.2k Jan 07, 2023
Prometheus integration for Starlette.

Starlette Prometheus Introduction Prometheus integration for Starlette. Requirements Python 3.6+ Starlette 0.9+ Installation $ pip install starlette-p

José Antonio Perdiguero 229 Dec 21, 2022
Was an interactive continuous Python profiler.

☠ This project is not maintained anymore. We highly recommend switching to py-spy which provides better performance and usability. Profiling The profi

What! Studio 3k Dec 27, 2022
ASGI middleware to record and emit timing metrics (to something like statsd)

timing-asgi This is a timing middleware for ASGI, useful for automatic instrumentation of ASGI endpoints. This was developed at GRID for use with our

Steinn Eldjárn Sigurðarson 99 Nov 21, 2022
GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

GoAccess What is it? GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through y

Gerardo O. 15.6k Jan 02, 2023
Exports osu! user stats to prometheus metrics for a specified set of users

osu! to prometheus exporter This tool exports osu! user statistics into prometheus metrics for a specified set of user ids. Just copy the config.json.

Peter Oettig 1 Feb 24, 2022
Scalene: a high-performance, high-precision CPU and memory profiler for Python

scalene: a high-performance CPU and memory profiler for Python by Emery Berger 中文版本 (Chinese version) About Scalene % pip install -U scalene Scalen

Emery Berger 138 Dec 30, 2022
Monitor Memory usage of Python code

Memory Profiler This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for pyth

3.7k Dec 30, 2022
A watch dog providing a piece in mind that your Chia farm is running smoothly 24/7.

Photo by Zoltan Tukacs on Unsplash Watchdog for your Chia farm So you've become a Chia farmer and want to maximize the probability of getting a reward

Martin Mihaylov 466 Dec 11, 2022
Output provisioning profiles in a diffable way

normalize-profile This tool reads Apple's provisioning profile files and produces reproducible output perfect for diffing. You can easily integrate th

Keith Smiley 8 Oct 18, 2022
Line-by-line profiling for Python

line_profiler and kernprof NOTICE: This is the official line_profiler repository. The most recent version of line-profiler on pypi points to this repo

OpenPyUtils 1.6k Dec 31, 2022
Display machine state using Python3 with Flask.

Flask-State English | 简体中文 Flask-State is a lightweight chart plugin for displaying machine state data in your web application. Monitored Metric: CPU,

622 Dec 18, 2022
Real-time metrics for nginx server

ngxtop - real-time metrics for nginx server (and others) ngxtop parses your nginx access log and outputs useful, top-like, metrics of your nginx serve

Binh Le 6.4k Dec 22, 2022
Watch your Docker registry project size, then monitor it with Grafana.

Watch your Docker registry project size, then monitor it with Grafana.

Nova Kwok 33 Apr 05, 2022