The Python agent for Apache SkyWalking

Last update: Dec 12, 2022

Overview

SkyWalking Python Agent

SkyWalking-Python: The Python Agent for Apache SkyWalking, which provides the native tracing abilities for Python project.

SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.

Documentation

Official documentation
Blog about the Python Agent Profiling Feature

Installation Requirements

SkyWalking Python Agent requires SkyWalking 8.0+ and Python 3.6+.

If you would like to try out the latest features that are not released yet, please refer to this guide to build from sources.

Contributing

Before submitting a pull request or pushing a commit, please read our contributing and developer guide.

Contact Us

Submit an GitHub Issue by using [Python] as title prefix.
Mail list: [email protected]. Mail to [email protected], follow the reply to subscribe the mail list.
Join skywalking channel at Apache Slack. If the link is not working, find the latest one at Apache INFRA WIKI.
Twitter, ASFSkyWalking

License

Apache 2.0

Comments

feat: Add support for MeterReportService

Add support for MeterReportService

Behavior

This service runs along with the agent like other services and reports roughly every 20 seconds.

Syntactic sugar

Counter

c = Counter('c2', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time the with-wrapped codes consumed
with c.create_timer():
    # some codes may consume a certain time

c = Counter('c3', CounterMode.INCREMENT)
c.build()

# increase Counter c by num once counter_decorator_test gets called
@Counter.increase(name='c3', num=2)
def counter_decorator_test():
    # some codes

c = Counter('c4', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time counter_decorator_test consumed
@Counter.timer(name='c4')
def counter_decorator_test(s):
    # some codes may consume a certain time

Histogram

h = Histogram('h3', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time the with-wrapped codes consumed
with h.create_timer():
    # some codes may consume a certain time

h = Histogram('h2', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time histogram_decorator_test consumed
@Histogram.timer(name='h2')
def histogram_decorator_test(s):
    time.sleep(s)

Simple test

insert following codes below this line agent/__init__.py#L151

from skywalking.meter.gauge import Gauge
import gc

def generator_for_gc_g0():
    while(True):
        yield gc.get_stats()[0]['collected']

def generator_for_gc_g1():
    while(True):
        yield gc.get_stats()[1]['collected']

def generator_for_gc_g2():
    while(True):
        yield gc.get_stats()[2]['collected']


gc_g0_meter = Gauge("gc_g0", generator_for_gc_g0())
gc_g0_meter.build()

gc_g1_meter = Gauge("gc_g1", generator_for_gc_g1())
gc_g1_meter.build()

gc_g2_meter = Gauge("gc_g2", generator_for_gc_g2())
gc_g2_meter.build()

feature core

opened by jiang1997 19

fix: grpc timeout segment data loss

Please review this thoroughly because all I wanted to do was resend segments which failed to send due to a grpc timeout. But unexpectedly the behavior of grpc timeout changed, it doesn't time out anymore, the heartbeat seems to keep it alive now. I don't understand why this behavior changed so that is why I am requesting a thorough looking at this.
enhancement

opened by tom-pytel 18
Add http ignore by method
Added SW_HTTP_IGNORE_METHOD which will allow ignoring http operations by method (GET, POST, HEAD, OPTIONS, etc...)

Set default SW_AGENT_MAX_BUFFER_SIZE back to 10000 from 1000, which turns out to be too small for python (many unnecessarily dropped spans).

feature
opened by tom-pytel 14
feat: report metrics related to pvm

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

cpu utilization of the current process

thread_active_count

number of threads invoked by the current process and its children

Memory

total_mem_utilization

process_mem_utilization

memory utilization of the current process

GC

gc_g0

number of objects of generation 0

gc_g1

number of objects of generation 1

gc_g2

number of objects of generation 2

gc_time

For now, MeterReportService is available with gRPC only. So does the e2e test. I will implement MeterReportService and enable e2e test for Kafka later.

Welcome any advice about naming.
feature core

opened by jiang1997 12
[Fix][Plugin] sw_flask general exceptions handled
sw_flask fix will handle errors like returning the wrong type from a handler or other internal errors.

Updated StackedSpan to track depth, and made depth variable instance instead of class level (this was a bug).

Changed how SpanContext decides when all spans finished to write Segment data, now counts span start / stops which should work better across different async scenarios.

Changed new_exit_span() with span.inject() to work simpler like the NodeJS agent, now plugins inject directly themselves if they need to.

Removed carrier from plugins which didn't actually use it.

bug plugin
opened by tom-pytel 12
Add plugin for bottle
[x] Add a test case for the new plugin

[x] Add a CHANGELOG entry for the new plugin

[x] Add a component id in the main repo

[x] Add a logo in the UI repo

[x] Rebuild the requirements.txt by running tools/env/build_requirements_(linux|windows).sh

[x] Rebuild the Plugins.md documentation by running make doc-gen

plugin
opened by jiang1997 11
Enable HTTP log reporting

This is a work in progress.

Now reporting logs in JSON through HTTP to http://oap/v3/logs does work.

But, I'm not sure if the oap/v3/logs endpoint is for such usage(It seems designed for fluent-bit batch reporting?). Reporting logs one by one through HTTP may not be ideal in terms of performance. I'm not sure whether the Java agent only implements gRPC reporter intentionally out of this reason.

Please advise.

Signed-off-by: YihaoChen [email protected]
feature

opened by Superskyyy 11
Feature: collect and report logs
This is a OSPP Summer 2021 project supervised by @Humbertzhang | apache/skywalking#7118

The feature implements optional log reporter functionalities in alignment with the SkyWalking Java agent.

Intercepts logs from Python logging module.

Reports logs via a new temporary gRPC channel(to be removed in the future).

Supports unformatted/ formatted logs with custom layout.

Supports custom logging level threshold.

Bumps up submodule to support log collection protocols.

Bumps up skywalking-eyes and adds a config entry to ignore .gitignore during license checks.

feature core
opened by Superskyyy 11
Introduce better coding style

This PR migrates existing usage of old-style string manipulation (%, .format and +) to f-strings for better clarity and some performance boost.

Adds stricter coding style.

Optimizes debug messages for performance.

Signed-off-by: Superskyyy [email protected]
enhancement chore

opened by Superskyyy 10
Add E2E coverage for Trace and Logging
Change Fastapi to FastAPI component name (sync with OAP change)

Add E2E coverage for Python agent (Trace and Logging), profiling is not tested for now.

Renamed a build variable to remove possible mix-ups with CLI environment Var.

Enhance plugin test to reuse images.

Combined CI to pass if changes are non-essential to agent and workflows.

Note: There's a flaky test due to a possible {{- contains }} issue in the Infra-e2e verifier, the logs seem to be queried in an unstable order by the CLI where later log often placed first in log list, combining with the contains bug, it becomes flaky.

A time.sleep() workaround makes sure the log arrives in clear order which passes the e2e for now.
test
opened by Superskyyy 9
[Enhancement] Async tasks should work 100%

These changes should address all remaining async-related problems not handled in #88. The spans list has been moved from an instance variable in SpanContext to a global task-local variable which allows "forking" it for new subtasks so that multiple concurrent async subtasks don't interfere with one another. spans is now a top-level module var because Python stipulates that ContextVar should be such and only one context exists at any given time so it is essentially a singleton anyway.
enhancement core

opened by tom-pytel 9
Enhance redis plugin to adapt Virtual Cache

Resolves: #10212
enhancement

opened by Jedore 2

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)
What's Changed

spans correctly reference finished parents by @tom-pytel in https://github.com/apache/skywalking-python/pull/161

Refactor current Python agent docs to serve on SkyWalking official website by @Superskyyy in https://github.com/apache/skywalking-python/pull/162

fix broken url by @JaredTan95 in https://github.com/apache/skywalking-python/pull/163

Remove docs from main README.md - Add website doc links. by @Superskyyy in https://github.com/apache/skywalking-python/pull/164

Refactor SkyWalking Python to use the CLI for CI instead of legacy setup by @Superskyyy in https://github.com/apache/skywalking-python/pull/165

Remove places that mentions Python 3.5 support due to EOL by @Superskyyy in https://github.com/apache/skywalking-python/pull/166

Cleanup and Python 3.10 support by @Superskyyy in https://github.com/apache/skywalking-python/pull/167

filled out rest of psycopg 3 plugin by @tom-pytel in https://github.com/apache/skywalking-python/pull/168

Move flake configs all together by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/169

Enable faster CI by categorical parallelism by @Superskyyy in https://github.com/apache/skywalking-python/pull/170

Introduce better coding style by @Superskyyy in https://github.com/apache/skywalking-python/pull/171

Minimize exclude in lint by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/173

Introduce another set of flake8 extensions by @Superskyyy in https://github.com/apache/skywalking-python/pull/174

fix aiohttp outgoing request url by @tom-pytel in https://github.com/apache/skywalking-python/pull/175

bugfix: flask + nginx got KeyError: 'REMOTE_PORT' in sw_flask.py plugin by @VxCoder in https://github.com/apache/skywalking-python/pull/176

Add missing ending quote by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/177

updated deprecated configuration by @doddi in https://github.com/apache/skywalking-python/pull/179

Add plugin for mysqlclient by @katelei6 in https://github.com/apache/skywalking-python/pull/178

Fix sw-rabbitmq TypeError when there are no headers by @dcryans in https://github.com/apache/skywalking-python/pull/182

Fix agent bootup traceback not shown in sw-python CLI by @Superskyyy in https://github.com/apache/skywalking-python/pull/183

Add plugin for FastAPI by @katelei6 in https://github.com/apache/skywalking-python/pull/181

Support CI container logs by @Superskyyy in https://github.com/apache/skywalking-python/pull/185

Update mySQL plugin to support two different parameter keys. by @katelei6 in https://github.com/apache/skywalking-python/pull/186

add the code > 400 is error by @katelei6 in https://github.com/apache/skywalking-python/pull/187

Doc: add how to use with uwsgi by @arcosx in https://github.com/apache/skywalking-python/pull/188

Fix: right doc link for How To Use With uWSGI by @arcosx in https://github.com/apache/skywalking-python/pull/189

The local log stack depth is not truncated by @wzy960520 in https://github.com/apache/skywalking-python/pull/190

Revert "The local log stack depth is not truncated (#190)" by @Superskyyy in https://github.com/apache/skywalking-python/pull/191

Fix sw_formatter wrongly set cache that impacts user handlers by @Superskyyy in https://github.com/apache/skywalking-python/pull/192

Fix typo that cause failure in loading user sitecustomize.py by @Superskyyy in https://github.com/apache/skywalking-python/pull/193

Drop support for Flask 1.x due to EOL & Jinja2 issue by @Superskyyy in https://github.com/apache/skywalking-python/pull/195

Update agent docs, changelogs and PR template. by @Superskyyy in https://github.com/apache/skywalking-python/pull/197

Fix Python shown as UNKNOWN language in OAP by @Superskyyy in https://github.com/apache/skywalking-python/pull/194

Fix logging level not properly set according to config by @Superskyyy in https://github.com/apache/skywalking-python/pull/196

Fix the properties are not set correctly by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/198

Add E2E coverage for Trace and Logging by @Superskyyy in https://github.com/apache/skywalking-python/pull/199

Support log reporter safe mode by @Superskyyy in https://github.com/apache/skywalking-python/pull/200

Fix scheduled event fails changes CI by @Superskyyy in https://github.com/apache/skywalking-python/pull/201

Fix deadlink and CI on schedule by @Superskyyy in https://github.com/apache/skywalking-python/pull/203

Remove namespace to instance properties and add pid by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/205

Enhance Traceback depth default to 10 by @Superskyyy in https://github.com/apache/skywalking-python/pull/206

Unify the tag name with other agents by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/208

Improved ignore path regex by @tom-pytel in https://github.com/apache/skywalking-python/pull/210

Fix sw_psycopg2 register_type() by @tom-pytel in https://github.com/apache/skywalking-python/pull/211

Fix psycopg2 register_type() second arg default by @tom-pytel in https://github.com/apache/skywalking-python/pull/212

Add plugin doc check in CI by @JarvisG495 in https://github.com/apache/skywalking-python/pull/213

Fix typo in docker/Makefile by @jiang1997 in https://github.com/apache/skywalking-python/pull/216

Update UI repository link in PR template by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/217

Add plugin for bottle by @jiang1997 in https://github.com/apache/skywalking-python/pull/214

Draft release 0.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/219

Migrate license checker to license-eye and adjust the release script by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/220

New Contributors

@JaredTan95 made their first contribution in https://github.com/apache/skywalking-python/pull/163

@VxCoder made their first contribution in https://github.com/apache/skywalking-python/pull/176

@doddi made their first contribution in https://github.com/apache/skywalking-python/pull/179

@katelei6 made their first contribution in https://github.com/apache/skywalking-python/pull/178

@dcryans made their first contribution in https://github.com/apache/skywalking-python/pull/182

@arcosx made their first contribution in https://github.com/apache/skywalking-python/pull/188

@wzy960520 made their first contribution in https://github.com/apache/skywalking-python/pull/190

@JarvisG495 made their first contribution in https://github.com/apache/skywalking-python/pull/213

@jiang1997 made their first contribution in https://github.com/apache/skywalking-python/pull/216

Full Changelog: https://github.com/apache/skywalking-python/compare/v0.7.0...v0.8.0
Source code(tar.gz)
Source code(zip)
v0.7.0(Sep 13, 2021)
Feature:

Support collecting and reporting logs to backend (#147)

Support profiling Python method level performance (#127

Add a new sw-python CLI that enables agent non-intrusive integration (#156)

Add exponential reconnection backoff strategy when OAP is down (#157)

Support ignoring traces by http method (#143)

NoopSpan on queue full, propagation downstream (#141)

Support agent namespace. (#126)

Support secure connection option for GRPC and HTTP (#134)

Plugins:

Add Falcon Plugin (#146)

Update sw_pymongo.py to be compatible with cluster mode (#150)

Add Python celery plugin (#125)

Support tornado5+ and tornado6+ (#119)

Fixes:

Remove HTTP basic auth credentials from log, stacktrace, segment (#152)

Fix @trace decorator not work (#136)

Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138)

Others:

Chore: bump up requests version to avoid license issue (#142)

Fix module wrapt as normal install dependency (#123)

Explicit component inheritance (#132)

Provide dockerfile & images for easy integration in containerized scenarios (#159)

Source code(tar.gz)
Source code(zip)
v0.6.0(Mar 31, 2021)
Fixes:

Segment data loss when gRPC timing out. (#116)

sw_tornado plugin async handler status set correctly. (#115)

sw_pymysql error when connection haven't db. (#113)

Source code(tar.gz)
Source code(zip)
v0.5.0(Dec 31, 2020)
New plugins

Pyramid Plugin (#102)

AioHttp Plugin (#101)

Sanic Plugin (#91)

API and enhancements

@trace decorator supports async functions

Supports async task context

Optimized path trace ignore

Moved exception check to Span.__exit__

Moved Method & Url tags before requests

Fixes:

BaseExceptions not recorded as errors

Allow pending data to send before exit

sw_flask general exceptions handled

Make skywalking logging Non-global

Chores and tests

Make tests really run on specified Python version

Deprecate 3.5 as it's EOL

Source code(tar.gz)
Source code(zip)
v0.4.0(Nov 24, 2020)
Feature: Support Kafka reporter protocol (#74)

BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72)

BugFix: Agent cannot reconnect after server is down (#79)

Test: Mitigate unsafe yaml loading (#76)

Source code(tar.gz)
Source code(zip)
v0.3.0(Aug 28, 2020)
New plugins

Urllib3 Plugin (#69)

Elasticsearch Plugin (#64)

PyMongo Plugin (#60)

Rabbitmq Plugin (#53)

Make plugin compatible with Django (#52)

API

Add process propagation (#67)

Add tags to decorators (#65)

Add Check version of packages when install plugins (#63)

Add thread propagation (#62)

Add trace ignore (#59)

Support snapshot context (#56)

Support correlation context (#55)

Chores and tests

Test: run multiple versions of supported libraries (#66)

Chore: add pull request template for plugin (#61)

Chore: add dev doc and reorganize the structure (#58)

Test: update test health check (#57)

Chore: add make goal to package release tar ball (#54)

Source code(tar.gz)
Source code(zip)
v0.2.0(Jul 28, 2020)
New plugins

Kafka Plugin (#50)

Tornado Plugin (#48)

Redis Plugin (#44)

Django Plugin (#37)

PyMsql Plugin (#35)

Flask plugin (#31)

API

Add ignore_suffix Config (#40)

Add missing log method and simplify test codes (#34)

Add content equality of SegmentRef (#30)

Validate carrier before using it (#29)

Chores and tests

Test: print the diff list when validation failed (#46)

Created venv builders for linux/windows and req flashers + use documentation (#38)

Source code(tar.gz)
Source code(zip)
v0.1.0(Jun 28, 2020)
API: agent core APIs, check the APIs and the examples

Plugin: built-in libraries http, urllib.request and third-party library requests are supported.

Test: agent test framework is setup, and the corresponding tests of aforementioned plugins are also added.

Source code(tar.gz)
Source code(zip)

Owner

The Apache Software Foundation

GitHub Repository https://skywalking.apache.org/

Simple script with AminoLab to send ghost messages

1 Nov 22, 2021

Practice10 - Operasi String With Python

Operasi String MY SOSIAL MEDIA : Apa itu Python String ? String adalah urutan si

1 Jan 05, 2022

Developing a python based app prototype with KivyMD framework for a competition :))

1 Jan 10, 2022

Python for Microscopists and other image processing enthusiasts

The YouTube channel associated with this code walks you through the entire process of learning to code in Python; all the way from basics to advanced machine learning and deep learning.

2.3k Jan 01, 2023

Extend the maya channel box with searchability and colour

channel-box-plus will add search-ability over its attributes, and it will colour user defined attributes, making them easier to distinguish.

12 Jun 08, 2022

一个IDA脚本，可以检测出哈希算法（无论是否魔改常数）并生成frida hook 代码。

findhash 在哈希算法上，比Findcrypt更好的检测工具，同时生成Frida hook代码。使用方法把findhash.xml和findhash.py扔到ida plugins目录下 ida -edit-plugin-findhash 试图解决的问题哈希函数的初始化魔数被修改想快速

266 Dec 29, 2022

A compilation of useful scripts to automate common tasks

Scripts-To-Automate-This A compilation of useful scripts for common tasks Name What it does Type Add file extensions Adds ".png" to a list of file nam

0 Nov 05, 2021

A simple service that allows you to run commands on the server using text

Server Text A simple flask service that allows you to run commands on the server/computer over sms. Think of it as a shell where you run commands over

49 Nov 09, 2021

An alternative site to emplea.do due to inconsistent service of the app.

feline a agile and fast alternative to emplea.do License: MIT Settings Moved to settings. Basic Commands Setting Up Your Users To create a normal user

8 Nov 10, 2021

Python project setup, updater, and launcher

pyLaunch Python project setup, updater, and launcher Purpose: Increase project productivity and provide features easily. Once installed as a git submo

1 Jan 07, 2022

Prop-based map editor for the Apex Legends mod, R5Reloaded

R5R Map Editor A tool to build maps out of props in the Apex Legends mod, R5Reloaded Instuctions Install R5R Download this program Get the prop spawne

7 Dec 16, 2022

Python library for parsing Godot scene files

Godot Parser This is a python library for parsing Godot scene (.tscn) and resource (.tres) files. It's intended to make it easier to automate certain

30 Jan 04, 2023

Small tool to use hero .json files created with Optolith for The Dark Eye/ Das Schwarze Auge 5 to perform talent probes.

DSA5-ProbeMaker A little tool for The Dark Eye 5th Edition (Das Schwarze Auge 5) to load .json from Optolith character generation and easily perform t

2 Jan 06, 2022

Pyfetch - Simple Fetch written in Python

pyfetch Simple Fetch written in Python Screenshots Install Clone this repository

2 Sep 02, 2022

Push Prometheus metrics to VictoriaMetrics or other exporters

Push metrics from your periodic long-running jobs to existing Prometheus/VictoriaMetrics monitoring system.

14 Nov 04, 2022

A bash-like intrepreted language

A Bash-like interpreted scripting language.

1 Oct 28, 2021

IDA Pro plugin that shows the comments in a database

ShowComments A Simple IDA Pro plugin that shows the comments in a database Installation Copy the file showcomments.py to the plugins folder under IDA

32 Dec 10, 2022

A (hopefully) considerably copious collection of classical cipher crackers

ClassicalCipherCracker A (hopefully) considerably copious collection of classical cipher crackers Written in Python3 (and run with PyPy) TODOs Write a

2 Feb 22, 2022

sfgp is a package that aggregates individual scripts and notebooks, primarily written for the basic analysis tasks of genetics and pharmacogenomics data.

1 Mar 31, 2022

Python library to decode the EU Covid-19 vaccine certificate

DCC Utils Python library to decode the EU Covid-19 vaccine certificate, as specified by the EU. Setup pip install dcc-utils Make sure zbar is installe

13 Mar 11, 2022

The Python agent for Apache SkyWalking

Related tags

Overview

SkyWalking Python Agent

Documentation

Installation Requirements

Contributing

Contact Us

License

Comments

Behavior

Syntactic sugar

Counter

Histogram

Simple test

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

thread_active_count

Memory

total_mem_utilization

process_mem_utilization

GC

gc_g0

gc_g1

gc_g2

gc_time

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)

What's Changed

New Contributors

v0.7.0(Sep 13, 2021)

v0.6.0(Mar 31, 2021)

v0.5.0(Dec 31, 2020)

v0.4.0(Nov 24, 2020)

v0.3.0(Aug 28, 2020)

v0.2.0(Jul 28, 2020)

v0.1.0(Jun 28, 2020)