The Python agent for Apache SkyWalking

Last update: Dec 12, 2022

Overview

SkyWalking Python Agent

SkyWalking-Python: The Python Agent for Apache SkyWalking, which provides the native tracing abilities for Python project.

SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.

Documentation

Official documentation
Blog about the Python Agent Profiling Feature

Installation Requirements

SkyWalking Python Agent requires SkyWalking 8.0+ and Python 3.6+.

If you would like to try out the latest features that are not released yet, please refer to this guide to build from sources.

Contributing

Before submitting a pull request or pushing a commit, please read our contributing and developer guide.

Contact Us

Submit an GitHub Issue by using [Python] as title prefix.
Mail list: [email protected]. Mail to [email protected], follow the reply to subscribe the mail list.
Join skywalking channel at Apache Slack. If the link is not working, find the latest one at Apache INFRA WIKI.
Twitter, ASFSkyWalking

License

Apache 2.0

Comments

feat: Add support for MeterReportService

Add support for MeterReportService

Behavior

This service runs along with the agent like other services and reports roughly every 20 seconds.

Syntactic sugar

Counter

c = Counter('c2', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time the with-wrapped codes consumed
with c.create_timer():
    # some codes may consume a certain time

c = Counter('c3', CounterMode.INCREMENT)
c.build()

# increase Counter c by num once counter_decorator_test gets called
@Counter.increase(name='c3', num=2)
def counter_decorator_test():
    # some codes

c = Counter('c4', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time counter_decorator_test consumed
@Counter.timer(name='c4')
def counter_decorator_test(s):
    # some codes may consume a certain time

Histogram

h = Histogram('h3', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time the with-wrapped codes consumed
with h.create_timer():
    # some codes may consume a certain time

h = Histogram('h2', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time histogram_decorator_test consumed
@Histogram.timer(name='h2')
def histogram_decorator_test(s):
    time.sleep(s)

Simple test

insert following codes below this line agent/__init__.py#L151

from skywalking.meter.gauge import Gauge
import gc

def generator_for_gc_g0():
    while(True):
        yield gc.get_stats()[0]['collected']

def generator_for_gc_g1():
    while(True):
        yield gc.get_stats()[1]['collected']

def generator_for_gc_g2():
    while(True):
        yield gc.get_stats()[2]['collected']


gc_g0_meter = Gauge("gc_g0", generator_for_gc_g0())
gc_g0_meter.build()

gc_g1_meter = Gauge("gc_g1", generator_for_gc_g1())
gc_g1_meter.build()

gc_g2_meter = Gauge("gc_g2", generator_for_gc_g2())
gc_g2_meter.build()

feature core

opened by jiang1997 19

fix: grpc timeout segment data loss

Please review this thoroughly because all I wanted to do was resend segments which failed to send due to a grpc timeout. But unexpectedly the behavior of grpc timeout changed, it doesn't time out anymore, the heartbeat seems to keep it alive now. I don't understand why this behavior changed so that is why I am requesting a thorough looking at this.
enhancement

opened by tom-pytel 18
Add http ignore by method
Added SW_HTTP_IGNORE_METHOD which will allow ignoring http operations by method (GET, POST, HEAD, OPTIONS, etc...)

Set default SW_AGENT_MAX_BUFFER_SIZE back to 10000 from 1000, which turns out to be too small for python (many unnecessarily dropped spans).

feature
opened by tom-pytel 14
feat: report metrics related to pvm

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

cpu utilization of the current process

thread_active_count

number of threads invoked by the current process and its children

Memory

total_mem_utilization

process_mem_utilization

memory utilization of the current process

GC

gc_g0

number of objects of generation 0

gc_g1

number of objects of generation 1

gc_g2

number of objects of generation 2

gc_time

For now, MeterReportService is available with gRPC only. So does the e2e test. I will implement MeterReportService and enable e2e test for Kafka later.

Welcome any advice about naming.
feature core

opened by jiang1997 12
[Fix][Plugin] sw_flask general exceptions handled
sw_flask fix will handle errors like returning the wrong type from a handler or other internal errors.

Updated StackedSpan to track depth, and made depth variable instance instead of class level (this was a bug).

Changed how SpanContext decides when all spans finished to write Segment data, now counts span start / stops which should work better across different async scenarios.

Changed new_exit_span() with span.inject() to work simpler like the NodeJS agent, now plugins inject directly themselves if they need to.

Removed carrier from plugins which didn't actually use it.

bug plugin
opened by tom-pytel 12
Add plugin for bottle
[x] Add a test case for the new plugin

[x] Add a CHANGELOG entry for the new plugin

[x] Add a component id in the main repo

[x] Add a logo in the UI repo

[x] Rebuild the requirements.txt by running tools/env/build_requirements_(linux|windows).sh

[x] Rebuild the Plugins.md documentation by running make doc-gen

plugin
opened by jiang1997 11
Enable HTTP log reporting

This is a work in progress.

Now reporting logs in JSON through HTTP to http://oap/v3/logs does work.

But, I'm not sure if the oap/v3/logs endpoint is for such usage(It seems designed for fluent-bit batch reporting?). Reporting logs one by one through HTTP may not be ideal in terms of performance. I'm not sure whether the Java agent only implements gRPC reporter intentionally out of this reason.

Please advise.

Signed-off-by: YihaoChen [email protected]
feature

opened by Superskyyy 11
Feature: collect and report logs
This is a OSPP Summer 2021 project supervised by @Humbertzhang | apache/skywalking#7118

The feature implements optional log reporter functionalities in alignment with the SkyWalking Java agent.

Intercepts logs from Python logging module.

Reports logs via a new temporary gRPC channel(to be removed in the future).

Supports unformatted/ formatted logs with custom layout.

Supports custom logging level threshold.

Bumps up submodule to support log collection protocols.

Bumps up skywalking-eyes and adds a config entry to ignore .gitignore during license checks.

feature core
opened by Superskyyy 11
Introduce better coding style

This PR migrates existing usage of old-style string manipulation (%, .format and +) to f-strings for better clarity and some performance boost.

Adds stricter coding style.

Optimizes debug messages for performance.

Signed-off-by: Superskyyy [email protected]
enhancement chore

opened by Superskyyy 10
Add E2E coverage for Trace and Logging
Change Fastapi to FastAPI component name (sync with OAP change)

Add E2E coverage for Python agent (Trace and Logging), profiling is not tested for now.

Renamed a build variable to remove possible mix-ups with CLI environment Var.

Enhance plugin test to reuse images.

Combined CI to pass if changes are non-essential to agent and workflows.

Note: There's a flaky test due to a possible {{- contains }} issue in the Infra-e2e verifier, the logs seem to be queried in an unstable order by the CLI where later log often placed first in log list, combining with the contains bug, it becomes flaky.

A time.sleep() workaround makes sure the log arrives in clear order which passes the e2e for now.
test
opened by Superskyyy 9
[Enhancement] Async tasks should work 100%

These changes should address all remaining async-related problems not handled in #88. The spans list has been moved from an instance variable in SpanContext to a global task-local variable which allows "forking" it for new subtasks so that multiple concurrent async subtasks don't interfere with one another. spans is now a top-level module var because Python stipulates that ContextVar should be such and only one context exists at any given time so it is essentially a singleton anyway.
enhancement core

opened by tom-pytel 9
Enhance redis plugin to adapt Virtual Cache

Resolves: #10212
enhancement

opened by Jedore 2

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)
What's Changed

spans correctly reference finished parents by @tom-pytel in https://github.com/apache/skywalking-python/pull/161

Refactor current Python agent docs to serve on SkyWalking official website by @Superskyyy in https://github.com/apache/skywalking-python/pull/162

fix broken url by @JaredTan95 in https://github.com/apache/skywalking-python/pull/163

Remove docs from main README.md - Add website doc links. by @Superskyyy in https://github.com/apache/skywalking-python/pull/164

Refactor SkyWalking Python to use the CLI for CI instead of legacy setup by @Superskyyy in https://github.com/apache/skywalking-python/pull/165

Remove places that mentions Python 3.5 support due to EOL by @Superskyyy in https://github.com/apache/skywalking-python/pull/166

Cleanup and Python 3.10 support by @Superskyyy in https://github.com/apache/skywalking-python/pull/167

filled out rest of psycopg 3 plugin by @tom-pytel in https://github.com/apache/skywalking-python/pull/168

Move flake configs all together by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/169

Enable faster CI by categorical parallelism by @Superskyyy in https://github.com/apache/skywalking-python/pull/170

Introduce better coding style by @Superskyyy in https://github.com/apache/skywalking-python/pull/171

Minimize exclude in lint by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/173

Introduce another set of flake8 extensions by @Superskyyy in https://github.com/apache/skywalking-python/pull/174

fix aiohttp outgoing request url by @tom-pytel in https://github.com/apache/skywalking-python/pull/175

bugfix: flask + nginx got KeyError: 'REMOTE_PORT' in sw_flask.py plugin by @VxCoder in https://github.com/apache/skywalking-python/pull/176

Add missing ending quote by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/177

updated deprecated configuration by @doddi in https://github.com/apache/skywalking-python/pull/179

Add plugin for mysqlclient by @katelei6 in https://github.com/apache/skywalking-python/pull/178

Fix sw-rabbitmq TypeError when there are no headers by @dcryans in https://github.com/apache/skywalking-python/pull/182

Fix agent bootup traceback not shown in sw-python CLI by @Superskyyy in https://github.com/apache/skywalking-python/pull/183

Add plugin for FastAPI by @katelei6 in https://github.com/apache/skywalking-python/pull/181

Support CI container logs by @Superskyyy in https://github.com/apache/skywalking-python/pull/185

Update mySQL plugin to support two different parameter keys. by @katelei6 in https://github.com/apache/skywalking-python/pull/186

add the code > 400 is error by @katelei6 in https://github.com/apache/skywalking-python/pull/187

Doc: add how to use with uwsgi by @arcosx in https://github.com/apache/skywalking-python/pull/188

Fix: right doc link for How To Use With uWSGI by @arcosx in https://github.com/apache/skywalking-python/pull/189

The local log stack depth is not truncated by @wzy960520 in https://github.com/apache/skywalking-python/pull/190

Revert "The local log stack depth is not truncated (#190)" by @Superskyyy in https://github.com/apache/skywalking-python/pull/191

Fix sw_formatter wrongly set cache that impacts user handlers by @Superskyyy in https://github.com/apache/skywalking-python/pull/192

Fix typo that cause failure in loading user sitecustomize.py by @Superskyyy in https://github.com/apache/skywalking-python/pull/193

Drop support for Flask 1.x due to EOL & Jinja2 issue by @Superskyyy in https://github.com/apache/skywalking-python/pull/195

Update agent docs, changelogs and PR template. by @Superskyyy in https://github.com/apache/skywalking-python/pull/197

Fix Python shown as UNKNOWN language in OAP by @Superskyyy in https://github.com/apache/skywalking-python/pull/194

Fix logging level not properly set according to config by @Superskyyy in https://github.com/apache/skywalking-python/pull/196

Fix the properties are not set correctly by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/198

Add E2E coverage for Trace and Logging by @Superskyyy in https://github.com/apache/skywalking-python/pull/199

Support log reporter safe mode by @Superskyyy in https://github.com/apache/skywalking-python/pull/200

Fix scheduled event fails changes CI by @Superskyyy in https://github.com/apache/skywalking-python/pull/201

Fix deadlink and CI on schedule by @Superskyyy in https://github.com/apache/skywalking-python/pull/203

Remove namespace to instance properties and add pid by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/205

Enhance Traceback depth default to 10 by @Superskyyy in https://github.com/apache/skywalking-python/pull/206

Unify the tag name with other agents by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/208

Improved ignore path regex by @tom-pytel in https://github.com/apache/skywalking-python/pull/210

Fix sw_psycopg2 register_type() by @tom-pytel in https://github.com/apache/skywalking-python/pull/211

Fix psycopg2 register_type() second arg default by @tom-pytel in https://github.com/apache/skywalking-python/pull/212

Add plugin doc check in CI by @JarvisG495 in https://github.com/apache/skywalking-python/pull/213

Fix typo in docker/Makefile by @jiang1997 in https://github.com/apache/skywalking-python/pull/216

Update UI repository link in PR template by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/217

Add plugin for bottle by @jiang1997 in https://github.com/apache/skywalking-python/pull/214

Draft release 0.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/219

Migrate license checker to license-eye and adjust the release script by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/220

New Contributors

@JaredTan95 made their first contribution in https://github.com/apache/skywalking-python/pull/163

@VxCoder made their first contribution in https://github.com/apache/skywalking-python/pull/176

@doddi made their first contribution in https://github.com/apache/skywalking-python/pull/179

@katelei6 made their first contribution in https://github.com/apache/skywalking-python/pull/178

@dcryans made their first contribution in https://github.com/apache/skywalking-python/pull/182

@arcosx made their first contribution in https://github.com/apache/skywalking-python/pull/188

@wzy960520 made their first contribution in https://github.com/apache/skywalking-python/pull/190

@JarvisG495 made their first contribution in https://github.com/apache/skywalking-python/pull/213

@jiang1997 made their first contribution in https://github.com/apache/skywalking-python/pull/216

Full Changelog: https://github.com/apache/skywalking-python/compare/v0.7.0...v0.8.0
Source code(tar.gz)
Source code(zip)
v0.7.0(Sep 13, 2021)
Feature:

Support collecting and reporting logs to backend (#147)

Support profiling Python method level performance (#127

Add a new sw-python CLI that enables agent non-intrusive integration (#156)

Add exponential reconnection backoff strategy when OAP is down (#157)

Support ignoring traces by http method (#143)

NoopSpan on queue full, propagation downstream (#141)

Support agent namespace. (#126)

Support secure connection option for GRPC and HTTP (#134)

Plugins:

Add Falcon Plugin (#146)

Update sw_pymongo.py to be compatible with cluster mode (#150)

Add Python celery plugin (#125)

Support tornado5+ and tornado6+ (#119)

Fixes:

Remove HTTP basic auth credentials from log, stacktrace, segment (#152)

Fix @trace decorator not work (#136)

Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138)

Others:

Chore: bump up requests version to avoid license issue (#142)

Fix module wrapt as normal install dependency (#123)

Explicit component inheritance (#132)

Provide dockerfile & images for easy integration in containerized scenarios (#159)

Source code(tar.gz)
Source code(zip)
v0.6.0(Mar 31, 2021)
Fixes:

Segment data loss when gRPC timing out. (#116)

sw_tornado plugin async handler status set correctly. (#115)

sw_pymysql error when connection haven't db. (#113)

Source code(tar.gz)
Source code(zip)
v0.5.0(Dec 31, 2020)
New plugins

Pyramid Plugin (#102)

AioHttp Plugin (#101)

Sanic Plugin (#91)

API and enhancements

@trace decorator supports async functions

Supports async task context

Optimized path trace ignore

Moved exception check to Span.__exit__

Moved Method & Url tags before requests

Fixes:

BaseExceptions not recorded as errors

Allow pending data to send before exit

sw_flask general exceptions handled

Make skywalking logging Non-global

Chores and tests

Make tests really run on specified Python version

Deprecate 3.5 as it's EOL

Source code(tar.gz)
Source code(zip)
v0.4.0(Nov 24, 2020)
Feature: Support Kafka reporter protocol (#74)

BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72)

BugFix: Agent cannot reconnect after server is down (#79)

Test: Mitigate unsafe yaml loading (#76)

Source code(tar.gz)
Source code(zip)
v0.3.0(Aug 28, 2020)
New plugins

Urllib3 Plugin (#69)

Elasticsearch Plugin (#64)

PyMongo Plugin (#60)

Rabbitmq Plugin (#53)

Make plugin compatible with Django (#52)

API

Add process propagation (#67)

Add tags to decorators (#65)

Add Check version of packages when install plugins (#63)

Add thread propagation (#62)

Add trace ignore (#59)

Support snapshot context (#56)

Support correlation context (#55)

Chores and tests

Test: run multiple versions of supported libraries (#66)

Chore: add pull request template for plugin (#61)

Chore: add dev doc and reorganize the structure (#58)

Test: update test health check (#57)

Chore: add make goal to package release tar ball (#54)

Source code(tar.gz)
Source code(zip)
v0.2.0(Jul 28, 2020)
New plugins

Kafka Plugin (#50)

Tornado Plugin (#48)

Redis Plugin (#44)

Django Plugin (#37)

PyMsql Plugin (#35)

Flask plugin (#31)

API

Add ignore_suffix Config (#40)

Add missing log method and simplify test codes (#34)

Add content equality of SegmentRef (#30)

Validate carrier before using it (#29)

Chores and tests

Test: print the diff list when validation failed (#46)

Created venv builders for linux/windows and req flashers + use documentation (#38)

Source code(tar.gz)
Source code(zip)
v0.1.0(Jun 28, 2020)
API: agent core APIs, check the APIs and the examples

Plugin: built-in libraries http, urllib.request and third-party library requests are supported.

Test: agent test framework is setup, and the corresponding tests of aforementioned plugins are also added.

Source code(tar.gz)
Source code(zip)

Owner

The Apache Software Foundation

GitHub Repository https://skywalking.apache.org/

An open source recipe book from the awesome staff of Clinical Genomics

meatballs An open source recipe book from the awesome staff of Clinical Genomics.

2 Dec 07, 2021

AMTIO aka All My Tools in One

AMTIO AMTIO aka All My Tools In One. I plan to put a bunch of my tools in this one repo since im too lazy to make one big tool. Installation git clone

3 Jul 29, 2021

Mnemosyne: efficient learning with powerful digital flash-cards.

Mnemosyne: Optimized Flashcards and Research Project Mnemosyne is: a free, open-source, spaced-repetition flashcard program that helps you learn as ef

359 Dec 24, 2022

Improving the Transferability of Adversarial Examples with Resized-Diverse-Inputs, Diversity-Ensemble and Region Fitting

7 Oct 20, 2022

A way to write regex with objects instead of strings.

Py Idiomatic Regex (AKA iregex) Documentation Available Here An easier way to write regex in Python using OOP instead of strings. Makes the code much

18 Nov 15, 2021

Msgpack serialization/deserialization library for Python, written in Rust using PyO3 and rust-msgpack. Reboot of orjson. msgpack.org[Python]

ormsgpack ormsgpack is a fast msgpack library for Python. It is a fork/reboot of orjson It serializes faster than msgpack-python and deserializes a bi

139 Dec 30, 2022

Easy to use phishing tool with 65 website templates. Author is not responsible for any misuse.

PyPhisher [+] Description : Ultimate phishing tool in python. Includes popular websites like facebook, twitter, instagram, github, reddit, gmail and m

1.1k Dec 31, 2022

Cloud-native SIEM for intelligent security analytics for your entire enterprise.

Microsoft Sentinel Welcome to the Microsoft Sentinel repository! This repository contains out of the box detections, exploration queries, hunting quer

2.9k Jan 02, 2023

Addon to give a keybind to automatically enable contact shadows on all lights in a scene

3-2-1 Contact(Shadow) An easy way to let you enable contact shadows on all your lights, because Blender doesn't enable it by default, and doesn't give

3 Feb 02, 2022

Improve current data preprocessing for FTM's WOB data to analyze Shell and Dutch Governmental contacts.

We're the hackathon leftovers, but we are Too Good To Go ;-). A repo by Lukas Schubotz and Raymon van Dinter. We aim to improve current data preprocessing for FTM's WOB data to analyze Shell and Dutc

5 Dec 09, 2021

This is a project for socks card label validation where the socks card is validated comparing with the correct socks card whose coordinates are stored in the database. When the test socks card is compared with the correct socks card(master socks card) the software checks whether both test and master socks card mathches or not.

Automation_in_socks_label_validation THEME: MACHINE LEARNING This is a project for socks card label validation where the socks card is validated compa

3 Apr 21, 2022

The Python agent for Apache SkyWalking

Related tags

Overview

SkyWalking Python Agent

Documentation

Installation Requirements

Contributing

Contact Us

License

Comments

Behavior

Syntactic sugar

Counter

Histogram

Simple test

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

thread_active_count

Memory

total_mem_utilization

process_mem_utilization

GC

gc_g0

gc_g1

gc_g2

gc_time

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)

What's Changed

New Contributors

v0.7.0(Sep 13, 2021)

v0.6.0(Mar 31, 2021)

v0.5.0(Dec 31, 2020)

v0.4.0(Nov 24, 2020)

v0.3.0(Aug 28, 2020)

v0.2.0(Jul 28, 2020)

v0.1.0(Jun 28, 2020)

Owner

The Apache Software Foundation

An open source recipe book from the awesome staff of Clinical Genomics

AMTIO aka All My Tools in One

Mnemosyne: efficient learning with powerful digital flash-cards.

Improving the Transferability of Adversarial Examples with Resized-Diverse-Inputs, Diversity-Ensemble and Region Fitting

A way to write regex with objects instead of strings.

Msgpack serialization/deserialization library for Python, written in Rust using PyO3 and rust-msgpack. Reboot of orjson. msgpack.org[Python]

Easy to use phishing tool with 65 website templates. Author is not responsible for any misuse.

Cloud-native SIEM for intelligent security analytics for your entire enterprise.

Addon to give a keybind to automatically enable contact shadows on all lights in a scene

Improve current data preprocessing for FTM's WOB data to analyze Shell and Dutch Governmental contacts.

Analisador de strings feito em Python // String parser made in Python

Simple AoC helper program you can use to develop your own solutions in python.

清晰易读的7x7像素点阵中文字体和取模工具

Just some mtk tool for exploitation, reading/writing flash and doing crazy stuff

Blender-miHoYo-Shaders - Shaders for Blender attempting to replicate the shading of games developed by miHoYo

python3 scrip for case conversion of source code files writen in fixed form fortran

WhyNotWin11 - Detection Script to help identify why your PC isn't Windows 11 Release Ready

Provide error messages for Python exceptions, even if the original message is empty

Helper to organize your windows on your desktop.