PcapXray - A Network Forensics Tool - To visualize a Packet Capture offline as a Network Diagram

Overview

PcapXray Build Status codecov defcon27

A Network Forensics Tool - To visualize a Packet Capture offline as a Network Diagram including device identification, highlight important communication and file extraction

Alt text

PcapXray Design Specification

Wiki has more help too.

Goal:

Given a Pcap File, plot a network diagram displaying hosts in the network, network traffic, highlight important traffic and Tor traffic as well as potential malicious traffic including data involved in the communication.

Problem:

  • Investigation of a Pcap file takes a long time given initial glitch to start the investigation

  • Faced by every forensics investigator and anyone who is analyzing the network

  • Location: https://github.com/Srinivas11789/PcapXray

Solution: Speed up the investigation process

  • Make a network diagram with the following features from a Pcap file Tool Highlights:
  • Network Diagram – Summary Network Diagram of full network
  • Information:
    • Web Traffic with Server Details
    • Tor Traffic
    • Possible Malicious traffic
    • Data Obtained from Packet in Report – Device/Traffic/Payloads
    • Device Details

Tool Image:

Alt text

Alt text

Components:

  • Network Diagram
  • Device/Traffic Details and Analysis
  • Malicious Traffic Identification
  • Tor Traffic
  • GUI – a gui with options to upload pcap file and display the network diagram

Setup

  • Python 3
apt install python3-pip
apt install python3-tk
apt install graphviz
apt install python3-pil python3-pil.imagetk
pip3 install -r requirements.txt
python3 Source/main.py

( Make sure to escalate privilege to allow file creations - Run with sudo )

For MAC:

brew install graphviz
  • Python 2
apt install python-tk
apt install graphviz
pip install -r requirements.txt
python Source/main.py

( Make sure to escalate privilege to allow file creations - Run with sudo )

Python Libraries Used: - All these libraries are required for functionality

  • Tkinter and TTK – Install from pip or apt-get – Ensure Tkinter and graphviz is installed (Most Linux contain by default)
    • apt install python-tk
    • apt install graphviz
    • apt install python3-tk (for python3 support)
    • Sometimes ImageTk errors are thrown in python3 env --> use apt install python3-pil python3-pil.imagetk
  • All these are included in the requirements.txt file
    • Scapy – rdpcap to read the packets from the pcap file
    • Ipwhois – to obtain whois information from ip
    • Netaddr – to check ip information type
    • Pillow – image processing library
    • Stem – tor consensus data fetch library
    • pyGraphviz – plot graph
    • Networkx – plot graph
    • Matplotlib – plot graph (not used as of now)

Demo

Alt text

Getting started:

  • Clone the repository
  • pip install -r requirements.txt
  • python Source/main.py

Additional Information:

  • Tested on Linux
  • Options for Traffic include - Web (HTTP and HTTPS), Tor, Malicious, ICMP, DNS

Challenges:

  • Unstability of the TK GUI:
    • Decision on the GUI between Django and TK, settled upon tk for a simple local interface, but the unstability of the tk gui caused a number of problems
  • Graph Plotting:
    • Plotting a proper network graph which is readable from the data obtained was quite an effort, used different libraries to arrive at one.
  • Performance and Timing:
    • The performance and timing of the total application was a big challenge with different data gathering and output generation

Known Bugs:

  • Memory Hogging

    • Sometimes memory hogging occurs when lower RAM is present in the system as the data stored in the memory from the pcap file is huge
    • Should be Fixed by moving data into a database than the memory itself
  • Race Condition

    • Due to mainloop of the TK gui, other threads could undergo a race condition
    • Should be fixed by moving to a better structured TK implementation or Web GUI
  • Tk GUI Unstability:

    • Same reason as above
  • Code:

    • clumsy and unstructured code flow
  • Current Fix in rare occasions: If any of the above issue occurs the progress bar keeps running and no output is generated, a restart of the app would be required.

Docker Containers of PcapXray

  • Dockerfile present in the root folder was used to build images
  • Already built docker images are found at dockerhub
    • srinivas11789/pcapxray-1.0
    • srinivas11789/pcapxray-2.2
  • Performing the steps in run.sh file manually would work to launch the tool via docker (I can help with errors)
  • Running run.sh scripts is an attempt to automate (would not work 100 percent)
    • tested on mac and linux - will be better soon!...

Immediate Future Tasks: (Target: 3.0)

  • Clean up code (beautify code base from being a prototype)
  • Report generation on unique folders for all assets of a packet capture
  • Suspicious activity detection
  • Support more pcap reader engine
  • Traffic support: ICMP, DNS
  • Known file type detection and Extract
  • Python2 and Python3
  • Interactive map

Future:

  • Structured and clean code flow
  • Change the database from JSON to sqlite or prominent database, due to memory hogging
  • Change fronend to web based such as Django
  • Make the application more stable
  • More protocol support
  • Clean up code

Credits:

  • Thanks for making it better,
    • Professor Marc Budofsky
    • Kevin Gallagher
  • Thanks for all the dependent libraries used
  • Logo created with logomakr.com and www.inkscape.org

Analytics

Just for Security Fun!

Comments
  • Error on Start

    Error on Start

    Traceback (most recent call last):
      File "Source/main.py", line 12, in <module>
        import userInterface
    ImportError: No module named userInterface
    

    I get this error on macOS 10.13.5, running Python 2.7.15. This error occurs after running pip install -r requirements.txt.

    Looks like a very cool project, hoping there's an easy package to install that makes it mac compatible, but I'm not seeing anything by googling around.

    opened by seajaysec 3
  • Modernize Python 2 code to get ready for Python 3

    Modernize Python 2 code to get ready for Python 3

    Make the minimal, safe changes required to convert the repo's code to be syntax compatible with both Python 2 and Python 3. There may be other changes required to complete a port to Python 3 but this PR is a minimal, safe first step.

    Run: futurize --stage1 -w **/*.py

    See Stage 1: "safe" fixes http://python-future.org/automatic_conversion.html#stage-1-safe-fixes
    
    opened by cclauss 3
  • NSOpenPanel: NSInternalInconsistencyException with tk ui in mac os

    NSOpenPanel: NSInternalInconsistencyException with tk ui in mac os

    2022-02-15 09:17:23.967 Python[9628:502841] *** Assertion failure in -[NSOpenPanel beginServicePanel:asyncExHandler:], NSVBOpenAndSavePanels.m:1907
    2022-02-15 09:17:24.004 Python[9628:502841] -[NSSavePanel beginWithCompletionHandler:]_block_invoke caught non-fatal NSInternalInconsistencyException '<NSOpenPanel: 0x7fcecb72f600> is attempting to advance this Open/Save panel to run phase while another self.advanceToRunPhaseCompletionHandler is in waiting for a previous attempt. An Open/Save panel cannot start to advance more than once.' with user dictionary {
    ...
    
    opened by Srinivas11789 1
  • Fallback toggle for interactive graph

    Fallback toggle for interactive graph

    In recent python version py3.8, the app does not start due to dependency library support.

    • As a partial fix, add a fallback method to resort to launching interactive graph in system default browser.
    • This should auto-fix once the respective support is added upstream.
    opened by Srinivas11789 1
  • PcapXray 2.8

    PcapXray 2.8

    Changes:

    • Design: Add flow charts for PcapXray
    • Feature: File Signature analysis for covert traffic - add magic number analysis on payload
    • Enhance: Update pyshark engine code to work with engine selections in the UI
      • The UI is disabled for now due to pending issue solves with pyshark
    • Fix for some ctf problems and MAC spoofing scenarios --> shows up as weird traffic
    opened by Srinivas11789 1
  • PcapXray 2.7

    PcapXray 2.7

    Features:

    • Covert communication ( focus on icmp and dns for now)
      • Block and Algorithm to predict covert ICMP and DNS traffic.
    • Mac bug hacky fix - launch interactive on a browser until main bug gets solved

    Test:

    • Built primarily with learning from network forensics challenges from ( will improvise )
      • DNSCAP —> https://ctftime.org/task/3418 [DNS]
      • PcapMeIfYouCan —> https://ctftime.org/task/7087 [DNS]
      • Biz4rre —> https://ctftime.org/task/6748 [ICMP]
      • FromOurFriendsNextHop —> https://ctftime.org/task/7099 [DNS]
      • Fuzzy —> https://ctftime.org/task/6928 [DNS]
      • Data Exfil —> https://ctftime.org/task/5735 [DNS]
      • https://www.netresec.com/?page=PcapFiles
    opened by Srinivas11789 1
  • Covert communication

    Covert communication

    • Block and Algorithm to predict covert ICMP and DNS traffic.
    • Alter lan hosts schema to support solving CTF chals
    • covert traffic in graph
    • Built primarily with network forensics challenges from
      • DNSCAP —> https://ctftime.org/task/3418 [DNS]
      • PcapMeIfYouCan —> https://ctftime.org/task/7087 [DNS]
      • Biz4rre —> https://ctftime.org/task/6748 [ICMP]
      • Sniffed Off the wire —> https://ctftime.org/task/4758 [TCP]
      • FromOurFriendsNextHop —> https://ctftime.org/task/7099 [DNS]
      • Fuzzy —> https://ctftime.org/task/6928 [DNS]
      • Data Exfil —> https://ctftime.org/task/5735 [DNS]
    opened by Srinivas11789 1
  • Interactive map

    Interactive map

    • Interactive Maps (with python _ interaction)
      • CEF method
    • Python2 fixes for stability ( hopefully stable now )
    • Partial MAC support now ( limited )
      • No interactive maps in mac due to Cef crashes

    Screen Shot 2019-06-30 at 4 48 31 PM

    opened by Srinivas11789 1
  • Graph Enhancements

    Graph Enhancements

    • gateway identification logic
    • hybrid L2 + L3 routing
    • refactor of Reports + PcapRead
    • Graph enhancements - different alignment and arrangement for larger graphs
    • Options additions
    • Image resolution changes ( >= 600 makes the loading slow )
    • Huge number of nodes now get different alignment (circo)
    • improved payload arrangement

    Still require improvements on:

    • Higher image resolution can be set to make a big graph more legible but makes tkinter image loading much slower - hot fix soon
    Screen Shot 2019-05-10 at 9 35 41 AM

    Example FTP Payload:

    Screen Shot 2019-05-10 at 9 49 11 AM
    opened by Srinivas11789 1
  • PcapXray 2.0

    PcapXray 2.0

    Release bump to 2.0 (A bulk set of features)

    • UI features - Browse, Zoom, Icon
    • Docker image + automated app start (run.sh + docker) --> supports mac and linux (pending test on windows)
    • Bug: init main.py (system path fix)
    • More tests
    opened by Srinivas11789 1
  • startup error

    startup error

    Hello, What is this error:

    ┌──(root㉿kali)-[~/PcapXray] └─# python3 Source/main.py Interactive graph in app wont work as python version/platform is not supported (will launch in default browser) Traceback (most recent call last): File "/root/PcapXray/Source/main.py", line 52, in main() File "/root/PcapXray/Source/main.py", line 41, in main base = Tk() File "/usr/lib/python3.10/tkinter/init.py", line 2299, in init self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use) _tkinter.TclError: no display name and no $DISPLAY environment variable

    opened by nmaphacker 1
  • Added Sqlite DB functionality for packet data and pcap session loading.

    Added Sqlite DB functionality for packet data and pcap session loading.

    on-behalf-of: @org [email protected]

    Technica has added Sqlite db functionality for the packet data and a prototype for the destination hosts was also added, though not used/tested. This version is also capable of loading previously analyzed pcap data sessions; based on the name of the pcap file/sqlite db file. There is a bug in this version to be noted. The device information that is used during visualization is not persisted in the database, but kept in Global Memory which is erased when the application is closed. If the same PCAP is analyzed again, the data is pulled from the Sqlite database but the analyze portion is skipped so the device information will be missing.

    opened by mbernardo 0
  • Exception: Python version not supported: 3.8.2

    Exception: Python version not supported: 3.8.2

    Python 3.8.2

    sudo python3 Source/main.py Traceback (most recent call last): File "Source/main.py", line 12, in from cefpython3 import cefpython as cef File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/cefpython3/init.py", line 64, in raise Exception("Python version not supported: " + sys.version) Exception: Python version not supported: 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18) [Clang 6.0 (clang-600.0.57)]

    opened by M0dred 2
  • Interactive graph does not work if the full file path name has more than one

    Interactive graph does not work if the full file path name has more than one "."

    • pyvis has a check for assert len(name.split(".")) == 2 which fails when a filename with full path containing "."

    Hacky Fix:

    • Have no "." when along the folders where PcapXray is located.
    opened by Srinivas11789 0
  • Infinite progress bar problem

    Infinite progress bar problem

    Issue: Sometimes the progress bar keeps loading even when the backend thread has finished its job. Temperory Fix: Restarting the tool solves this error. ( bad fix )

    opened by Srinivas11789 0
Releases(3_0)
  • 3_0(Aug 22, 2019)

    🚀 🎆 PcapXray 3.0 🎆 🚀

    • Bump release as a checkpoint for 2.0 -> 3.0 work
    • Update defcon27 demolabs asset

    HighLights

    • Covert communication
    • Better UI and controls
    • Interactive Map
    • Python2 and Python3
    • TLS Insight
    Source code(tar.gz)
    Source code(zip)
  • v2_9(Aug 12, 2019)

  • 2_8_1(Aug 7, 2019)

    🕵 Version 2.8 🕵️‍♀️ ➕ ➕

    Changes:

    • Design: Add flow charts for PcapXray
    • Feature: File Signature analysis for covert traffic - add magic number analysis on payload
    • Enhance: Update pyshark engine code to work with engine selections in the UI
      • The UI is disabled for now due to pending issue solves with pyshark
    • Fix for some ctf problems and MAC spoofing scenarios --> shows up as weird traffic
    Source code(tar.gz)
    Source code(zip)
  • 2_8(Aug 6, 2019)

    Changes:

    • Design: Add flow charts for PcapXray
    • Feature: File Signature analysis for covert traffic - add magic number analysis on payload
    • Enhance: Update pyshark engine code to work with engine selections in the UI
      • The UI is disabled for now due to pending issue solves with pyshark
    • Fix for some ctf problems and MAC spoofing scenarios --> shows up as weird traffic
    Source code(tar.gz)
    Source code(zip)
  • v2_7(Aug 3, 2019)

    🕵 Version 2.7 🕵️‍♀️

    Features:

    • Covert communication ( focus on icmp and dns for now)
    • Block and Algorithm to predict covert ICMP and DNS traffic.
    • Mac bug hacky fix - launch interactive on a browser until main bug gets solved
    Source code(tar.gz)
    Source code(zip)
  • v2_6(Jul 2, 2019)

  • v2_6_beta(Jul 1, 2019)

  • 2.5(May 15, 2019)

  • v2.4(May 4, 2019)

    PcapXray v2.4 :sparkler:

    screen2_2_4

    • Include ICMP and DNS traffic category and record payloads
    • Add provision for more pcap engines
    • License compliance with respect to the other libraries
    • Huge refactor
    • Memory component to hold data and record
    • Custom destination for Report
    • Support both python3 and python2
    • Increase test coverage
    Source code(tar.gz)
    Source code(zip)
  • v2.3-beta(May 2, 2019)

    • Include ICMP and DNS traffic category and record payloads
    • Add provision for more pcap engines
    • License compliance with respect to the other libraries
    • Huge refactor
    • Memory component to hold data and record
    • Custom destination for Report
    • Support both python3 and python2
    • Increase test coverage
    Source code(tar.gz)
    Source code(zip)
  • v2.2(Jan 17, 2019)

  • v2.2-beta(Jan 17, 2019)

  • v2.1-beta(Jan 15, 2019)

  • v2.0-beta(Jan 14, 2019)

    Bump version to 2.0 - Bunch of features and fixes

    Features:

    • UI features - Browse, Zoom, Icon
    • Docker image + automated app start (run.sh + docker) --> supports mac and linux (pending test on windows)
    • Bug: init main.py (system path fix)
    • More tests
    screen shot 2019-01-14 at 10 36 02 am Source code(tar.gz)
    Source code(zip)
  • v1.0(Jan 14, 2019)

Owner
Srinivas P G
Love building/breaking things --> Code || Test, Plan, Break, Debug (Loop!)
Srinivas P G
This Tool Help To Information gathering for domain name or ip address...

Owl-Eye This Tool Help To Information gathering for domain name or ip address... follow this command $apt update && upgrade $apt install python apt in

Black Owl 6 Nov 12, 2022
Python Program to connect to different VPN servers autoatically using Windscribe VPN.

AutomateVPN What is VPN ? VPN stands for Virtual Private Network , it is a technology that creates a safe and encrypted connectionover a less secure n

Vivek 1 Oct 27, 2021
Tsunami-Fi is simple multi-tool bash application for Wi-Fi attacks

🪴 Tsunami-Fi 🪴 Русская версия README 🌿 Description 🌿 Tsunami-Fi is simple multi-tool bash application for Wi-Fi WPS PixieDust and NullPIN attack,

【Kiko】 35 Dec 09, 2022
PySocks lets you send traffic through SOCKS proxy servers.

PySocks lets you send traffic through SOCKS proxy servers. It is a modern fork of SocksiPy with bug fixes and extra features. Acts as a drop-i

1.1k Dec 07, 2022
sshuttle: where transparent proxy meets VPN meets ssh

Transparent proxy server that works as a poor man's VPN. Forwards over ssh. Doesn't require admin. Works with Linux and MacOS. Supports DNS tunneling.

9.4k Jan 09, 2023
Usbkill - an anti-forensic kill-switch that waits for a change on your USB ports and then immediately shuts down your computer.

Usbkill - an anti-forensic kill-switch that waits for a change on your USB ports and then immediately shuts down your computer.

Hephaestos 4.1k Dec 30, 2022
Network monitoring tool

netmeter If you are looking for a tool to monitor your network interfaces, here you are. See netmeter-exporter to export Prometheus metrics. Installat

Saeid Bostandoust 97 Dec 03, 2022
Tiny Interactive File Transfer Application

TIFTA: Tiny Interactive File Transfer Application This repository holds all the source code, tests and documentation of the TIFTA software. The main g

Jorge Martínez 2 Dec 08, 2021
A Python library to ease the integration with the Beem Africa (SMS, AIRTIME, OTP, 2WAY-SMS, BPAY, USSD)

python-client A Python library to easy the integration with the Beem Africa SMS Gateway Features to be Implemented Airtime OTP SMS Two way SMS USSD Bp

Beem Africa 24 Oct 29, 2022
tradingview socket api for fetching real time prices.

tradingView-API tradingview socket api for fetching real time prices. How to run git clone https://github.com/mohamadkhalaj/tradingView-API.git cd tra

MohammadKhalaj 35 Dec 31, 2022
Blockchain-Enabled IoT Sensor Framework that uses Augmented Reality and Artificial Intelligence.

Arduino + Raspberry Pi + Unity3D + Cloud + Hyperledger Our Mission: Keep it simple, leave no one behind. Blockchain-Enabled Smart Sensor Framework usi

DappAR 23 Dec 05, 2021
Tripwire monitors ports and icmp to send the admin a message if somebody is scanning a machine that shouldn't be touched

Tripwire monitors ports and icmp to send the admin a message if somebody is scanning a machine that shouldn't be touched

3 Apr 05, 2022
BaseSpec is a system that performs a comparative analysis of baseband implementation and the specifications of cellular networks.

BaseSpec is a system that performs a comparative analysis of baseband implementation and the specifications of cellular networks. The key intuition of BaseSpec is that a message decoder in baseband s

SysSec Lab 35 Dec 06, 2022
jarbou3 is rat tool coded in python with C&C which can accept multiple connections from clients

jarbou3 Jarbou3 is rat tool with coded in python with C&C which can accept multi

youhacker55 108 Dec 29, 2022
A fire and forget command-line tool to allow for easy transitions of VPN connections between a pool of AWS machines.

VPN Swapper A fire and forget command-line tool to allow for easy transitions of VPN connections between a pool of AWS machines. Dependencies poetry -

Workday 5 Jul 07, 2022
With Py-Autocrack you can crack WPA2 networks in no time.

With Py-Autocrack you can crack WPA2 networks in no time. All based on Aircrack-ng and Crunch.

Paul - FloatDesign 1 Dec 10, 2021
FIRM-AFL is the first high-throughput greybox fuzzer for IoT firmware.

FIRM-AFL is the first high-throughput greybox fuzzer for IoT firmware. FIRM-AFL addresses two fundamental problems in IoT fuzzing

356 Dec 23, 2022
GlokyPortScannar is a really fast tool to scan TCP ports implemented in Python.

GlokyPortScannar is a really fast tool to scan TCP ports implemented in Python. Installation: This program requires Python 3.9. Linux

gl0ky 5 Jun 25, 2022
This script aims to make the dynamic public ip of your local server, public.

EZ DDNS CLOUDFLARE This script aims to make the dynamic ip of your local server, public. It does this by regularly updating cloudflare's dns record. B

3 Feb 13, 2022
This tool will scans your wi-fi/wlan and show you the connected clients

This tool will scans your wi-fi/wlan and show you the connected clients

VENKAT SAI SAGAR 3 Mar 24, 2022