Class Renames: Dyld -> MachOImageLoader ImageHeader -> MachOImageHeader

LD64 class was removed insert_load_cmd(load_command, index=-1, suffix=None) and remove_load_command(index) added to MachOImageHeader which supports adding and removing all types of load commands now.

ktool.dyld module renamed to ktool.loader, now contains only code relevant specifically to loading the Image class from a standard MachO Image class and a few others moved to new image.py, in which contained code is a non-platform-specific abstraction not tied to MachO.

Fixes:

• Rewritten load command injection should fix issues with round-tripping and producing bad patches. A ton of unit testing for this area was added to try and maintain this.
• Fix for some issues on certain weird linux environments

Improvements:

• ktool no longer tries to guess the property getter/setter; it decodes it from the actual standard attr_string or generates it from the property name if none is specified. This avoids potential false positives and also clarifies when non-standard ones are used. We also decode whether a property is @ dynamic but do not encode that in the header yet.
• json output for properties now embeds attr_string, getter, and setter.
• More unit tests

New

• ImageHeader can now be created from arbitrarily values (ImageHeader.from_values())
• New Constructable LoadCommand + Segment currently used in tandem with old Segment wrapper.
• Unit tests!
• --fdec flag for dump, which forward declares private class imports

Fixes

• BytesIO fixes
• Fix lc insertion
• Fix with mmaped IO loader
• Program no longer crashes on unimplemented load commands
• Better VM segment input verification
• Properly null pad strings in structs
• Minor fixes in help messages
• Fix a crash with the codesign parser
• Fix (one) missing Chained Fixup structure.
• Fix relative method list parsing when relative addresses are negative
• Much better error handling in objc parsing
• Better info on bad load commands

• Added entitlement parsing
• Added Codesign information to GUI.
• Added json dumping for all mach-o/obj-c metadata
• Fixed an issue with Slice.find() on 32 bit files
• Further sped up struct loading substantially.
• Fixed issues with the patcher.

Tool

• Added ktool cs for interacting with codesigning info
• Added ktool json for dumping metadata as json
• Implemented ktool -V

Library

Documentation has been mostly updated to reflect new additions

Codesign Info

• Added CodesignInfo class, accessible via image.codesign_info
  • Entitlements accessible via codesign_info.entitlements

Serialization

The majority of objects containing relevant/important metadata about the image now contain a .serialize() function, which returns json-formattable dictionaries containing relevant metadata.

Struct Parser

• Redid the method in which .raw bytes are generated, eliminating the bottleneck created by that feature
• Added support for field "sizes" being another Struct type

File Backing

• Added a BackingFile class to further abstract out the direct file reads
• Added a SlicedBackingFile class to even further abstract out slices within a file. This fixed a confusing issue with the Slice.find() search utility in 32 bit files. This also abstracts out the patching functionality in order to keep that code more sane.

Patcher

• Refactored the majority of the LD64 class and its patching methods to now rebuild the entire header and write it via a single slice.patch()

Parsing of more codesign related information, along with adding more relevant data to json dumps to come.

Nothing too crazy, a lot of cleanup/refactoring, a few additions

GUI

• Structs will now be rendered with an indentation for readability
• Scrolling now supports PGUP + PGDOWN

Image

.vm_realign() - Computes image alignment (after the imageheader has been loaded) and sets up (or re-sets-up) the proper VM translator for the image.

Slice

CPU subtype now masks out the ABI bits.

VM Address Translation

Added a new VM translator, based on how low-level memory paging works. It is up to 2x faster than the old manual translator.

It falls back to the legacy method whenever images cannot be mapped to 16k/4k pages.

• Refactor: -> get_file_address() -> translate()
• New Attribute: .detag_kern_64 - Set this to automatically detag 64 bit kernel pointers being translated
• New Attribute: .detag_64 - Set this to detag 64 bit pointers (remove chained fixup data)

ObjC

A lot more fallback/safe failure stuff. You can now load a binary extracted using the default dyld_extractor.bundle (although objc data will still not be processable, as the offsets are entirely junk and likely irreperable without access to the entire cache).

Structs

.render_indented() - Return a string similar to str(struct_instance) but with linebreaks and indentation.

Internal:

Struct sizes now actually contain two values; the high 2 bytes represent the type of the field, and the low 2 bytes represent the size of the data in that field.

loading a field as bytes uses the size type_bytes | <size>. loading a field as a string is done with char_t[size]. Unsigned ints now internally use the 0x10000 mask instead of being represented by -<size>

kcache parsing:

• Added version string parsing

watchOS kcache extraction + minor bug fixes

TUI

• Add support for --no-mmap flag (needed on windows) in open command
• Only detag kext addresses on 64 bit kcaches

ktool.macho

MachOFile -

• Fix an issue where 32 bit little endian files wouldn't be recognized.

ktool.kcache -

• Add support for armv7k kernelcaches

This release builds on the baseline kernelcache processing shipped with 1.1.0

• Added a modified version of the python stdlib plistlib that actually works in ktool.kplistlib
• Added the kcache stuff to the help strings.

GUI

• Temporarily dropped the non-implemented title menus
• Added a new one named "Help", and moved the old default info text to that

GUI Kernel Cache Loader

• Added a full Loader specifically for KernelCaches
• Info specific to Kernel Caches
• Creates a View listing kexts and allow browsing their attributes and prelink info

ktool.kcache

KernelCache

Added Attributes:

• .prelink_info -> prelink info dict for the Mach Kernel psuedoextension
• .version -> Release Version (semantic) of the kernel

Kext

Added Attributes: Note: Do not rely on any of these not being empty

• .prelink_info -> Dict[str, Any] with info for this kext pulled straight from the plist in __PRELINK_INFO:__info.
• .development_region
• .executable_name
• .id -> will always be the same as .name
• .bundle_name -> Plaintext readable proper name of the bundle (like, "Libkern Extension")
• .package_type -> Always KEXT
• .info_string -> Informative (maybe) string describing the kext
• .version_str -> (Probably) the same as .version

This release mainly adds merged-type kernelcache parsing.

kmacho module

• Added support for LC_THREAD, LC_UNIXTHREAD, and LC_MAIN
• Added mapping for LC_LOAD_UPWARD_DYLIB

ktool module

New! ktool.kcache

KernelCache class: .mach_kernel: Image -> Image representing the Mach Kernel itself .kexts: List[Kext] -> List of Kext objects embedded in this kernel

Kext class: .name -> kext name .version: str -> kext version .mach_header -> Mach-O Header of the Kext .image -> Image representing this Kext

_VirtualMemoryMap

• Added support (and detagging) for 64 bit kernel address space.

Image

• added .entry_point attribute, which points to the address (in VM Space) of the program's entry point, if an LC_THREAD, LC_UNIXTHREAD, or LC_MAIN was in the header
• added .thread_state attribute, which holds the entry thread state if an LC_THREAD/LC_UNIXTHREAD was defined

GUI

• Fix: Window wont crash when a view is empty

:tada: :tada: :tada: :tada:

This release includes the changelog from 1.0.0rc0

Changes

• Terminal Output is now highlighted
• Table rendering in terminal output now has ansi support
• Added --class flag to dump
• Chained Fixup Support! iOS 15 binaries, etc can now have their imports processed. It's rudimentary, but damnit, it works :)
• Fixed ENCRYPTION_INFO_COMMAND handling
• Function Starts table processing
• Remove unneeded packaging dependency in favor of just hackily using setuptools's vendored version. This is a hack, but reportedly, procursus (an iOS bootstrap) cant handle building one of packaging's build dependencies, so this makes that work.
• heavily improved non-mmap implementation

Swift

Extremely Rudimentary swift processing. It's just the groundwork, and shouldn't really be counted as a feature yet, but binaries with swift wont break it anymore, and it can read some basic info about swift types (and list swift types)

Structs

• Rewrite it again to handle signed int field processing automatically
• Rewrite it again again so my IDE properly recognizes the fields exist, without slowing down processing too much

Internal Stuff

• Migrated the entire project to poetry and refactored some of the project layout.

:tada: :tada: :tada: :tada:

Changes

• Chained Fixup Support! iOS 15 binaries, etc can now have their imports processed. It's rudimentary, but damnit, it works :)
• Fixed ENCRYPTION_INFO_COMMAND handling
• Function Starts table processing
• Remove unneeded packaging dependency in favor of just hackily using setuptools's vendored version. This is a hack, but reportedly, procursus (an iOS bootstrap) cant handle building one of packaging's build dependencies, so this makes that work.
• heavily improved non-mmap implementation

Swift

Extremely Rudimentary swift processing. It's just the groundwork, and shouldn't really be counted as a feature yet, but binaries with swift wont break it anymore, and it can read some basic info about swift types (and list swift types)

Structs

• Rewrite it again to handle signed int field processing automatically
• Rewrite it again again so my IDE properly recognizes the fields exist, without slowing down processing too much

Tool

• Initial file load in the GUI is now near-instant thanks to a refactor in how headers are loaded.
• non-mmaped processing now operates at a reasonable speed.

Library

• rewrite bio (non-mmap) backend stuff to use a bytearray buffer loaded at init instead of repeated seek()->read() calls.
• non-mmap now automatically kicks in if mmap fails
• GUI now supports a target-function-based lazy loading string buffer, which is used for header lazy-loading

Tool

• open now has --hard-fail flag, which will cause open to fail (and print a backtrace) if it hits any exceptions loading content
• GUI: objc header generation now functions on platforms without semaphore implementations

Library

Symbol

• Class now conforms to Constructable class.
  • .from_image() when loading from an nlist(32/64) struct
  • .from_values when loading from values we've already decoded elsewhere
• .addr -> .address
• .ext -> .external
• .types - new array of strings containing different types, if they're specified in an nlist64 entry (from a symtab)

Table Refactor

Essentially, table rendering in the GUI is now instant, with zero load time/freezing and no lag while scrolling large tables. This includes Hex Dumps

Class

• Added .dividers: bool attribute; when set to True, a real "table" with outlined cells/columns will be drawn.
• Refactored entire class to lazily process, render, and cache the dumps as the buffer is scrolled instead of every time it gets loaded. This makes everything with tables nearly instance

Hex Dump Tables

• HexDumpTables now override the fetch() method of Table, and lazily load/replace the .rows() attribute through only decoding the bytes we need at the time; and disables the cache, instead of loading all of the bytes into the decoding function at once.

Table Rendering Logic

• ScrollingDisplayBuffer().process_lines() no longer handles rendering tables; instead we fetch() the needed content in rendered_lines_from() every time an update is requested
• Pinning is no longer properly implemented

Library

• Added support for loading/parsing/dumping 32-bit binaries. (tested on armv7, should work on x86 (32) as well)
• General Code Cleanup

Mainly bugfixes, and a refactor that can potentially cut objc loading speed in half.

Tool

• GUI: Better multithreading for the objc header syntax highlighting in the objc loader.

Library

• Fix a lot of potential recursion/class-loading errors in objc
• FIx method/property sorting in public API
• Implement a Queueing system for loading Classes/Protocols/Categories
• Implement a Cache for loaded Classes/Protocols/Categories
• Update Repo Address

This release includes a few bugfixes, along with a massive internal refactor that shouldn't affect any of the API or process.

Tool

• GUI
  • Fix Exit button
  • Fix crash regression when image has no UUID
• add --membench global arg (benchmarks memory allocations across program)
• Add --use-stab-for-sel dump arg, which will force using the symtab to get selectors for methods, instead of using the actual selector strings.

Library

Bugfixes

• Fix a crash when unrecognized load commands are read
• Fix a crash when symtab is entirely empty (poorly extracted dyld binaries)
• Fix a crash when class/protocol pointers are bad (poorly extracted dyld binaries)

Refactors

• Constructable Abstract Base Class - Renamed from_bytes to from_image
• Image - Remove deprecated .linked attribute (replaced with .linked_images)
• Image - Add .import_table: Dict[int, Symbol], which is a lookup table for imported symbols
• Image - Add .export_table: Dict[int, Symbol], which is a lookup table for exported symbols
• _VirtualMemoryMap - Add .vm_check(vm_address) -> bool
• .objc - Implement Constructable API in ObjCImage, Class, Protocol, Category, Method, Property, and Ivar. This will allow api-compatible loading and header dumping of values obtained at runtime, not from an image.

Tool

• Should run a lot faster due to internal Library changes. Benchmarks have shown anywhere from 30-60% reductions in runtime.
• GUI
  • Moved Imports and Exports out of "Binding" group and merge the imports tables

Library

• Massive Speedups
  • Refactor almost all of the internals in the Struct class to speed things up
  • Added some caches for some other huge speedups
    • Loaded struct cache
    • Loaded CString cache
    • Typeresolver cache
  • Finished implementing MethodList, implemented to avoid duplicate code blocks in objc.py
  • Stopped using structs in MethodList parsing to make things a bit faster
• Expansive Refactor in the Image class.
  • .linked -> .linked_images
  • Binding tables merged, moved to new .imports: List[Symbol]
  • Exports moved to new .exports: List[Symbol], Export Trie object moved to .export_trie
• Added .attr to Symbol class, to be used with import symbols
• New LD64 class; holds methods for editing linking information, etc.
  • Moved load command editing functions to this class

Library

• Implemented the code and fixed some issues to allow loading (and patching) BytesIO objects (raw bytes in memory that dont exist on disk).

Example usage of this can be seen here: https://gist.github.com/KritantaDev/b577dafe844d26350b051b482bb71268#file-script-py-L20

Library

• Add macho_combine(slices: List[Slice]) -> BytesIO to public API
• Add ignore.OBJC_ERRORS to util class (used in ObjC header gen). Default is True, setting it to False will crash the program when an error is encountered loading any objc metadata.

Tool

• Setting --hard-fail on the CLI dump command will toggle the above ignore.OBJC_ERRORS to True, crashing the program whenever an error is encountered loading OBJC metadata.

Library

• Image class now has attributes .base_name and .install_name, designed to replace the ambiguous .name attribute, which is now deprecated. .install_name will be the Install name of the image, or "" if one isn't specified. .base_name will be the base name (not including path) of the install name, or basename of the filename if one isn't included. the .name attribute is now deprecated and shouldn't be used.
  • This should fix any issues with header gen, GUI related things, etc, regarding non-library images.

Tool

• info command now shows full Install name instead of a basename of the install name on Libraries.
• GUI should now always restore terminal even when exceptions break out of the window.py file.
• Fixed a minor range issue in GUI flavor text, updated the "welcome" GUI text.
• GUI now uses cleaner basenames of Install names or Image paths when displaying them.
• The debug menu in the GUI (click ktool in the top left corner), which shows the output of log, now highlights errors and warnings.

CLI Tool changes

• Missing flags error message will now show the original flag text (--headers instead of do_headers)
• Always attempt to tear down curses GUI after it closes (not just after caught exceptions)
• Implement the new public library API in the CLI tool
• Update some docstrings
• Add --no-mmap global flag, which loads a binary without using the mmap module (this is beyond horribly slow at the moment).

Library changes

• Add a new public API
• highly limit the classes imported via the init script in the ktool module (to a few classes, and the new public api
• Make log error output redirectable, and by default pipe it to stderr
• move the Table class to .util, since it can be used for both .window and CLI output
• Add the base structure for fixup processing (not yet implemented, but the pieces are there now.)
• A ton of project cleanup

• A ton of project cleanup, internal refactoring, etc.
• Clarified some help strings, README, etc.
• Fixed the 'edit' command
• Fixed the fat MachO Generator
• Implemented some new tests (which highlighted the two above issues)

• Cleaned up some ObjC method list code (this should not affect output)
• partially implemented support for the 0xD0 binding opcode (although binding info isn't quite right, it doesn't crash now.)

This release implements ObjC Method List "alternative" encoding styles for Categories and Protocols. (direct selectors and relative offsets).

Categories and Protocols encoded with these enabled (DyldExtractor output) should now load correctly.

I've forgotten to do github releases for a few months or however long. Starting again now.

Changes with this version:

• Refactored all variable/classnames using 'Library' to 'Image'. This is more accurate to what MachO Objects are called; ( they're really just called images, "Library" was a leftover from a previous project that only targeted libraries and this should've been refactored long ago. )

Regarding the changes between the previous release and this one, checking the 100+ commits since then is likely your best bet.

• Fixed quite a few issues with load command, binding info, etc processing
• Added some documentation and improved general state of project source code.

add lipo command

• Rewrite the header output processing
• Fix empty type declarations
• Fix category dumping in header refactor
• Fix some issues with type processing (UInt8, empty struct)
• Add some error handling to category parsing
• Improve Testing
• Add protocol list
• Add more thorough protocol recognition
• Add protocol header dumping

• Fix a failure on missing library ordinal (likely due to an inaccuracy in binding info processing
• Fix some issues with DyldEx Processing
• Fix some misc error handling

• Highly Improve Classlist Search Flexibility
• Fix recognition for certain arm64e binaries
• Add support for dumping Categories (Class Extensions)
• Fix some error handling
• Fix a couple of crashes
• Added some lookup tables to improve speeds

• Add improved error handling
• Add Option to pass existing objc class to TBD Generator ** For large objc libraries (e.g. SpringBoard.framework), when a script is dumping headers and the .tbd, avoiding having two ObjCLibraries loaded and in-memory results in a 2x + speedup

• Add VM Table Cache which massively speeds up repeat lookups
• Add base help menu for ktool with no args
• Add Accurate import for superclasses
• Fix regression in SuperClass resolution
• Add benchmark argument