An extremely configurable markdown reverser for Python3.

Overview

🔄 Unmarkd

codecov Code style: black CI PyPI - Downloads

A markdown reverser.


Unmarkd is a BeautifulSoup-powered Markdown reverser written in Python and for Python.

Why

This is created as a StackSearch (one of my other projects) dependency. In order to create a better API, I needed a way to reverse HTML. So I created this.

There are similar projects (written in Ruby) but I have not found any written in Python (or for Python) later I found a popular library, html2text. But Unmarkd still is still better. See comparison.

Installation

You know the drill

pip install unmarkd

Known issues

  • Nested lists are not properly indented (#4) Fixed in #11
  • Blockquote bug (#18) Fixed in #23

Comparison

TL;DR: Html2Text is fast. If you don't need much configuration, you could use Html2Text for the little speed increase.

Click to expand

Speed

TL;DR: Unmarkd < Html2Text

Html2Text is basically faster:

Benchmark

(The DOC variable used can be found here)

Unmarkd sacrifices speed for power.

Html2Text directly uses Python's html.parser module (in the standard library). On the other hand, Unmarkd uses the powerful HTML parsing library, beautifulsoup4. BeautifulSoup can be configured to use different HTML parsers. In Unmarkd, we configure it to use Python's html.parser, too.

But another layer of code means more code is ran.

I hope that's a good explanation of the speed difference.

Correctness

TL;DR: Unmarkd == Html2Text

I actually found two html-to-markdown libraries. One of them was Tomd which had an incorrect implementation:

Actual results

It seems to be abandoned, anyway.

Now with Html2Text and Unmarkd:

Epic showdown

In other words, they work

Configurability

TL;DR: Unmarkd > Html2Text

This is Unmarkd's strong point.

In Html2Text, you only have a limited set of options.

In Unmarkd, you can subclass the BaseUnmarker and implement conversions for new tags (e.g. ), etc. In my opinion, it's much easier to extend and configure Unmarkd.

Unmarkd was originally written as a StackSearch dependancy.

Html2Text has no options for configuring parsing of code blocks. Unmarkd does

Documentation

Here's an example of basic usage

I love markdown!")) # Output: **I *love* markdown!**">
import unmarkd
print(unmarkd.unmark("I love markdown!"))
# Output: **I *love* markdown!**

or something more complex (shamelessly taken from here):

Sample Markdown

This is some basic, sample markdown.

Second Heading

  • Unordered lists, and:
    1. One
    2. Two
    3. Three
  • More

Blockquote

And bold, italics, and even italics and later bold. Even strikethrough. A link to somewhere.

And code highlighting:

var foo = 'bar';

function baz(s) {
   return foo + ':' + s;
}

Or inline code like var foo = 'bar';.

Or an image of bears

bears

The end ...

""" print(unmarkd.unmark(html_doc))">
import unmarkd
html_doc = R"""

Sample Markdown

This is some basic, sample markdown.

Second Heading

  • Unordered lists, and:
    1. One
    2. Two
    3. Three
  • More

Blockquote

And bold, italics, and even italics and later bold. Even strikethrough. A link to somewhere.

And code highlighting:

var foo = 'bar';

function baz(s) {
   return foo + ':' + s;
}

Or inline code like var foo = 'bar';.

Or an image of bears

bears

The end ...

""" print(unmarkd.unmark(html_doc))

and the output:

    # Sample Markdown


    This is some basic, sample markdown.

    ## Second Heading



    - Unordered lists, and:
     1. One
     2. Two
     3. Three
    - More

    >Blockquote


    And **bold**, *italics*, and even *italics and later **bold***. Even ~~strikethrough~~. [A link](https://markdowntohtml.com) to somewhere.

    And code highlighting:


    ```js
    var foo = 'bar';

    function baz(s) {
       return foo + ':' + s;
    }
    ```


    Or inline code like `var foo = 'bar';`.

    Or an image of bears

    ![bears](http://placebear.com/200/200)

    The end ...

Extending

Brief Overview

Most functionality should be covered by the BasicUnmarker class defined in unmarkd.unmarkers.

If you need to reverse markdown from StackExchange (as in the case for my other project), you may use the StackOverflowUnmarker (or it's alias, StackExchangeUnmarker), which is also defined in unmarkd.unmarkers.

Customizing

If the above two classes do not suit your needs, you can subclass the unmarkd.unmarkers.BaseUnmarker abstract class.

Currently, you can optionally override the following methods:

  • detect_language (parameters: 1)
    • Parameters:
      • html: bs4.BeautifulSoup
    • When a fenced code block is approached, this function is called with a parameter of type bs4.BeautifulSoup passed to it; this is the element the code block was detected from (i.e. pre).
    • This function is responsible for detecting the programming language (or returning '' if none was detected) of the code block.
    • Note: This method is different from unmarkd.unmarkers.BasicUnmarker. It is simpler and does less checking/filtering

But Unmarkd is more flexible than that.

Customizable constants

There are currently 3 constants you may override:

  • Formats: NOTE: Use the Format String Syntax
    • UNORDERED_FORMAT
      • The string format of unordered (bulleted) lists.
    • ORDERED_FORMAT
      • The string format of ordered (numbered) lists.
  • Miscellaneous:
    • ESCAPABLES
      • A container (preferably a set) of length-1 str that should be escaped
Customize converting HTML tags

For an HTML tag some_tag, you can customize how it's converted to markdown by overriding a method like so:

from unmarkd.unmarkers import BaseUnmarker
class MyCustomUnmarker(BaseUnmarker):
    def tag_some_tag(self, child) -> str:
        ...  # parse code here

To reduce code duplication, if your tag also has aliases (e.g. strong is an alias for b in HTML) then you may modify the TAG_ALIASES.

If you really need to, you may also modify DEFAULT_TAG_ALIASES. Be warned: if you do so, you will also need to implement the aliases (currently em and strong).

Utility functions when overriding

You may use (when extending) the following functions:

  • __parse, 2 parameters:
    • html: bs4.BeautifulSoup
      • The html to unmark. This is used internally by the unmark method and is slightly faster.
    • escape: bool
      • Whether to escape the characters inside the string or not. Defaults to False.
  • escape: 1 parameter:
    • string: str
      • The string to escape and make markdown-safe
  • wrap: 2 parameters:
    • element: bs4.BeautifulSoup
      • The element to wrap.
    • around_with: str
      • The character to wrap the element around with. WILL NOT BE ESCPAED
  • And, of course, tag_* and detect_language.
Comments
  • Nested lists of same type don't work

    Nested lists of same type don't work

    Both unordered and ordered list don't work when nested of the same type:

    Two nested ordered lists

    HTML:

    <ol>
        <li>Top level 1</li>
        <li>Top level 2
            <ol>
                <li>A</li>
                <li>B</li>
                <li>C</li>
            </ol>
        </li>
        <li>Top level 3</li>
    </ol>
    

    Output:

    1. Top level 1
     2. Top level 2
            
     1. A
     2. B
     3. C
     3. Top level 3
    

    Two nested unordered lists

    HTML:

    <ul>
        <li>Top level 1</li>
        <li>Top level 2
            <ul>
                <li>A</li>
                <li>B</li>
                <li>C</li>
            </ul>
        </li>
        <li>Top level 3</li>
    </ul>
    

    Output:

    - Top level 1
    - Top level 2
            
    - A
    - B
    - C
    - Top level 3
    
    bug good first issue reproduced 
    opened by sirnacnud 3
  • [ImgBot] Optimize images

    [ImgBot] Optimize images

    Beep boop. Your images are optimized!

    Your image file size has been reduced by 39% 🎉

    Details

    | File | Before | After | Percent reduction | |:--|:--|:--|:--| | /assets/correct.png | 372.04kb | 224.67kb | 39.61% | | /assets/tomd_cant_handle.png | 347.74kb | 210.22kb | 39.55% | | /assets/benchmark.png | 219.28kb | 141.36kb | 35.53% | | | | | | | Total : | 939.06kb | 576.25kb | 38.64% |


    📝 docs | :octocat: repo | 🙋🏾 issues | 🏪 marketplace

    ~Imgbot - Part of Optimole family

    opened by imgbot[bot] 1
  • Fix indent getting added to list children that weren't other lists

    Fix indent getting added to list children that weren't other lists

    I was running in to an issue where list items using tags where getting indented when they shouldn't of been.

    Example:

    <ol>
        <li>A</li>
        <li>B</li>
        <li><b>C</b></li>
    </ol>
    

    Output:

    1. A
    2. B
    3.     **C**
    

    I added a test for this case as well. When doing the roundtrip style test, this indentation got lost, so I made the test compare the markdown output.

    opened by sirnacnud 1
  • Support for tables

    Support for tables

    While Unmarkd currently supports tables, it spits out the html it was given. It would be nice if it supported tables:

    | Syntax      | Description |
    | ----------- | ----------- |
    | Header      | Title       |
    | Paragraph   | Text        |
    
    enhancement 
    opened by ThatXliner 1
  • Nested lists are not properly indented

    Nested lists are not properly indented

    When the following HTML block is parsed:

    <ul>
        <li>Unordered lists, and:
            <ol>
                <li>One</li>
                <li>Two</li>
                <li>Three</li>
            </ol>
        </li>
        <li>More</li>
    </ul>
    

    The output is incorrect:

     * Unordered lists, and:
     0. One
     1. Two
     2. Three
     * More
    
    bug 
    opened by ThatXliner 1
  • Blockquote bug

    Blockquote bug

    Apply this patch:

    diff --git a/tests/test_roundtrip.py b/tests/test_roundtrip.py
    index a836024..5c1e097 100644
    --- a/tests/test_roundtrip.py
    +++ b/tests/test_roundtrip.py
    @@ -1,10 +1,9 @@
     import unicodedata
     
     import markdown_it
    -from hypothesis import assume, example, given
    -from hypothesis import strategies as st
    -
     import unmarkd
    +from hypothesis import assume, example, given, reproduce_failure
    +from hypothesis import strategies as st
     
     md = markdown_it.MarkdownIt()
     
    @@ -17,6 +16,7 @@ def helper(text: str, func=unmarkd.unmark) -> None:
     
     
     @given(text=st.text(st.characters(blacklist_categories=("Cc", "Cf", "Cs", "Co", "Cn"))))
    [email protected]_failure("6.10.1", b"AAEADgEADgEADgA=")
     def test_roundtrip_commonmark_unmark(text):
         assume(unicodedata.normalize("NFKC", text) == text)
         helper(text)
    
    
    
    

    Or add an example with text=">>>". Tests will fail

    bug 
    opened by ThatXliner 0
  • Update README for better comparison

    Update README for better comparison

    1. html2text is fast but not very configurable (there's only so any options)
    2. Tomd sucks
    3. Add an unmarker (with html2text-style configuration) to prove that unmarkd's configurability is at least equal to html2text
    documentation 
    opened by ThatXliner 0
  • Use a more reliable markdown parser

    Use a more reliable markdown parser

    Instead of using commonmark, maybe https://github.com/executablebooks/markdown-it-py, https://github.com/trentm/python-markdown2, https://github.com/lepture/mistune, or https://github.com/Python-Markdown/markdown.

    Also, I found tomd which might render this project useless 😬

    tests 
    opened by ThatXliner 0
  • Cannot handle nested bold and italics

    Cannot handle nested bold and italics

    When encountering input like <em>Italic and <strong>bold and italic</strong></em>, the output is wrong, usually shadowed by the outer tag (in this case, <em>)

    bug 
    opened by ThatXliner 0
  • Optimize code

    Optimize code

    I've noticed that unmarkers.BaseUnmarker been documented as an "abstract base class" when we're actually using it otherwise.

    Also, there's some dead code and we should actually sprinkle @staticmethod on some of them.

    Here's my idea:

    • Move all the tag_* methods in BaseUnmarker ➡️ BasicUnmarker
    • Rename: BaseUnmarker ➡️ AbstractUnmarker
    • Alias: BaseUnmarker ➡️ BasicMarker
    • Run shed on the whole codebase (with --refactor)

    Version bump: minor

    enhancement 
    opened by ThatXliner 0
  • Save CSS information

    Save CSS information

    1. Parse any css files or style tags found. Save it
    2. When a class attribute is found, try to resolve it to the css
    3. Add the resolved to the style attribute: convert to inline css
    enhancement 
    opened by ThatXliner 1
Releases(v0.1.9)
Owner
ThatXliner
I code Python. To me, programming is a logic puzzle. A fun one :D
ThatXliner
My Analysis of the VC4 Assembly Code from the RPI4

My Analysis of the VC4 Assembly Code from the RPI4

Nicholas Starke 31 Jul 13, 2022
A community-driven python bot that aims to be as simple as possible to serve humans with their everyday tasks

JARVIS on Messenger Just A Rather Very Intelligent System, now on Messenger! Messenger is now used by 1.2 billion people every month. With the launch

Swapnil Agarwal 1.3k Jan 07, 2023
Med to csv - A simple way to parse MedAssociate output file in tidy data

MedAssociates to CSV file A simple way to parse MedAssociate output file in tidy

Jean-Emmanuel Longueville 5 Sep 09, 2022
Contains a Jupyter Notebook for calculating remaining plants required based on field/lathhouse data.

Davis-Sunflowers-Su21 Project goals: Plants influence their reproduction and mating system in many ways. Various factors such as time of flowering, ab

1 Feb 10, 2022
tool to automate exploitation of android degubg bridge vulnerability

DISCLAIMER DISCLAIMER: ANY MALICIOUS USE OF THE CONTENTS FROM THIS ARTICLE WILL NOT HOLD THE AUTHOR RESPONSIBLE HE CONTENTS ARE SOLELY FOR EDUCATIONAL

6 Feb 12, 2022
Never get kicked for inactivity ever again!

FFXIV AFK Bot Tired of getting kicked from games due to inactivity? This Bot will make random movements in random intervals to prevent you from gettin

5 Jan 12, 2022
An app that mirrors your phone to your compute and maps controller input to the screen

What is 'Dragalia Control'? An app that mirrors your phone to your compute and maps controller input to the screen. Inputs are mapped specifically for

1 May 03, 2022
It was created to conveniently respond to events such as donation, follow, and hosting using the Alert Box provided by twip to streamers

This library is not an official library of twip. It was created to conveniently respond to events such as donation, follow, and hosting using the Alert Box provided by twip to streamers.

junah201 8 Nov 19, 2022
Projeto para ajudar no aprendizado da linguagem Pyhon

Economize Este projeto tem o intuito de criar desáfios para a codificação em Python, fazendo com que haja um maior entendimento da linguagem em seu to

Lucas Cunha Rodrigues 1 Dec 16, 2021
A Notifier Program that Notifies you to relax your eyes Every 15 Minutes👀

Every 15 Minutes is an application that is used to Notify you to Relax your eyes Every 15 Minutes, This is fully made with Python and also with the us

FSP Gang s' Admin 1 Nov 03, 2021
Projects using the Tkinter module in Python!

Tkinter projects This repository includes some Tkinter projects made by me. All of these are simple to understand. I create apps with good functionali

Amey 0 Sep 24, 2021
MDAnalysis tool to calculate membrane curvature.

The MDAkit for membrane curvature analysis is part of the Google Summer of Code program and it is linked to a Code of Conduct.

MDAnalysis 19 Oct 20, 2022
Feature engineering library that helps you keep track of feature dependencies, documentation and schema

Feature engineering library that helps you keep track of feature dependencies, documentation and schema

28 May 31, 2022
The repository for AnyMacro: a Fusion360 Add-In

AnyMacro AnyMacro is an Autodesk® Fusion 360™ add-in for chaining multiple commands in a row to form Macros. Macros are created from a set of commands

1 Jan 07, 2022
Check COVID locations of interest against Google location history

Location of Interest Checker Script to compare COVID locations of interest to Google location history. The script produces a map plot (as shown below)

9 Mar 30, 2022
Program to send ROM files to Turbo Everdrive; reverse-engineered and designed to be platform-independent

PCE_TurboEverdrive_USB What is this "TurboEverdrive USB" thing ? For those who have a TurboEverdrive v2.x from krikzz.com, there was originally an opt

David Shadoff 10 Sep 18, 2022
A simple and easy to use Python's PIP configuration manager, similar to the Arch Linux's Java manager.

PIPCONF - The PIP configuration manager If you need to manage multiple configurations containing indexes and trusted hosts for PIP, this project was m

João Paulo Carvalho 11 Nov 30, 2022
Generates images with semantic content from distribution A in the style of distribution B

A2B Generates images with semantic content from distribution A in the style of d

Richard Herbert 2 Dec 27, 2021
Run Python code right in your Telegram messages

Run Python code right in your Telegram messages Made with Telethon library, TGPy is a tool for evaluating expressions and Telegram API scripts. Instal

29 Nov 22, 2022
Objetivo: de forma colaborativa pasar de nodos de Dynamo a Python.

ITTI_Ed01_De-nodos-a-python ITTI. EXPERT TRAINING EN AUTOMATIZACIÓN DE PROCESOS BIM: OFFICIAL DE AUTODESK. Edición 1 Enlace al Master Enunciado: Traba

1 Jun 06, 2022