Dependencies error during the installation of the quark-engine on macOS Catalina - 10.15.7.

Error : - pkg_resources.DistributionNotFound: The 'androguard==3.4.0a1' distribution was not found and is required by quark-engine

Detect CWE-319 in Android Application (ovaa.apk)

This scenario seeks to find the Cleartext Transmission of Sensitive Information. See CWE-319 for more details.

Let's use this APK and the above APIs to show how the Quark script finds this vulnerability. This sample uses the package Retrofit to request Web APIs, but the APIs use cleartext protocols.

We first design a detection rule setRetrofitBaseUrl.json to spot on behavior that sets the base URL of the Retrofit instance. Then, we loop through a custom list of cleartext protocol schemes and use API behaviorInstance.hasString to filter arguments that are URL strings with cleartext protocol.

Quark Script CWE-319.py

from quark.script import runQuarkAnalysis, Rule

SAMPLE_PATH = "./ovaa.apk"
RULE_PATH = "setRetrofitBaseUrl.json"

PROTOCOL_KEYWORDS = [
    "http",
    "smtp",
    "ftp"
]


ruleInstance = Rule(RULE_PATH)
quarkResult = runQuarkAnalysis(SAMPLE_PATH, ruleInstance)

for setRetrofitBaseUrl in quarkResult.behaviorOccurList: 
    for protocol in PROTOCOL_KEYWORDS:
        
        regexRule = f"{protocol}://[0-9A-Za-z./-]+"
        cleartextProtocolUrl = setRetrofitBaseUrl.hasString(regexRule, True)
        
        if cleartextProtocolUrl:
            print(f"CWE-319 detected!")
            print(f"Here are the found URLs with cleartext protocol:")
            print("\n".join(cleartextProtocolUrl))

Quark Rule: setRetrofitBaseUrl.json

{
    "crime": "Set Retrofit Base Url",
    "permission": [],
    "api": 
    [
        {
            "descriptor": "()V",
            "class": "Lretrofit2/Retrofit$Builder;",
            "method": "<init>"
        },
        {
            "descriptor": "(Ljava/lang/String;)Lretrofit2/Retrofit$Builder;",
            "class": "Lretrofit2/Retrofit$Builder;",
            "method": "baseUrl"
        }
    ],
    "score": 1,
    "label": []
}

Quark Script Result

$ python3 CWE-319.py
CWE-319 detected!
Here are the found URLs with cleartext protocol:
http://example.com./api/v1/

Porting androguard version 3.4 to quark-engine project to prevent androguard from no longer being maintained.

In the past using androguard, we all have to rely on pip install androguard from Github, but there is a problem, if something goes wrong with androguard, quark-engine might crash.

But in fact, we only need the decompile function of androguard, so I ported this function to our project.

In addition to improving the stability of quark-engine, it also increases the speed of pipenv installation.

Detect CWE-532 in Android Application (dvba.apk)

This scenario seeks to find insertion of sensitive information into Log file. See CWE-532 for more details.

Let’s use this APK and the above APIs to show how the Quark script finds this vulnerability.

First, we use API findMethodInAPK to locate the method log.d. Then we use API methodInstance.getArguments to get the argument that input to log.d. Finally, we use keywords such as "token", "password", and "decrypt" to check if arguments include sensitive data. If the answer is YES, that may cause sensitive data leakage into log file.

You can use your own keywords in the keywords list to detect sensitive data.

API Spec

findMethodInAPK(samplePath, targetMethod)

• Description: Find the target method in APK
• params:
  1. samplePath: Target file
  2. targetMethod: A python list contains class name, method name, and descriptor of target method
• return: python list contains caller method instance of target method

Detect CWE-532 in Android Application (dvba.apk)

Quark Script CWE-532.py

from quark.script import findMethodInAPK

SAMPLE_PATH = "dvba.apk"
TARGET_METHOD = [
    "Landroid/util/Log;",                       # class name
    "d",                                        # method name
    "(Ljava/lang/String; Ljava/lang/String;)I"  # descriptor
]
CREDENTIAL_KEYWORDS = [
    "token",
    "decrypt",
    "password"
]

methodsFound = findMethodInAPK(SAMPLE_PATH, TARGET_METHOD)

for debugLogger in methodsFound:
    arguments = debugLogger.getArguments()

    for keyword in CREDENTIAL_KEYWORDS:
        if keyword in arguments[1]:
            print(f"CWE-532 is detected in method, {debugLogger.fullName}") 

Quark Script Result

$ python CWE-532.py 
CWE-532 is detected in method, Lcom/google/firebase/auth/FirebaseAuth; d (Lc/c/b/h/o;)V

Use the following code can get the url and the ip address:

from androguard.misc import AnalyzeAPK
import re

a,d,dx= AnalyzeAPK("Ahmyth.apk")


ipv4_address = re.compile(r"\b(?:[1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-2][0-3])\.(?:[0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-5][0-5])\.(?:[0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-5][0-5])\.(?:[0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-5][0-5])\b")

regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))"



for i in dx.get_strings():
    url = re.findall(regex,i.get_value())

    if url:

        print("[URL Found]")

        print([x[0] for x in url])

    ips = re.findall(ipv4_address,i.get_value())

    if ips:

        print("[IP Found]")

        print(ips)

Help section for --multi-process made more descriptive in respect to max number of process that can be used. w.r.t issue https://github.com/quark-engine/quark-engine/issues/315

Description

Please refer here. For the replacement of Androguard, I want to write tests to improve the test coverage of Quark. This is the final PR. (You can find the previous PR here )

In this PR, I focus on these files.

• quark/Objects/analysis.py
• quark/Objects/quarkrule.py
• quark/report.py
• quark/freshquark.py
• all seven files in quark/utils (colors.py, graph.py, pprint.py, output.py, etc.)

Code Changes

• For the existing tests: Divide them by their test scenarios.
• For the new tests: Add them according to two strategies and the coding guideline discussed in the above issue.

| Files | # Tests added for normal inputs | # Tests added for error inputs | # Tests modified | | ------------------- | :-----------------------------: | :----------------------------: | :--------------: | | test_analysis.py | 1 | 0 | 0 | | test_quarkrule.py | 1 | 4 | - | | test_report.py | 4 | 6 | - | | test_freshquark.py | 2 | 0 | - | | test_colors.py | 1 | 0 | - | | test_graph.py | 3 | 0 | - | | test_output.py | 3 | 0 | - | | test_pprint.py | 5 | 0 | - | | test_regex.py | 11 | 3 | - | | test_tools.py | 5 | 1 | 1 | | test_weight.py | 0 | 0 | 3 | | Total | 36 | 14 | 4 |

Related Discussions

1. issue https://github.com/quark-engine/gsoc2021-ShengFengLu/issues/1
2. Discussion https://github.com/quark-engine/quark-engine/discussions/173

With the following pull request we (me, @cryptax, @Dil3mm3 and @3aglew0) propose you to add another option to print a report based on labels specified inside a rule.

We have noticed they are not used and it could be interesting to print a short report taking into consideration these values. Here an example of output where it is printed for each label (found inside the rules) a description (see explanation below), the number of rules where this label is contained and other detailes described better below.

This option permits to print a report based on label with two different levels of details

1. quark -a malware_to_be_analysed.apk -r rule_dir -l max print the maximum score for each label (as image above), this would permit us to understand in which topic (represented by label) a malware is more aggressive. For example, looking at the previous output we can see the malware performs with success malicious action related to location, calllog and sms.
2. quark -a malware_to_be_analysed.apk -r rule_dir -l detailed print a detail report with all the previous information plus:
   • Number of rules (with that label) which have a score >= 80%
   • Average score and standard deviation (computed over the all the scores obtained by that specific label). Interesting considerations could be the following: label with high average and low standard deviation would allow us to say the malware performs a series of malicious actions (with success); then, a high standard deviation means there are some rules which take high score so the malware performs with success only some actions with that label; finally, a low standard deviation and a low average on a certain label means the malware is not performing malicious action on that topic. Example of output:

The column description allows to add a short and representative sentence about a label, for example for the callog the relative description is Retrieve or manipulate sensitive data from call log. In order to implement a flexible solution we have thought to add a csv file in the same directory of rules with the following structure label,description. We have chosen csv extension because it is easy to manipulate and it wasn't possible to use a json format since in that folder all json files are interpreted as rules. If this file is not present or a label,description pair is absent, the corresponding cell in the label report is filled with -. Example of output

I leave here a sample of the csv file to be put in the folder of the rules (label_desc.csv)

Do not hesitate to contact me for any type of clarification

CWE Showcases

• CWE-020 Improper Input Validation
• CWE-089 Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')
• CWE-094 Improper Control of Generation of Code ('Code Injection')
• CWE-312 Cleartext Storage of Sensitive Information
• CWE-319 Cleartext Transmission of Sensitive Information
• CWE-327 Use of a Broken or Risky Cryptographic Algorithm
• CWE-532 Insertion of Sensitive Information into Log File
• CWE-749 Exposed Dangerous Method or Function
• CWE-780 Use of RSA Algorithm without OAEP
• CWE-798 Use of Hard-coded Credentials
• CWE-921 Storage of Sensitive Data in a Mechanism without Access Control
• CWE-926 Improper Export of Android Application Components

Add new feature for generate Quark report. With the following command, we can easily analyze the Android sample and output the web report.

See the demo here.

quark -a sample.apk -s -w quark_report.html

Hey everyone!

Is your feature request related to a problem? Please describe. Quark does not separate optional dependency, like Click, from required dependency (I suppose everything else). Since Quark can be used as a module, and in this case Click it is not required at all, will solve some compatibility issues with other libraries, i.e. celery >= 5.0.0.

Describe the solution you'd like

Use the Optional Dependency feature in setup.py to separate the Click package from the remaining requirements. Explain in Readme.md how the user should call setup.py to be able to use the cli commands

Describe alternatives you've considered None

Additional context

If the solution is considered acceptable, or another solution that I did not think of, for this issue is found, I can work on the implementation and the PR myself.

Detect CWE-328 in Android Application (allsafe.apk)

This scenario seeks to find the use of weak Hash. See CWE-328 for more details.

Let’s use this APK and the above APIs to show how the Quark script finds this vulnerability.

First, we use API findMethodInAPK(samplePath, targetMethod) to find the method MessageDigest.getInstance(). Next, we use API methodInstance.getArguments() with a list to check if the method uses weak hashing algorithms. If YES, that causes CWE-328 vulnerability.

Quark Script CWE-328.py

from quark.script import findMethodInAPK

SAMPLE_PATH = "./allsafe.apk"

TARGET_METHOD = [
    "Ljava/security/MessageDigest;",                        # class name
    "getInstance",                                          # method name
    "(Ljava/lang/String;)Ljava/security/MessageDigest;"     # descriptor
]

HASH_KEYWORDS = [
    "MD2",
    "MD4",
    "MD5",
    "PANAMA",
    "SHA-0",
    "SHA-1",
    "HAVAL-128",
    "RIPEMD-128"
]

methodsFound = findMethodInAPK(SAMPLE_PATH, TARGET_METHOD)

for setHashAlgo in methodsFound:
    arguments = setHashAlgo.getArguments()

    for keyword in HASH_KEYWORDS:
        if keyword in arguments[0]:
            print(f"CWE-328 is detected in method, {setHashAlgo.fullName}")

Quark Script Result

$ python CWE-328.py
CWE-328 is detected in method, Lcom/google/firebase/database/core/utilities/Utilities; sha1HexDigest (Ljava/lang/String;)Ljava/lang/String;
CWE-328 is detected in method, Linfosecadventures/allsafe/challenges/WeakCryptography; md5Hash (Ljava/lang/String;)Ljava/lang/String;
CWE-328 is detected in method, Linfosecadventures/allsafe/challenges/SQLInjection; md5 (Ljava/lang/String;)Ljava/lang/String;

Detect CWE-295 in Android Application (InsecureShop.apk)

This scenario seeks to find Improper Certificate Validation. See CWE-295 for more details.

Let’s use this APK and the above APIs to show how the Quark script finds this vulnerability.

We use the API findMethodInAPK to locate all SslErrorHandler.proceed methods. Then we need to identify whether the method WebViewClient.onReceivedSslError is overridden by its subclass.

First, we check and make sure that the MethodInstance.name is onReceivedSslError, and the MethodInstance.descriptor is (Landroid/webkit/WebView; Landroid/webkit/SslErrorHandler; Landroid/net/http/SslError;)V.

Then we use the method API MethodInstance.findSuperclassHierarchyto get the supclass list of the method's caller class.

Finally, we check the Landroid/webkit/WebViewClient; is on the supclass list. If YES , that may cause CWE-295 vulnerability.

API Spec

MethodInstance.findSuperclassHierarchy()

• Description: Find all superclass hierarchy of this method object.
• params: None
• Return: Python list contains all superclas's name of the this method.

Quark Script CWE-295.py

from quark.script import findMethodInAPK

SAMPLE_PATH = "insecureShop.apk"
TARGET_METHOD = [
    "Landroid/webkit/SslErrorHandler;",  # class name
    "proceed",                          # method name
    "()V"                               # descriptor
]
OVERRIDE_METHOD = [
    "Landroid/webkit/WebViewClient;",  # class name
    "onReceivedSslError",              # method name
    # descriptor
    "(Landroid/webkit/WebView; Landroid/webkit/SslErrorHandler; Landroid/net/http/SslError;)V"
]

for sslProceedCaller in findMethodInAPK(SAMPLE_PATH, TARGET_METHOD):
    if (sslProceedCaller.name == OVERRIDE_METHOD[1] and
       sslProceedCaller.descriptor == OVERRIDE_METHOD[2] and
       OVERRIDE_METHOD[0] in sslProceedCaller.findSuperclassHierarchy()):
        print(f"CWE-295 is detected in method, {sslProceedCaller.fullName}")

Quark Script Result

$python3 CWE-295.py
Requested API level 29 is larger than maximum we have, returning API level 28 instead.
CWE-295 is detected in method, Lcom/insecureshop/util/CustomWebViewClient; onReceivedSslError (Landroid/webkit/WebView; Landroid/webkit/SslErrorHandler; Landroid/net/http/SslError;)V

Detect CWE-295 in Android Application (InsecureShop.apk)

This scenario seeks to find Improper Certificate Validation. See CWE-295 for more details.

Let’s use this APK and the above APIs to show how the Quark script finds this vulnerability.

We use the API findMethodInAPK to locate all SslErrorHandler.proceed methods. Then we need to identify whether the method WebViewClient.onReceivedSslError is overridden by its subclass.

First, we check and make sure that the MethodInstance.name is onReceivedSslError, and the MethodInstance.descriptor is (Landroid/webkit/WebView; Landroid/webkit/SslErrorHandler; Landroid/net/http/SslError;)V.

Then we use the method API MethodInstance.findSuperclassHierarchyto get the supclass list of the method's caller class.

Finally, we check the Landroid/webkit/WebViewClient; is on the supclass list. If YES , that may cause CWE-295 vulnerability.

API Spec

MethodInstance.findSuperclassHierarchy()

• Description: Find all superclass hierarchy of this method object.
• params: None
• Return: Python list contains all superclas's name of the this method.

Quark Script CWE-295.py

from quark.script import findMethodInAPK

SAMPLE_PATH = "insecureShop.apk"
TARGET_METHOD = [
    "Landroid/webkit/SslErrorHandler;",  # class name
    "proceed",                          # method name
    "()V"                               # descriptor
]
OVERRIDE_METHOD = [
    "Landroid/webkit/WebViewClient;",  # class name
    "onReceivedSslError",              # method name
    # descriptor
    "(Landroid/webkit/WebView; Landroid/webkit/SslErrorHandler; Landroid/net/http/SslError;)V"
]

for sslProceedCaller in findMethodInAPK(SAMPLE_PATH, TARGET_METHOD):
    if (sslProceedCaller.name == OVERRIDE_METHOD[1] and
       sslProceedCaller.descriptor == OVERRIDE_METHOD[2] and
       OVERRIDE_METHOD[0] in sslProceedCaller.findSuperclassHierarchy()):
        print(f"CWE-295 is detected in method, {sslProceedCaller.fullName}")

Quark Script Result

$python3 CWE-295.py
Requested API level 29 is larger than maximum we have, returning API level 28 instead.
CWE-295 is detected in method, Lcom/insecureshop/util/CustomWebViewClient; onReceivedSslError (Landroid/webkit/WebView; Landroid/webkit/SslErrorHandler; Landroid/net/http/SslError;)V

Quark CWE team

The Quark CWE team is responsible for developing Quark Scripts to detect Common Weakness Enumeration (CWE) vulnerabilities in APKs. We also maintain the Quark Script document, API, and repository.

Goals for 2023

Our goals for 2023 consist of three stages. First, we will focus on increasing the number of CWE Quark Scripts to 30 and optimizing the Quark Script API by developing CWE Quark Scripts.

Next, with a sufficient number of Quark Scripts, we will develop a system to automatically detect vulnerabilities in online APKs.

Finally, based on the sufficient and quality Quark Script API, we will focus on developing a web system that allows users to easily combine Quark Script APIs and create their own scripts without any coding knowledge.

Responsibilities

We aims to make the Quark Script development process as straightforward as possible, while ensuring that the scripts are accurate and reliable. We strive to create clear and concise documentation, as well as well-designed APIs that are easy to use. Our responsibilities include:

• Developing Quark Scripts through a five-step process:
  1. Choosing a CWE number and clearly explaining the vulnerability definition.
  2. Finding an APK sample and explaining the vulnerable code.
  3. Designing the detection process step by step.
  4. Defining a new Quark Script API (including description, input, and output) if necessary.
  5. Developing the Quark Script in a clear and easy-to-use manner.
• Managing the Quark Script repository by:
  • Updating the repository with new Quark Scripts.
  • Updating the documentation for Quark Scripts.
• Maintaining the Quark Script API by:
  • Developing test units for each Quark Script API.
  • Reviewing and modifying the description, input, and output for each API.

We aim to ensure that all of our work is easy to read and follows proper grammar and usage.

Describe the bug

When referring to a method or file in the Quark Script showcases, we mark the name with backticks to make it easy to distinguish (e.g., configureJsExecution.json). However, some showcases in the Quark Script doc don't follow this practice.

For example, the CWE-94 showcase doesn't mark the method name with backticks.

the CWE-780 showcase shows the method name in italics font.

Describe the solution you'd like

Unify the format of method names. For example, we could mark them with backticks.