Logo

Smarts Contracts Security Tools Comparison: MythX, Mythril, Securify v2.0 and Slither


Von Yamila Levalle, Security Researcher at Dreamlab Technologies

Ethereum Smart Contracts Security Tools

While the rise of blockchain provides a unique opportunity to create smart contracts for digital assets such as tokens and Non-Fungible Tokens (NFTs), smart contract applications come with unique security concerns that have led to the losses of millions of dollars in the past. Fortunately, the Ethereum ecosystem has matured in recent years, and there are now several high-quality tools available to check your Ethereum smart contracts for security vulnerabilities. But keep in mind that thesetools are not a substitute for auditing. They can only find a certain subset of vulnerabilities, and for important contracts you still need to have a professional smart contract audit performed. 

This blogpost will introduce four of the most popular smart contract security analysis tools: Slither, Mythril, MythX and Securify v2.0 and compare the results of analyzing three intentionally vulnerable smart contracts with them. Smartcheck was not included in the comparison as it seems to be outdated and unmaintained.Oyente was also excluded, as this tool only supports Solidity up to version 0.4.19.

Smart Contracts to Analyse

The analysis will be performed on the following smart contracts from Capture the Ether, a Capture The Flag (CTF) style game in which Ethereum smart contracts are hacked to learn more about smart contracts security:

MythX

MythX by ConsenSys is a paid security analysis service for Ethereum smart contracts, it allows any developer team to integrate security into the smart contract development lifecycle. MythX is integrated into widely used tools such as VsCode andRemix . When code is submitted to the API it is then analyzed by multiple microservices in parallel: A static analyzer, a symbolic analyzer and a fuzzer. It hasseveral detectors, described here.

Using MythX first requires signup at their website: https://mythx.io/. After signup a plan is then chosen. In this example the on-demand plan was selected and a pack of 3 scans then purchased. An API Key must then be generated to integrate MythX with Solidity IDEs like VSCode or Remix, or use the MythX CLI. In this caseRemix was chosen, so the plugin "MYTHX SECURITY VERIFICATION" was activated and configured with the API Key or credentials.

 

Figure 1: Mythx Access Token Configuration

 

To run the scan, the smart contract to be analyzed is selected and the blue button "Analyze" clicked (quick analysis mode), with the results appearing in the Report tab:

Figure 2: Mythx Analysis Results of Tokensale.sol

 

Figure 3: Mythx Analysis Results of PredictTheFuture.sol

 

Figure 4: Mythx Analysis Results of FiftyYears.sol

 

Slither

Logo

Slither is a Solidity static analysis framework written in Python 3. It runs a suite of vulnerability detectors, prints visual information about contract details, and provides an API to easily write custom analyses. Slither enables developers to find vulnerabilities, enhance their code comprehension, and quickly prototype custom analyses. It has the following characteristics:

  • Easily integrates into continuous integration and Truffle builds
  • Built-in 'printers' quickly report crucial contract information
  • Detector API to write custom analyses in Python
  • Ability to analyze contracts written with Solidity 0.4 or above
  • It has 76 detectors for vulnerabilities (https://github.com/crytic/slither/wiki/Detector-Documentation)

For simplicity, in this blogpost the eth-security-toolbox docker image was used. It includes all trailofbits security tools and every major version of Solidity in a single image.

To build the container (the Dockerfile was modified slightly to include solc 0.4.21, as that wasn't included by default):
docker build -t eth-security-toolbox .

To run the container:
docker run -it -v /home/<smart contracts directory>:/share eth-security-toolbox

To select the solidity compiler version:
solc-select 0.4.21

Figure 5: Running Eth-security-toolbox container

 

To run the scan on the selected contract: slither /share/<contract>.sol

Figure 6: Slither Analysis Result ofTokenSale.sol

 

Figure 7: Slither Analysis Result ofPredictTheFuture.sol

 

Figure 8: Slither Analysis Result of FiftyYears.sol

 

Mythril

Mythril is a security analysis tool for EVM bytecode. It detects security vulnerabilities in smart contracts built for Ethereum, Hedera, Quorum, Vechain, Roostock, Tron and other EVM-compatible blockchains. It uses symbolic execution, SMT solving and taint analysis to detect a variety of security vulnerabilities. It's also used (in combination with other tools and techniques) inthe MythX security analysis platform.

For simplicity, in this blogpost the docker version of Mythril was used.

To pull the container:

docker pull mythril/myth

 

To run the container and scan the selected contract (the contract and the solidity version must be specified):
docker run -v <smart contracts directory>:/tmp mythril/myth analyze /tmp/<contract>.sol --solv <version>

Figure 9: Running Mythril Container and analyzing TokenSale.solcontract

 

Figure 10: Running Mythril Container and analyzing PredictTheFuture.solcontract

 

Figure 11: Running Mythril Container and analyzing FiftyYears.sol contract

 

Securify

Securify v2.0 is a security scanner for Ethereum smart contracts supported by the Ethereum Foundation and ChainSecurity. Itis the successor of the popular Securify security scanner. It has the following characteristics:

  • Supports 37 vulnerabilities (see table)
  • Implements novel context-sensitive static analysis written in Datalog
  • Analyzes contracts written in Solidity 0.5.8 or above

For simplicity, in this blogpost the docker version of Securify v2.0 was used.

To build the container:
sudo docker build -t securify .

To run the container and analyze the selected smart contract:
sudo docker run -it -v <smart contracts directory>:/share securify /share/<contract>.sol

Note: If the Solidity version is different than 0.5.12, the variable ARG SOLC=0.5.12 at the top of the Dockerfile must be modified to point to the version in use.

In this case the solidity version of the analyzed smart contracts was 0.4.21, so to be able to compare the tools (Securify2 doesn't work with solidity versions prior to 0.5.8) the three smart contracts had to be refactored to solidity 0.5.12 by modifying the constructor and the corresponding functions, but without changing the functionality or vulnerabilities of the contract. The refactored smart contracts were renamed to TokenSale5.sol, PredictTheFuture5.sol and FiftyYears5.sol to avoidconfusion:

 

Figure 12: Running Securify2 Container and analyzing TokenSale5.sol contract

 

Figure 13: Running Securify2 Container and analyzing PredictTheFuture5.sol contract

 

Figure 14: Running Securify2 Container and analyzing FiftyYears5.sol contract

 

Comparing the results

The SWC Registry - the Smart Contract Weakness Classification Registry - was used for comparison. This contains a list of potential smart contract weaknesses and vulnerabilities, along with explanations, references, and test cases. In the case of "Missing Input Validation" no suitable vulnerability in SWC registry was found, so SWC-137* was added to account for this vulnerabilitydetected by both Slither and Securify2. The severity of each vulnerability was collected from the tools output, in case of discordance the one with higher severity was chosen. This test focused on only 3 smart contracts, a more detailed and improved comparison would analyze a larger number of smart contracts and include smart contracts developed with different solidity versions.

Below is a table summarizing the results of the three analyzed smart contracts scans, the number below each tool represent the total instances of the vulnerability detected by the tool in the 3 contracts analyzed:

Figure 15: Summary of Smart Contracts Scan

 

Figure 16: Quantity of Vulnerability Instances detected by Each Tool Grouped by SWC ID

 

Figure 17: Percentage of vulnerability instances detected by all tools by SWD ID

 

Figure 18: Quantity of Vulnerabilities detected by each tool grouped by severity

 

Comparing the output, results and vulnerability classes detected by the four smart contract security tools, demonstrates that each tool has advantages and disadvantages:

 

  • Mythx is not free, unlike the other analyzed tools, and it has three different analysis modes: quick, standard and deep. In quick mode, it detected integer overflows and underflows, bad randomness, out of bounds array access and unprotected ether withdrawal but failed to detect transaction order dependency and outdated compiler version. In summary, Mythx was able to detect the 3 more obvious and dangerous vulnerabilities of the contracts that could lead to exploitation. MythX's output iscolour-coded by severity and includes SWC ID to link the vulnerabilities detected to the SWC Registry.
  • Mythril detected integer overflows and underflows, bad randomness, out of bounds array access and unprotected ether withdrawal, but failed to detect transaction order dependency, outdated compiler version and also failed to warn about solidity coding best practices: deprecated functions, state variables default visibility and floating pragma like MythX. In summary, Mythril was able to detect the 3 more obvious and dangerous vulnerabilities of the contracts that could lead to exploitation.In Mythril output, findings are very detailed, includes SWC ID to link the vulnerabilities detected to the SWC Registry and even includes example transactions and transaction sequences that can trigger the vulnerabilities found. 
  • Slither does not have detectors for integer overflow and underflow and therefore did not detect these vulnerabilities, but it was able to detect out of bounds array access (as SWC-109), missing input validations, bad randomness and timestamp dependence. In summary, Slither was able to detect the 2 of the 3 more obvious and dangerous vulnerabilities of the contracts that could lead to exploitation. Slither's output is colour-coded by severity, with high-severity detectors printed first,followed by the medium-severity and low-severity outputs and includes references for more information about the vulnerability. In addition, it includes some solidity coding best practices warnings that are useful but do not represent vulnerabilities.
  • Securify2 does not perform numerical analysis and therefore did not detect integer overflow/underflow, also did not detect unprotected ether withdrawal or bad source of randomness, but it did detect out of bounds array access (as SWC-109) and was the only tool to detect transaction order dependence. In summary, Securify v2.0 was able to detect the 1 of the 3 more obvious and dangerous vulnerabilities of the contracts that could lead to exploitation. Securify2 was the tool that detectedmost vulnerabilities overall, but also the tool with the most false positives, solidity coding best practices warnings and informational findings, among the 4 tools compared. The output is also rather confusing, as the colours do not seem to correspond to the severity level and the details of the reported vulnerabilities are insufficient. Care must be taken when reading the output to avoid misinterpreting the results.

 

Smart Contract security scanners like these certainly have their place in the contract development process and should be used. Using multiple tools, rather than relying on just one, seems to be the best approach for detecting potential bugs, as each tool is better suited to detect specific vulnerability classes and has specific methods to check for vulnerabilities in smart contracts (static analysis, symbolic execution, taint analysis, fuzzing). However, these tools are not capable ofdetecting vulnerabilities beyond their predefined list, and no tool or combination of tools can replace a full audit. Most of them fail in complex attacks that require interaction between multiple contracts. Always be aware that the more money your smart contract deals with, the higher the risk that the contract and its functions could be abused or attacked. By choosing to audit your code early in the development cycle, you can prevent devastating vulnerabilities and complex issues afterlaunch, as well as provide transparency to investors and the community.

 

__

 

References and Useful Links:

 

Yamila Levalle
Security Researcher at Dreamlab Technologies

Smarts Contracts Security Tools Comparison: MythX, Mythril, Securify v2.0 and Slither

Alle Blog-Posts