< Node Types | Currently Supported XML Formats >

Semantic Structure

MagicTree root node is a branch node and is always called "magictree".

All data about the test targets gathered in the course of a project is kept under "testdata" node which is a child node of "magictree".

Nodes for hosts, fully qualified domain names (fqdn), network blocks (netblock) and possibly web application url (webapp) are created directly under "testdata".

Network Blocks

Network blocks are represented by simple nodes

The network blocks have "netblock" as type and the IP address and netmask as value. The is represented using the dotted decimal notation followed by slash and the number of "1" bits in netmask, for example "192.168.1.0/24" (but not "192.168.1.0/255.255.255.0").

XML:

<netblock>10.1.1.128/25</netblock>

Fully Qualified Domain Names

Fully qualified domain names are represented by simple nodes

Fully qualified domain names have "fqdn" as type and the domain name, including the trailing dot as value, for example "www.gremwell.com.".

XML:

<fqdn>www.gremwell.com.</fqdn>

Hosts

Hosts are represented by simple nodes

Hosts have "host" as type and the IP address as value, for example "192.168.1.1".

XML:

<host>10.1.1.1</host>

In MagicTree hosts are identified by IP address, not by DNS name. All the data related to a host is stored under the "host" node.

We are aware that a DNS name may resolve to more than one IP address, a physical host may have multiple IP addresses, and one IP address may have multiple host names (virtual hosts). We believe that in a context of a network security assessment, an IP address is usually a more convinient way of identifying entities being tested. To store one or more DNS names that a given IP address may have, we use hostname objects.

Web Applications

Web application vulnerability scanners sometimes do not report the IP address of the host being scanned. They do report the top-level URL of the application. To accomodate the findings of such scanners "webapp" nodes may be created directly under "testdata" nodes. Those "webapp" nodes are intended to represent the entry point of a web application. It is not advisable to create a webapp node for every URL found in the web application. Instead "url" nodes may be created under "webapp" node or, preferebly, if the information about the host IP address and port number is available, under the appropriate host, port and service node. "webapp" nodes may also be located under a "service" node, if the host IP address and port is known.

Hostnames

DNS host names are represented by simple nodes and are located directly under a host node. One host can have multiple DNS names and therefore multiple hostname objects. DNS host names do not need a trailing dot.

XML:

<host>192.168.1.1
<hostname>shiny2.gremwell.com</hostname>
</host>

MAC addresses

MAC addresses are represented by simple nodes and are located directly under a host node.

XML:

<host>192.168.1.1
<mac>00:21:70:a0:47:17</mac>
</host>

Operating System

Operating systems are represented by simple nodes and are located directly under a host node.

XML:

<host>192.168.1.1
<os>Linux</os>
</host>

A host may hve more then one "os" nodes, because different tools may have different opinion on the operating syustem of the remote host. Even one tool may be in doubt about it and return several guesses.

NetBIOS names

NetBIOS names are represented by simple nodes and are located directly under a host node.

XML:

<host>192.168.1.1
<nbname>SHINY2</nbname>
</host>

TCP and UDP ports

TCP ports are located under <ipproto>tcp</ipproto> node under host node. UDP ports are located under <ipproto>udp</ipproto> node under host node. Both are represented by simple nodes.

XML:

<host>192.168.1.1
<ipproto>tcp
<port>80</port>
</ipproto>
<ipproto>udp
<port>161</port>
</ipproto>
</host>

Ports may have states, such as "open", "closed" or "filtered". They are represented by a "state" node under the port node:

<port>80
<state>open</state>
</port>

IP protocols

IP protocols are located directly under the "host" node and have "ipproto" type. TCP and UDP are represented as <ipproto>tcp</ipproto> and <ipproto>udp</ipproto> respectively. Other IP protocols may be represented using the protocol number instead of protocol name, for example <iproto>1</ipproto> if the name is not known.

XML:

<host>
<ipproto>tcp
<port>80</port>
</ipproto>
<iproto>icmp</ipproto>
<iproto>99</ipproto>
</host>

IP protocols, similar to ports, may have state,such as "open", "closed" or "filtered". They are represented by a "state" node under the protocol node:

<ipproto>99
<state>open</state>
</ipproto>

Application Protocols / Serices

Application level protocols (a.k.a. services) are located under the "port" node where the service was detected, for example

<iproto>tcp
<port>80
<service>http</service>
</port>
</ipproto>

An exception to this rule are services that work over SSL (for example, HTTPS, LDAPS, etc.). The nodes for those services are located under a <tunnel>ssl</tunnel> node, which is located directly under the port node:

<iproto>tcp
<port>443
<tunnel>ssl
<service>http</service>
</tunnel>
</port>
</ipproto>

Please note that the service above is listed as "http", not "https". This is a convention we follow for all SSL services. Instead of using a different service name for an SSL-based service, we use the same name as its non-SSL counterpart and wrap it in an <tunnel>ssl</tunnel>. This allows us to handle similar SSL and non-SSL services together (for example, run a CGI scan for both HTTP and HTTPS servers), and also allows us easily select all SSL-based services, for instance, to test for the presence of SSLv3/TLSv1 renegotiation vulnerbility.

In order to map the names of the SSL-based services returned by various tools to non-SSL names, we keep a list of mappings in the file called service-map.xml, stored in the XSLT directory. The file also serves to unify the service names used by different tools. For example, various tools may use "www" or "http" or "http-alt" for HTTP service. "www" and "http-alt" are mapped to "http" in service-map.xml

Software

If the software name and version is detected, it can be listed under the service:

<service>http
<software>Apache
<version>2.2.2</version>
</software>
</service>

Vulnerability Scanner Findings

Vulnerability scanner (Nessus, Nikto, Burp, Qualys, Acunetix, W3AF, etc.) findings are represented by Text objects. The finding nodes can be located under the service node to which the finding apply, under the host node, if the finding applies to the host as a whole, or under an "url" node for web application scanner findings.

Qualys findings are an unfortunate exception, because Qualys XML format does not allow to determine easily which port and service the finding applies to, so Qualys findings generally end up under the host node.

If you are writing an XSLT transform for vulnerability scanner output, try whenever possible to place findings under the appropriate port and service node.

Not all web application scanners include the IP address of the scanned host in the output XML. This makes placing the finding under the appropriate host node difficult. Since the XML file may be read by MagicTree at a different time and on a different host from when and where the scanner was executed, looking up the IP address during import may produce different result or no result at all.

All web application scanners appear to report the entry URL of the application being scanned. It is therefore seems natural to use the web application entry URL as the parent node for the findings nodes. The application entry URLs are represented by webapp nodes.

The "title" attribute of the Text object representing the finding contains a short title of the vulnerability, for example "SSL/TLS Renegotiation is Enabled". The text of the Text object contains the description of the finding.

The element name for findings is always "finding".

Findings should have a child node called "source" that indicates which vulnerability scanner has returned this finding.

Another child node called "id" represents the finding id returned by the scanner. For example, for Nessus, this will be Nessus plugin id.

Findings also have "source-severity" child node, that can have the values of "high", "medium", "low", "info" and "unknown". This node indicates the severity of the finding as reported by the source vulnerability scanner. Scanners may use other ways of grading the findings, so they have to be mapped to "high", "medium", "low" or "unknown" in XSLT. To simplify sorting, "source-severity" node has "numeric" child node, that gives a numeric representation of severity. The severities are numerically represented as follows: high = 3, medium = 2, low = 1, info = 0, unknown = -1.

<source-severity>high
<numeric>3</numeric>
</source-severity>

If the vulnerability scanner outputs a short summary of vulnerability in addition to a more detailed description, it is represented by "synopsis" Text node. The "title" attribute of this node is set to "Synopsis".

If the vulnerability scanner outputs solution or recommendations, it is represented by "solution" Text node. The "title" attribute of this node is set to "Solution".

For reporting convinience, the "finding" node can have one or more child nodes representing the affected system, URL and or parameter. Those nodes have the type "affects" and the value representing the thing affected, for example:

<affects>192.168.1.0:21 (tcp)</affects>
<affects>http://www.example.com/test.php [test parameter]</affects>
<affects>http://test.example.com/</affects>

Web application vulnerability scanners may provide the HTTP method that was used to trigger the vulnerability. It may be specified using "http-method" node, for example:

<http-method>POST</http-method>

The CVE IDs of the findings can be represented by one or more "cve" nodes located under the "finding" node. The BugTraq IDs can be represented by one or more "bid" nodes under "finding" node. OSVDB IDs can be represented by "osvdb" nodes.

References to external resources, such as URLs of advisories, are represented by "reference" nodes. The value of the "reference" node contains the reference name, for example "RFC 2549 - IP over Avian Carriers with Quality of Service". A "url" child node may contain the URL of the referenced document:

<reference>
RFC 2549 - IP over Avian Carriers with Quality of Service
<url>http://tools.ietf.org/html/rfc2549</url>
</reference>

Vulnerability scanners often output some information that is specific to a particular tested host/port. For example, Nessus provides "plugin output", and Burp provides "issue detail". This data is represented by "output" Text node with title attribute set to "Details".

The HTTP requests and responses, if available are represented as follows:

<requestresponse>1
<request class="MtTextObject" title="request"> GET / HTTP/1.1
Host: www.example.com
</request>
<reponse>
HTTP/1.1 404 Not Found Content-Length: 19

404 File Not found

</response>
</requestresponse>
<requestresponse>1
<request class="MtTextObject" title="request"> GET / HTTP/1.1
Host: www.example.com
</request>
<reponse>
HTTP/1.1 404 Not Found Content-Length: 19

404 File Not found

</response>
</requestresponse>

MagicTree allows grouping similar findings in reports, so that information that is the same for multiple hosts and ports (description, solution, references, etc.) is listed only once, while host/port specific output is still clearly reported. The sample report template "summary-of-findings-with details" provides an example of such grouping.

If the vulnerability scanner outputs CVSS information, it can be represented by "cvss vector" and "cvss base score" nodes.

As menitioned above findings are represented by Text nodes.Text nodes are compared by name and title, the text content is not used for comparison. Sometimes a vulnerability scanner can output multiple findings with the same title for a given endpoint. For example, multiple XSS vulnerabilities may be found on a given HTTP port. When an XML file is loaded by MagicTree, the sibling nodes that are equal are always merged together. However, multiple XSS findings are separate entities and really should not be merged together. To solve this problem we have introduced a special attribute "mergeID". It's value can be an arbitrary string. When two nodes are compared for equality for the purpose of merging, the presense of mergeID is checked. If both have mergeID and the two mergeIDs are different, the two nodes are considered not equal and are not merged together. In all other cases (no mergeID is present, or only one of the two nodes have mergeID) the two nodes are compared using the usual rules. If you are wrting an XSLT and you have nodes that are equal acording to MagicTree node comparison rules, while in fact they represent separate real world entities, add a unique mergeID attribute to each one of them, so that they don't get merged together. We do this in Burp XSLT using Burp's finding serial number for mergeID value.

Example:

<finding title="SSL Certificate with Wrong Hostname" class="MtTextObject">
The commonName (CN) of the SSL certificate presented on this port is for a different machine.
<source class="MtSimpleObject">nessus</source>
<id class="MtSimpleObject">45411</id>
<source-severity class="MtSimpleObject">medium</source-severity>
<output title="Details"class="MtTextObject">
The following hostnames were checked :
www.example.com
</output>
<synopsis title="Synopsis"class="MtTextObject">
The SSL certificate for this service is for a different host.
</synopsis>
<solution title="Solution" class="MtTextObject">
Purchase or generate a proper certificate for this service.
</solution>
<cvss vector class="MtSimpleObject">
CVSS2#AV:N/AC:L/Au:N/C:N/I:P/A:N
</cvss vector>
<cvss base score class="MtSimpleObject">
5.0
</cvss base score>
</finding>

< Node Types | Currently Supported XML Formats >