A quick introduction to the Shark architecture

This document is a brief introduction to the capabilities and organization of the Shark appliance. No background or previous experience with Shark or Pilot is assumed, but you are encouraged to watch the Shark demo video and/or spend some time using Pilot to familiarize yourself with the platform and its capabilities.

Views

All analysis and reporting from a Shark appliance revolves around the concept of a view. Views are extremely flexible and can be used for many different things but at the core, a view consists simply of a source of packets to analyze and a set of statistics to be computed on those packets. The power of the Shark platform derives from the number of statistics that can be exported and the number of ways those statistics can be combined and organized.

An example of a simple view is "Bandwidth over Time". In this view, the length statistic is summed across all packets for fixed small time intervals (e.g., 1 second). By plotting the sums for a series of consecutive time intervals, we get a graph of how the traffic load on a specific network link has varied over time.

A more complex example is the Pilot "Served Web Pages by Client IP" view. In Pilot, this view presents a table with three columns: ip address, URL, and count. There is a separate row in the table for each unique (client ip address, URL) pair, showing the number of times the given URL has been fetched from the given client ip address. This view could additionally be filtered, for example showing only URLs fetched using the Chrome web browser or showing only URLs with mime type image/jpeg. In this case, the statistics extracted are more complex -- extracting the URL (or browser or content type) from an HTTP session requires reassembling the underlying TCP connection from individual packets. And the organization of the statistics is a little more complex, the Shark internally constructs and updates a table much like the one presented in the Pilot user interface.

These examples are just meant to help illustrate what views are at a high level. The details about exactly how views are contstructed and accessed are later in this document and elsewhere in the FlyScript documentation.

Packet Sources

This section describes the abstractions provided by a Shark appliance for organizing and identifying the packets that can be used as the basis for creating a view.

Interfaces

Perhaps the simplest packet source is a raw network interface, also called a capture port. A capture port on a Shark appliance is typically connected to a switch port that mirrors live traffic to the Shark for monitoring and analysis (often called a SPAN port). When a view is applied to an interface, every packet that arrives on the interface is available to the view for processing. A view applied to an interface is also a live view, meaning that as new packets arrive, they are immediately processed by the view and show up in the view outputs.

Capture Jobs

A capture job is a long-running task on a Shark appliance that captures, stores, and indexes live traffic. It logically consists of a physical network interface (a capture port) and an optional filter to limit which packets should be captured. The actual packets and associated indexing information are all stored locally on the Shark appliance, using a custom filesystem tailored to the task of capturing traffic at a high rate and indexing the capture traffic for efficient queries. The capabilities of capture jobs are accessed from Python code using CaptureJob objects.

There are actually two different logical packets source associated with a capture job. The job itself refers to all the packets stored on disk that have been previously captured as part of the job. A view created with a capture job as the packet source is applied to all packets that are available on disk, but is not applied to new packets as they arrive. Instead, each capture job has an associated virtual device that is used for ongoing live traffic. The virtual device corresponding to a capture job appears like an interface, as described above.

The virtual device associated with a capture job may appear redundant with just applying a view with a filter to an interface. The filters associated with capture jobs are generally more efficient than view filters, so on an appliance handling a high sustained rate of packet capture, using a capture job is preferrable. A full description of the tradeoffs is beyond the scope of this document, for more details see the Shark manual.

Trace Clips

The packets captured as part of a capture job are kept in a first-in-first-out (FIFO) structured store. For cases when some packets need to be retained for longer, a trace clip is used. A trace clip identifies some subset of the packets stored as part of a capture job and optionally locks them, meaning that those packets will be kept on disk until the trace clip is deleted. Note that locking packets on disk as part of a trace clip effectively reduces the size of the FIFO store available for other ongoing capture jobs.

The exact set of packets that should go into a trace clip is identified using filters. Filters are described more fully below. But trace clips typically include at least a time filter to select packets from a specific time interval, and may include filters to select just traffic to/from a specific IP address and/or a specific TCP or UDP port number. Trace clips are accessed in FlyScript via the methods Shark.get_clips and CaptureJob.add_clip.

Trace Files

XXX explain

Extractors

Now that we have established how packets are directed into views, we turn to the topic of how information is extracted from packets. The central concept in this section is extractor fields. An extractor field is an individual piece of information. Simple extractor fields are computed from a single packet, typical examples are the packet length in bytes, or the IP source address of an IP packet. More complex extractor fields work on a stream of packets to compute higher level statistics. Examples of more compliated extractor fields are the jitter in a VOIP call or the number of retransmissions on a TCP connection -- these statistics can only be derived by looking at the relationship between multiple packets.

Every extractor field has a name of the form extractor.name used to uniquely identify the field, as well as a human-readable description that briefly explains what the field is. Extractor is a high-level category such as generic for fields that apply to any type of packet, tcp for fields that are specific to TCP, voip for fields specific to voice-over-IP, etc. The name is a unique short descriptive name. For example, generic.bits is the length of a packet in bits, and tcp.src_port is the source port from the TCP header of a TCP packet. In addition, each field has a fixed type (e.g., string, integer, IP address, etc.) and always yields values of that type.

The documentation for using extractor fields from FlyScript is located here. The next two sections explain how extractor fields are used in packet filters and in views.

Filters

Packet filters were mentioned above as part of creating views and trace clips. As the name suggests, a filter is applied to a packet source to select some subset of the original packets for view processing or for inclusion in a trace clip. Shark implements two types of filters: native filters and Wireshark filters. A native filter is simply a test applied to an extractor field. For example, ip.src="10.1.2.3" matches all packets with source IP address 10.1.2.3, or generic.bytes >= 100 matches all packets at least 100 bytes long.

Shark includes an extensive set of built-in extractor fields for common protocols including IP, TCP, UDP, RTP, HTTP, and others. Many powerful filters can be built using these extractors, but the number of applications that Shark has built-in extractors for is modest compared to the Wireshark network analyzer. Wireshark has well over 100,000 extractor fields -- for cases that are not covered by the shark built-in extractors, a wireshark filter can be used instead. A wireshark filter is specified using the Wireshark Display Filter syntax. Although wireshark filters offer great flexibility, they are implemented by starting a separate wireshark process and sending packets to that process for filtering, so they are significantly slower than native filters and may not be appropriate for applying to very large traces or high-rate live traffic.

The abstractions for working with filters from FlyScript are documented here.

Inside a View

We now have all the concepts needed to explain how Shark views are constructed. A view consists of two major pieces: which packets should be analyzed, and what analysis should be done on those packets. The first piece (which packets should be analyzed) is specified as a packet source and optionally, some filters.

The actual analysis is organized around extractor fields. When a view is created, a list of extractor fields is supplied, each of which is designated as either a key or a value. We begin with value fields. Each value field corresponds to a single output for the view, but of course there can be many packets processed by the view, so the field has an associated calculation that specifies how the indvidiual samples are aggregated into a single statistic. Possible calculations include summing the individual samples, computing the average, or finding the minimum or maximum value.

To make this concrete, consider a simple example of a view with just one field: packet size. If we use the averaging calculation, this view computes the average size of all packets in the input. By choosing a different calculation we could easily compute the minimum or maximum packet size, or the total number of bytes across all packets. This computation is automatically organized into a series of discrete time intervals, at a frequency specified when the view is created. Using our example above, by using the addition operator to aggregate packet sizes, we actually get a time series with the total bandwidth used in each 1 second interval -- effectively a view of bandwidth over time. Similarly, with any other extractor fields and aggregation operators, the view output is a time series indicating how the aggregated value field has changed over time.

Returning to key fields, a view is actually organized as a table with a column for each extractor field. If there are no key fields, as in our examples above, the table simply has one default row where all statistics are collected. But when key fields are specified, a tuple is created consisting of the values of all the key columns, and a new row is added to the table for each unique tuple. To make this concrete, one could create a simple view to compute a list of all TCP/IP connections by specifying four key columns: source IP address, source TCP port, destination IP address, and destination TCP port. In this way, a new row would be created in the view table for each unique connection 4-tuple, so reading the contents of the view table yields a list of all connections.

Combining key and value columns, we can merge our examples above by creating a view with key columns for the source and destination addresses and ports and a value column for total packet size. We now get a table of all connections with the total number of bytes sent on each connection. If we sort this table by the total bytes sent, we have a "top talkers" report!

XXX describe more examples...?