Metadata-Version: 2.1
Name: emews
Version: 0.9.3
Summary: Network traffic automation and prototyping.
Home-page: https://mews.sv.cmu.edu/research/emews/
License: GPL-3.0-only
Author: Brian Ricks
Author-email: absolutefunk@utdallas.edu
Requires-Python: >=3.8
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Networking
Requires-Dist: bs4 (>=0.0.1)
Requires-Dist: linuxfd (>=1.4.4)
Requires-Dist: pexpect (>=4.6)
Requires-Dist: psutil (>=5.8)
Requires-Dist: ruamel.yaml (>=0.17)
Requires-Dist: scapy (>=2.5)
Requires-Dist: scipy (>=1.5.4)
Requires-Dist: urllib3 (>=1.25.8)
Project-URL: Repository, https://github.com/absolutefunk/emews
Description-Content-Type: text/markdown

# eMews - Network Traffic Automation and Prototyping

eMews is a framework which automates client-side computer network traffic generation, while providing centralized monitoring of a network while traffic is being generated.  eMews provides an API to write network applications which abstract away basic functionality such as peer-to-peer communication.  These applications, called _services_, provide client-side traffic generation, but can also be used to prototype complex network protocols, or even agent-environment interaction, in which the environment is the network itself [2].  eMews is designed for emulated networks such as those built with CORE, enabling quick deployment over potentially large-scale networks which reside on a single physical host [1].

Bundled with eMews are services which can generate SSH and HTTPS client-side network traffic in a connection-orientated paradigm.  This is different from a playback approach, in that the traffic generated factors in response traffic from the server, and can be captured as PCAP data representing valid TCP sessions.  By writing your own eMews services, you can generate any client-side traffic for any network protocol you desire.

With eMews you can quickly deploy large-scale botnets for flooding-type network traffic captures.  eMews comes with a botnet service which can generate distributed UDP or TCP traffic, and supports multiple concurrent botnets.

- eMews version used in IRI-18 [1] paper: v0.32.
- eMews version used in IRI-19 [2] paper: v0.4.

## Quick Start
We will assume that eMews is going to be run with the CORE 9 network emulator on an Ubuntu 24.04 system.  To install CORE, see [here](https://coreemu.github.io/core/).  While CORE has an Ubuntu package, it may be more straightforward to build CORE from source.  eMews is tested to work with CORE 8 and 9, through CORE 9.2.1, the most recent version as of this writing.  While this guide is written for CORE 9, if a step is different to perform in CORE 8, the difference will be mentioned so CORE 8 users can also work with eMews.

Create a Python virtual environment (venv) that you will install eMews in.  The venv must be created using the venv tool, and the venv must be named '.venv'.

If you plan to write services for eMews, then a straightforward way to create a venv is to use an IDE such as [PyCharm](https://www.jetbrains.com/pycharm/) and create a new project for eMews development.  By default, PyCharm will create a venv for the project (using the venv tool and with name '.venv'), and then you can enter it by simply opening a terminal session in PyCharm.

Assuming you have created and entered your virtual environment, download / install eMews using `pip`:

```shell
pip install emews
```

You need to know where eMews was installed so you can modify CORE's configuration accordingly.  Type the following to get the eMews package path:

```shell
pip show emews
```

The path is located under `Location` in the output, for example:

```
Location: /home/username/emews/.venv/lib/python3.12/site-packages
```

Copy the path given at Location, which we will call `<emews_pkg_path>`, and open `core.conf`.  In Ubuntu 24.04, `core.conf` is usually located at `/opt/core/etc`. Modify the following line in `core.conf`:

```
custom_services_dir = <emews_pkg_path>/emews/core_emu/core_services
```

If using CORE 8, you'll need to modify the following lines in `core.conf` instead:

```
custom_services_dir = <emews_pkg_path>/emews/core_emu/core8_legacy_services
custom_config_services_dir = <emews_pkg_path>/emews/core_emu/core8_config_services
```

eMews provides a sample `core.conf` file under `<emews_pkg_path>/emews/core_emu/example_conf`.  eMews interfaces with CORE through CORE services which map to eMews nodes and services.  These CORE services are generated by eMews and will be discussed in a bit.  Note that CORE services are different from eMews services.  Starting from CORE 8, CORE service structure changed and these services were called 'config services'.  Legacy CORE services were still supported and called 'services'.  With CORE 9, support for legacy services was dropped and 'config services' were renamed to 'services', including API changes which required updating eMews CORE service generation. 

- The venv containing eMews does not need to be active when running a CORE network as CORE will know where eMews is located via the `custom_services_dir` entry in `core.conf`.

### Building CORE Networks
CORE networks can be built using `core_gui`.  When building a CORE network to run eMews, one CORE node in the network should be designated to run the eMews hub.  This node should be the first one created in the network ('n1') and should be a host type like a PC.  In `core_gui`, right-click on this node, and select 'Services'.  If running CORE 8, you'll select 'Config Services' instead.  Under 'Groups', select 'eMews' and then select 'HubNode' as the service.  Note that multiple eMews services can run on a single CORE node.

- If you do not see 'eMews' as a group option, then most likely your `custom_services_dir` (or `custom_config_services_dir` if using CORE 8) was not set correctly in `core.conf`.

By default, the IP address assigned to the first node created in a CORE network is 10.0.0.20.  This IP address is also where eMews expects the HubNode service to reside.  If the IP address of the CORE node running the eMews hub is not 10.0.0.20, then this IP address will need to be set in the eMews system configuration file located at `<emews_pkg_path>/emews/system.yml`.  Under the `hub` section of `system.yml` change this field:

```
address: 10.0.0.20
```

from 10.0.0.20 to the IP address of the CORE node running the eMews HubNode service.

- To skip having to change the hub node IP address in `system.yml`, simply assign the eMews 'HubNode' service to the first CORE node created.
- eMews was not originally designed with PyPi packaging in mind, and thus the global system configuration (along with services) are contained within the package.

All remaining CORE nodes that will run eMews services need to additionally run an eMews host.  In `core_gui`, right-click on a CORE node which should be an eMews host, and select 'Services' (or 'Config Services' if using CORE 8).  Select 'eMews' as the group, and select 'HostNode' as the service.  Then select additional eMews services that you'd like to run on the selected node.

- The eMews hub can itself run eMews services without requiring the 'HostNode' service.

When running a CORE network with eMews-enabled nodes, you can both control eMews and monitor what it is doing by connecting to the hub node.  In `core_gui`, you need to first enable a control network for your CORE network so you can connect to the eMews hub node externally.  With your CORE network open in `core_gui`, click on the 'Session' menu, select 'Options', and type an IP address range for 'Control Network', for example 172.16.0.0/24.

- The IP address range for the control network should be private and not conflict with existing networks your physical host interfaces with, for example your school or home network.  See [here](https://www.arin.net/reference/research/statistics/address_filters/) for a list of private IPv4 address ranges.

Example CORE networks running eMews nodes are located under `<emews_pkg_path>/emews/core_emu/networks`.

### Controlling and Monitoring eMews
When running your CORE network, which is called a _CORE session_, you may want to run an eMews console and monitor to interact with eMews and view log output, respectively.  To start a session of your network, click the 'Start Session' button, and wait for your network to finish start-up (for large networks this can take some time).  Once your network is running, double-click on the hub node in `core_gui`.  This opens a root shell, and in this shell type:

```shell
ifconfig
```

The NIC for the control network will be called `ctrl0`.  Copy the IP address assigned to this NIC, which we will call `<hub_ctrl_ip>`.  Then, open a local console, activate the eMews venv (or if using an IDE like PyCharm open the eMews project and then open a terminal session within the IDE) and type:

```shell
python3 -m emews -n monitor1 monitor --hub_address <hub_ctrl_ip>
```

This will create a local standalone eMews node called 'monitor1', which will connect to the hub node and output log messages.  Now open another local console, activate the eMews venv as before (of if using an IDE open another terminal session) and type:

```shell
python3 -m emews -n console1 console --hub_address <hub_ctrl_ip>
```

This will create a local standalone eMews node called 'console1', which will connect to the hub node and provide a console in which you can type commands to eMews.

### Console Node Commands
When a CORE session starts, eMews services will be suspended.  To manually start all eMews services, type in the console node:

```
exp start
```

Here, 'exp' is short for 'experiment', which is what we refer to as eMews' goal, for example network traffic generation.

- eMews can start an experiment automatically by setting the `system.yml` field `scenario_autostart` to `True` under the `hub` section

To check if there are any issues with a running experiment, type:

```
exp status
```

If the status is green, then all eMews services are running normally.  To get a detailed list of all services and their execution state, type:

```
list services
```

To stop all eMews services, type:

```
exp stop
```

The eMews console supports many commands to interact with eMews.  You can check resource usage, start and stop individual services, and even create service instances on a node.  Type `help` to get a list of commands.  Type `help <command>` to get help on a specific command.

## eMews Services
eMews comes with client-side services to interact with servers such as SSH and HTTPS, along with bots for generating various DDoS traffic.  All services which eMews comes with are located here:

```
<emews_pkg_path>/emews/services
```

Each service is in a sub-folder named accordingly.  Services are configured using `.yml` files within the service folder.  There is at least one configuration per service called `default.yml`, but there may be others.  When adding an eMews service to a CORE node in `core_gui`, the configuration loaded is `default.yml`.  To load another configuration instead, right-click the CORE node and click 'Services'.  Then under 'Selected', click the eMews services you would like to configure and click 'Configure'.  All the available configurations are given in a drop-down list.

## Helpful Hints
CORE network nodes are run as containers under root, and are built under `/tmp/pycore.###` (the '###' is a number).  In this directory are directories for each CORE node, named `<node_name>.conf`, where `<node_name>` is the name of the node as assigned in `core_gui` (such as 'n1').  For example, the path to a CORE node called 'n1' would be:

```
/tmp/pycore.###/n1.conf
```

All CORE nodes running eMews contain log files which start with `emews` and end with `.log`.  These files log console output (including exceptions) from each eMews daemon and service launcher, and may contain useful messages related to local issues.  For example, a CORE node running an eMews host has two log files:  `emews_node_host.log` and `emews_servspawn.log`.  The first provides local (to the CORE node) log output of the eMews host, whereas the second provides local log output for eMews service spawning.  To obtain eMews service log output among all nodes in the network, launch a monitor node.
 
If you want eMews services to start automatically when your CORE network is started, under `<emews_pkg_path>/emews/system.yml`, set `hub.scenario_autostart` to `True`.  When your CORE network starts, eMews services will start automatically after an amount of seconds given by `hub.scenario_autostart_delay`.  This enables you to launch a CORE network and start eMews services without having to connect a console session to the hub node.

References:

[1] B. Ricks, P. Tague and B. Thuraisingham, "Large-Scale Realistic Network Data Generation on a Budget," IEEE Intl Conference on Information Reuse and Integration for Data Science (IRI), Jul 2018
- URL: https://mews.sv.cmu.edu/papers/iri-18.pdf

[2] B. Ricks, B. Thuraisingham, and P. Tague, "Mimicking Human Behavior in Shared-Resource Computer Networks", IEEE Intl Conference on Information Reuse and Integration for Data Science (IRI), Jul 2019
- URL: https://mews.sv.cmu.edu/papers/iri-19.pdf

