# Server-Client Simulations

## FEMNIST Classifier Instructions
- Ensure that the ```data/femnist/data/train``` and ```data/femnist/data/test``` directories contain data
- Run ```python3 main.py -dataset femnist -model cnn```
- For more simulation options and details, see 'Additional Notes' section

## Sentiment140 Classifier Instructions
- Ensure that the ```data/sent140/data/train``` and ```data/sent140/data/test``` directories contain data
- If you already have a .txt file containing word embeddings (i.e. 'glove.6B.300d.txt') in the sent140 directory, run ```sent140/get_embs.py -f fp```, where fp is the file path to the .txt file, to generate a .json file for the classifier to load. Otherwise, run ```./sent140/get_embs.sh```, which downloads the embeddings and creates the .json file.
- Run ```python3 main.py -dataset sent140 -model stacked_lstm```
- For more simulation options and details, see 'Additional Notes' section

## Shakespeare Classifier Instructions
- Ensure that the ```data/shakespeare/data/train``` and ```data/shakespeare/data/test``` directories contain data
- Run ```python3 main.py -dataset shakespeare -model stacked_lstm```
- For more simulation options and details, see 'Additional Notes' section

## Additional Notes
- In order to run these reference implementations, the ```-t sample``` tag must have been used when running the ```./preprocess.sh``` script for the respective dataset
- The total number of clients simulated equals the total number of users in the respective dataset's training data
- For optimal model performance, generate data using arguments similar to those listed in the 'large-sized dataset' option in the respective dataset README file.
- ```main.py``` supports these additional tags:
    - ```--model```: name of model; options are listed the respective dataset folder, for example ```cnn``` for femnist; defaults to first model in the respective dataset folder
    - ```--num_rounds```: number of rounds to simulate
    - ```--eval_every```: evaluate every ___ rounds
    - ```--clients_per_round```: number of clients trained per round
    - ```--batch_size```: batch size when clients train on data
    - ```--num_epochs```: number of epochs when clients train on data
    - ```-t```: simulation time: small, medium, or large; greater time corresponds to higher accuracy; for large runs, generate data using arguments similar to those listed in the 'large-sized dataset' option in the respective dataset README file for optimal model performance; default: large
    - ```-lr```: learning rate for local optimizers. 
- After running a classifier, open ```metrics.ipynb``` to view systems and statistical metrics from the last run.
- Metrics generated by models are stored in ```metrics.json```, which contains the following 'key: value' pairs:
    - dataset: name of the dataset
    - num_rounds: number of rounds simulated
    - eval_every: integer *x* such that model evaluated every *x* rounds
    - accuracies: length-num_rounds list of (ids, groups, num_samples, accs), where 
        - ids is a list of client id strings corresponding to the clients tested in the respective round, 
        - groups is a list containing the group names of each of those clients (list of Nones if clients don't have an associated group), 
        - num_samples is a list containing the number of test samples for each of those clients, and
        - accs is a list containing accuracy from each client's evaluation
    - client_computations: dictionary with client ids as keys and length-num_rounds lists as values; the elements in the lists are ints representing the number of FLOPs computed by the respective client in the corresponding round
    - bytes_written: dictionary with client ids as keys and length-num_rounds lists as values; the elements in the lists are ints representing the number of bytes written to the server by the resepctive client in the corresponding round
    - bytes_read: dictionary with client ids as keys and length-num_rounds lists as values; the elements in the lists are ints representing the number of bytes read from the server by the resepctive client in the corresponding round
